[jira] [Commented] (HDFS-14109) Improve hdfs auditlog format and support federation friendly
[
https://issues.apache.org/jira/browse/HDFS-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910155#comment-16910155
]
He Xiaoqiao commented on HDFS-14109:
my first thought is that add `nsid` for namenode audit log to support
federation more friendly and make a distinction between multiply namespaces if
we collect all namenode audit log together, however it seems that this is not
the common requirement, so I will cancel this JIRA and set to `not a problem`.
> Improve hdfs auditlog format and support federation friendly
>
>
> Key: HDFS-14109
> URL: https://issues.apache.org/jira/browse/HDFS-14109
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-14109.patch
>
>
> The following auditlog format does not well meet requirement for federation
> arch currently. Since some case we need to aggregate all namespace audit log,
> if there are some common path request(e.g. /tmp, /user/ etc. some path may
> not appear in mountTable, but the path is very real), we will have no idea to
> split them that which namespace it request to. So I propose add column
> {{nsid}} to support federation more friendly.
> {quote}2018-11-27 13:20:30,028 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs/hostn...@realm.com (auth:KERBEROS) ip=/10.1.1.2 cmd=getfileinfo
> src=/path dst=null perm=null proto=rpc clientName=null
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14109) Improve hdfs auditlog format and support federation friendly
[
https://issues.apache.org/jira/browse/HDFS-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910126#comment-16910126
]
Hadoop QA commented on HDFS-14109:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-14109 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14109 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949846/HDFS-14109.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/27555/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Improve hdfs auditlog format and support federation friendly
>
>
> Key: HDFS-14109
> URL: https://issues.apache.org/jira/browse/HDFS-14109
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-14109.patch
>
>
> The following auditlog format does not well meet requirement for federation
> arch currently. Since some case we need to aggregate all namespace audit log,
> if there are some common path request(e.g. /tmp, /user/ etc. some path may
> not appear in mountTable, but the path is very real), we will have no idea to
> split them that which namespace it request to. So I propose add column
> {{nsid}} to support federation more friendly.
> {quote}2018-11-27 13:20:30,028 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs/hostn...@realm.com (auth:KERBEROS) ip=/10.1.1.2 cmd=getfileinfo
> src=/path dst=null perm=null proto=rpc clientName=null
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14109) Improve hdfs auditlog format and support federation friendly
[
https://issues.apache.org/jira/browse/HDFS-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910123#comment-16910123
]
Wei-Chiu Chuang commented on HDFS-14109:
Is this still active? Does this duplicate HDFS-14625 in any ways? HDFS-14625 is
meant to support RBF.
> Improve hdfs auditlog format and support federation friendly
>
>
> Key: HDFS-14109
> URL: https://issues.apache.org/jira/browse/HDFS-14109
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-14109.patch
>
>
> The following auditlog format does not well meet requirement for federation
> arch currently. Since some case we need to aggregate all namespace audit log,
> if there are some common path request(e.g. /tmp, /user/ etc. some path may
> not appear in mountTable, but the path is very real), we will have no idea to
> split them that which namespace it request to. So I propose add column
> {{nsid}} to support federation more friendly.
> {quote}2018-11-27 13:20:30,028 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs/hostn...@realm.com (auth:KERBEROS) ip=/10.1.1.2 cmd=getfileinfo
> src=/path dst=null perm=null proto=rpc clientName=null
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14109) Improve hdfs auditlog format and support federation friendly
[
https://issues.apache.org/jira/browse/HDFS-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712348#comment-16712348
]
He Xiaoqiao commented on HDFS-14109:
Thanks [~xkrogen],[~kihwal] for discussing this issue.
{quote}I think as with most recent additions to the audit log, it should be
protected by a config which defaults to off. In particular, in an environment
using only a single namespace, we definitely don't want this information.{quote}
+1, only for federation with multi-namespace, and switch off default by a
config.
{quote}People deal with logs from multiple systems today without having to
insert the source identity in every single log line. {quote}
Actually, there are multiple system can deal with mass logs data. my opinion is:
1) the lowest-cost method to deal with logs. e.g. 10B audit-log records may
cost our amount computing resource if relay with other system.
2) another point, I consider this is scope of hdfs rather than push to other
system.
Maybe I missing some information, please give your feedback if there are
something wrong.
Thanks [~xkrogen], [~kihwal] again.
> Improve hdfs auditlog format and support federation friendly
>
>
> Key: HDFS-14109
> URL: https://issues.apache.org/jira/browse/HDFS-14109
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-14109.patch
>
>
> The following auditlog format does not well meet requirement for federation
> arch currently. Since some case we need to aggregate all namespace audit log,
> if there are some common path request(e.g. /tmp, /user/ etc. some path may
> not appear in mountTable, but the path is very real), we will have no idea to
> split them that which namespace it request to. So I propose add column
> {{nsid}} to support federation more friendly.
> {quote}2018-11-27 13:20:30,028 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs/hostn...@realm.com (auth:KERBEROS) ip=/10.1.1.2 cmd=getfileinfo
> src=/path dst=null perm=null proto=rpc clientName=null
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14109) Improve hdfs auditlog format and support federation friendly
[
https://issues.apache.org/jira/browse/HDFS-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711810#comment-16711810
]
Kihwal Lee commented on HDFS-14109:
---
This has less to do with federation itself. Rather, it is more about the way
audit logs are collected and processed from multiple namenodes. People deal
with logs from multiple systems today without having to insert the source
identity in every single log line.
> Improve hdfs auditlog format and support federation friendly
>
>
> Key: HDFS-14109
> URL: https://issues.apache.org/jira/browse/HDFS-14109
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-14109.patch
>
>
> The following auditlog format does not well meet requirement for federation
> arch currently. Since some case we need to aggregate all namespace audit log,
> if there are some common path request(e.g. /tmp, /user/ etc. some path may
> not appear in mountTable, but the path is very real), we will have no idea to
> split them that which namespace it request to. So I propose add column
> {{nsid}} to support federation more friendly.
> {quote}2018-11-27 13:20:30,028 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs/hostn...@realm.com (auth:KERBEROS) ip=/10.1.1.2 cmd=getfileinfo
> src=/path dst=null perm=null proto=rpc clientName=null
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14109) Improve hdfs auditlog format and support federation friendly
[
https://issues.apache.org/jira/browse/HDFS-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711766#comment-16711766
]
Erik Krogen commented on HDFS-14109:
I think as with most recent additions to the audit log, it should be protected
by a config which defaults to off. In particular, in an environment using only
a single namespace, we definitely don't want this information, and an
installation may already have some way of adding this information back at a
later time without the NameNode having to write it out on every single audit
entry.
> Improve hdfs auditlog format and support federation friendly
>
>
> Key: HDFS-14109
> URL: https://issues.apache.org/jira/browse/HDFS-14109
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-14109.patch
>
>
> The following auditlog format does not well meet requirement for federation
> arch currently. Since some case we need to aggregate all namespace audit log,
> if there are some common path request(e.g. /tmp, /user/ etc. some path may
> not appear in mountTable, but the path is very real), we will have no idea to
> split them that which namespace it request to. So I propose add column
> {{nsid}} to support federation more friendly.
> {quote}2018-11-27 13:20:30,028 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs/hostn...@realm.com (auth:KERBEROS) ip=/10.1.1.2 cmd=getfileinfo
> src=/path dst=null perm=null proto=rpc clientName=null
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14109) Improve hdfs auditlog format and support federation friendly
[
https://issues.apache.org/jira/browse/HDFS-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702689#comment-16702689
]
He Xiaoqiao commented on HDFS-14109:
[~kihwal] Thanks for your comments firstly.
{quote}NN audit log is already huge and making it even bigger by adding
redundant bytes is far from ideal.{quote}
1. I agree with you one hundred percent.
2. NN audit log is incomplete information now if we stand unified filesystem
perspective for Federation.
Please correct me if there are something wrong. Thanks again.
> Improve hdfs auditlog format and support federation friendly
>
>
> Key: HDFS-14109
> URL: https://issues.apache.org/jira/browse/HDFS-14109
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-14109.patch
>
>
> The following auditlog format does not well meet requirement for federation
> arch currently. Since some case we need to aggregate all namespace audit log,
> if there are some common path request(e.g. /tmp, /user/ etc. some path may
> not appear in mountTable, but the path is very real), we will have no idea to
> split them that which namespace it request to. So I propose add column
> {{nsid}} to support federation more friendly.
> {quote}2018-11-27 13:20:30,028 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs/hostn...@realm.com (auth:KERBEROS) ip=/10.1.1.2 cmd=getfileinfo
> src=/path dst=null perm=null proto=rpc clientName=null
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14109) Improve hdfs auditlog format and support federation friendly
[
https://issues.apache.org/jira/browse/HDFS-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702148#comment-16702148
]
Hadoop QA commented on HDFS-14109:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 47s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 217 unchanged - 1 fixed = 219 total (was 218) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 0s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestFsck |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.fs.viewfs.TestViewFileSystemLinkMergeSlash |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14109 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949846/HDFS-14109.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 97b66ca16c8d 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3ce99e3 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/25661/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/25661/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
[jira] [Commented] (HDFS-14109) Improve hdfs auditlog format and support federation friendly
[
https://issues.apache.org/jira/browse/HDFS-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701969#comment-16701969
]
Kihwal Lee commented on HDFS-14109:
---
I don't think it is a good idea. NN audit log is already huge and making it
even bigger by adding redundant bytes is far from ideal.
> Improve hdfs auditlog format and support federation friendly
>
>
> Key: HDFS-14109
> URL: https://issues.apache.org/jira/browse/HDFS-14109
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-14109.patch
>
>
> The following auditlog format does not well meet requirement for federation
> arch currently. Since some case we need to aggregate all namespace audit log,
> if there are some common path request(e.g. /tmp, /user/ etc. some path may
> not appear in mountTable, but the path is very real), we will have no idea to
> split them that which namespace it request to. So I propose add column
> {{nsid}} to support federation more friendly.
> {quote}2018-11-27 13:20:30,028 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs/hostn...@realm.com (auth:KERBEROS) ip=/10.1.1.2 cmd=getfileinfo
> src=/path dst=null perm=null proto=rpc clientName=null
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
