[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[
https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534581#comment-14534581
]
Kihwal Lee commented on HDFS-6888:
--
bq. One of the key points of the HDFS audit log is to show accesses to files,
including for security purposes. If a user can legitimately use getFileInfo(),
then it needs to get logged.
The patch allows users to exclude certain ops from audit log. {{getFileIno}}
needs to be removed from {{hdfs-default.xml}}, so that it is logged by default.
I will update the title.
Remove audit logging of getFIleInfo()
-
Key: HDFS-6888
URL: https://issues.apache.org/jira/browse/HDFS-6888
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Kihwal Lee
Assignee: Chen He
Labels: BB2015-05-TBR, log
Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-4.patch,
HDFS-6888-5.patch, HDFS-6888-6.patch, HDFS-6888.patch
The audit logging of getFileInfo() was added in HDFS-3733. Since this is a
one of the most called method, users have noticed that audit log is now
filled with this. Since we now have HTTP request logging, this seems
unnecessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14512817#comment-14512817 ] Allen Wittenauer commented on HDFS-6888: One of the key points of the HDFS audit log is to show accesses to files, including for security purposes. If a user can legitimately use getFileInfo(), then it needs to get logged. Remove audit logging of getFIleInfo() - Key: HDFS-6888 URL: https://issues.apache.org/jira/browse/HDFS-6888 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.0 Reporter: Kihwal Lee Assignee: Chen He Labels: log Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-4.patch, HDFS-6888-5.patch, HDFS-6888-6.patch, HDFS-6888.patch The audit logging of getFileInfo() was added in HDFS-3733. Since this is a one of the most called method, users have noticed that audit log is now filled with this. Since we now have HTTP request logging, this seems unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[
https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14512878#comment-14512878
]
Hadoop QA commented on HDFS-6888:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 35s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | javac | 7m 28s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 34s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 7m 54s | The applied patch generated 2
additional checkstyle issues. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 3s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | native | 3m 11s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 163m 59s | Tests failed in hadoop-hdfs. |
| | | 212m 16s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12676436/HDFS-6888-6.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a00e001 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/10395/artifact/patchprocess/checkstyle-result-diff.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/10395/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/10395/testReport/ |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10395/console |
This message was automatically generated.
Remove audit logging of getFIleInfo()
-
Key: HDFS-6888
URL: https://issues.apache.org/jira/browse/HDFS-6888
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Kihwal Lee
Assignee: Chen He
Labels: log
Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-4.patch,
HDFS-6888-5.patch, HDFS-6888-6.patch, HDFS-6888.patch
The audit logging of getFileInfo() was added in HDFS-3733. Since this is a
one of the most called method, users have noticed that audit log is now
filled with this. Since we now have HTTP request logging, this seems
unnecessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[
https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14180823#comment-14180823
]
Hadoop QA commented on HDFS-6888:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12676436/HDFS-6888-6.patch
against trunk revision a36399e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8483//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8483//console
This message is automatically generated.
Remove audit logging of getFIleInfo()
-
Key: HDFS-6888
URL: https://issues.apache.org/jira/browse/HDFS-6888
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Kihwal Lee
Assignee: Chen He
Labels: log
Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-4.patch,
HDFS-6888-5.patch, HDFS-6888-6.patch, HDFS-6888.patch
The audit logging of getFileInfo() was added in HDFS-3733. Since this is a
one of the most called method, users have noticed that audit log is now
filled with this. Since we now have HTTP request logging, this seems
unnecessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[
https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156658#comment-14156658
]
Eric Payne commented on HDFS-6888:
--
[~airbots], thank you for fixing this issue. +1 (non-binding) on the patch.
I have one very tiny nit and one question:
- In the description of {{hdfs-default.xml}}, I would have used the phrase
writes to instead of outputs
- Is there someplace that documents all of the configuration properties? Is
that generated automatically from the descriptions in the *xml files?
Remove audit logging of getFIleInfo()
-
Key: HDFS-6888
URL: https://issues.apache.org/jira/browse/HDFS-6888
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Kihwal Lee
Assignee: Chen He
Labels: log
Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-4.patch,
HDFS-6888.patch
The audit logging of getFileInfo() was added in HDFS-3733. Since this is a
one of the most called method, users have noticed that audit log is now
filled with this. Since we now have HTTP request logging, this seems
unnecessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[
https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14157037#comment-14157037
]
Hadoop QA commented on HDFS-6888:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12672571/HDFS-6888-5.patch
against trunk revision c2fa5d1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.server.balancer.TestBalancer
org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8299//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8299//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8299//console
This message is automatically generated.
Remove audit logging of getFIleInfo()
-
Key: HDFS-6888
URL: https://issues.apache.org/jira/browse/HDFS-6888
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Kihwal Lee
Assignee: Chen He
Labels: log
Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-4.patch,
HDFS-6888-5.patch, HDFS-6888.patch
The audit logging of getFileInfo() was added in HDFS-3733. Since this is a
one of the most called method, users have noticed that audit log is now
filled with this. Since we now have HTTP request logging, this seems
unnecessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[
https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14145683#comment-14145683
]
Hadoop QA commented on HDFS-6888:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12670807/HDFS-6888-4.patch
against trunk revision f48686a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8171//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8171//console
This message is automatically generated.
Remove audit logging of getFIleInfo()
-
Key: HDFS-6888
URL: https://issues.apache.org/jira/browse/HDFS-6888
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Kihwal Lee
Assignee: Chen He
Labels: log
Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-4.patch,
HDFS-6888.patch
The audit logging of getFileInfo() was added in HDFS-3733. Since this is a
one of the most called method, users have noticed that audit log is now
filled with this. Since we now have HTTP request logging, this seems
unnecessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[
https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132981#comment-14132981
]
Gera Shegalov commented on HDFS-6888:
-
Thanks, [~airbots] for updating the patch. +1 (non-binding)
Two nits:
debugCmdSet can be declared as a general Set
{code}
private SetString debugCmdSet = new HashSetString();
{code}
Its initialization can be more brief:
{code}
debugCmdSet.addAll(Arrays.asList(conf.getTrimmedStrings(
DFSConfigKeys.DFS_AUDIT_LOG_DEBUG_CMDLIST)));
{code}
Remove audit logging of getFIleInfo()
-
Key: HDFS-6888
URL: https://issues.apache.org/jira/browse/HDFS-6888
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Kihwal Lee
Assignee: Chen He
Labels: log
Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888.patch
The audit logging of getFileInfo() was added in HDFS-3733. Since this is a
one of the most called method, users have noticed that audit log is now
filled with this. Since we now have HTTP request logging, this seems
unnecessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[
https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129417#comment-14129417
]
Hadoop QA commented on HDFS-6888:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667859/HDFS-6888-3.patch
against trunk revision b02a4b4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7988//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7988//console
This message is automatically generated.
Remove audit logging of getFIleInfo()
-
Key: HDFS-6888
URL: https://issues.apache.org/jira/browse/HDFS-6888
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Kihwal Lee
Assignee: Chen He
Labels: log
Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888.patch
The audit logging of getFileInfo() was added in HDFS-3733. Since this is a
one of the most called method, users have noticed that audit log is now
filled with this. Since we now have HTTP request logging, this seems
unnecessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14129431#comment-14129431 ] Chen He commented on HDFS-6888: --- unit test failure is because of HDFS-6694 Remove audit logging of getFIleInfo() - Key: HDFS-6888 URL: https://issues.apache.org/jira/browse/HDFS-6888 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.0 Reporter: Kihwal Lee Assignee: Chen He Labels: log Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888.patch The audit logging of getFileInfo() was added in HDFS-3733. Since this is a one of the most called method, users have noticed that audit log is now filled with this. Since we now have HTTP request logging, this seems unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[
https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113786#comment-14113786
]
Hadoop QA commented on HDFS-6888:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org
against trunk revision c4c9a78.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7820//console
This message is automatically generated.
Remove audit logging of getFIleInfo()
-
Key: HDFS-6888
URL: https://issues.apache.org/jira/browse/HDFS-6888
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Kihwal Lee
Assignee: Chen He
Labels: log
Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888.patch
The audit logging of getFileInfo() was added in HDFS-3733. Since this is a
one of the most called method, users have noticed that audit log is now
filled with this. Since we now have HTTP request logging, this seems
unnecessary.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113795#comment-14113795 ] Chen He commented on HDFS-6888: --- It works on my mac. Is this because Hadoop repository move to git? Remove audit logging of getFIleInfo() - Key: HDFS-6888 URL: https://issues.apache.org/jira/browse/HDFS-6888 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.0 Reporter: Kihwal Lee Assignee: Chen He Labels: log Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888.patch The audit logging of getFileInfo() was added in HDFS-3733. Since this is a one of the most called method, users have noticed that audit log is now filled with this. Since we now have HTTP request logging, this seems unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113797#comment-14113797 ] Chen He commented on HDFS-6888: --- HEAD is now at c4c9a78 HADOOP-11001. Fix test-patch to work with the git repo. (kasha) Removing patchprocess/ Switched to branch 'trunk' Your branch is up-to-date with 'origin/trunk'. Current branch trunk is up to date. /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/jira: No such file or directory /bin/grep: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/jira: No such file or directory /bin/grep: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/jira: No such file or directory HDFS-6888 patch is being downloaded at Thu Aug 28 14:11:35 UTC 2014 from http://issues.apache.org /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/patch: No such file or directory cp: cannot stat '/home/jenkins/buildSupport/lib/*': No such file or directory /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/dev-support/smart-apply-patch.sh: line 44: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/patch: No such file or directory /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/dev-support/smart-apply-patch.sh: line 100: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/patch: No such file or directory /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/dev-support/smart-apply-patch.sh: line 102: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/patch: No such file or directory The patch does not appear to apply with p0 to p2 rm: cannot remove '/tmp/tmp.paths.7370': No such file or directory PATCH APPLICATION FAILED Remove audit logging of getFIleInfo() - Key: HDFS-6888 URL: https://issues.apache.org/jira/browse/HDFS-6888 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.0 Reporter: Kihwal Lee Assignee: Chen He Labels: log Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888.patch The audit logging of getFileInfo() was added in HDFS-3733. Since this is a one of the most called method, users have noticed that audit log is now filled with this. Since we now have HTTP request logging, this seems unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[
https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114141#comment-14114141
]
Hadoop QA commented on HDFS-6888:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12664998/HDFS-6888-3.patch
against trunk revision c4c9a78.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.security.TestRefreshUserMappings
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDecommission
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7829//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7829//console
This message is automatically generated.
Remove audit logging of getFIleInfo()
-
Key: HDFS-6888
URL: https://issues.apache.org/jira/browse/HDFS-6888
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Kihwal Lee
Assignee: Chen He
Labels: log
Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-3.patch,
HDFS-6888.patch
The audit logging of getFileInfo() was added in HDFS-3733. Since this is a
one of the most called method, users have noticed that audit log is now
filled with this. Since we now have HTTP request logging, this seems
unnecessary.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114200#comment-14114200 ] Chen He commented on HDFS-6888: --- Test error is because of HADOOP-11020 Remove audit logging of getFIleInfo() - Key: HDFS-6888 URL: https://issues.apache.org/jira/browse/HDFS-6888 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.0 Reporter: Kihwal Lee Assignee: Chen He Labels: log Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-3.patch, HDFS-6888.patch The audit logging of getFileInfo() was added in HDFS-3733. Since this is a one of the most called method, users have noticed that audit log is now filled with this. Since we now have HTTP request logging, this seems unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107249#comment-14107249 ] Chen He commented on HDFS-6888: --- Hi [~jira.shegalov], you lost me there. We need to let the getfileinfo cmd only log when the auditlog is in debug level, right? Remove audit logging of getFIleInfo() - Key: HDFS-6888 URL: https://issues.apache.org/jira/browse/HDFS-6888 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.0 Reporter: Kihwal Lee Assignee: Chen He Labels: log Attachments: HDFS-6888-2.patch, HDFS-6888.patch The audit logging of getFileInfo() was added in HDFS-3733. Since this is a one of the most called method, users have noticed that audit log is now filled with this. Since we now have HTTP request logging, this seems unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[
https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107394#comment-14107394
]
Gera Shegalov commented on HDFS-6888:
-
Hi [~airbots], sorry for being unclear.
[~kihwal] suggests:
bq. We could have logAuditEvent() check cmd against getfileinfo or a
*collection of such commands* and log at debug level.
Picking this idea up can you introduce some conf like
dfs.audit.loglevel.cmdlist=getfileinfo,anotherLogFloodingCmd,...
In {{o.a.h.hdfs.server.namenode.FSNamesystem.DefaultAuditLogger#initialize}}
you could read the list using {{auditDebugCmds
conf.getTrimmedStrings(dfs.audit.debug.cmdlist}} and use it for filtering.
Currenly v2 hardcodes getfileinfo.
{code}
---
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
+++
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
@@ -359,6 +359,9 @@ private void logAuditEvent(boolean succeeded,
UserGroupInformation ugi, InetAddress addr, String cmd, String src,
String dst, HdfsFileStatus stat) {
FileStatus status = null;
+if(cmd.equals(getfileinfo) !auditLog.isDebugEnabled()) {
+ return;
+}
if (stat != null) {
Path symlink = stat.isSymlink() ? new Path(stat.getSymlink()) : null;
Path path = dst != null ? new Path(dst) : new Path(src);
{code}
Also auditLog.isDebugEnabled() is a cheaper check, and should be done before
{{equals}}
Remove audit logging of getFIleInfo()
-
Key: HDFS-6888
URL: https://issues.apache.org/jira/browse/HDFS-6888
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Kihwal Lee
Assignee: Chen He
Labels: log
Attachments: HDFS-6888-2.patch, HDFS-6888.patch
The audit logging of getFileInfo() was added in HDFS-3733. Since this is a
one of the most called method, users have noticed that audit log is now
filled with this. Since we now have HTTP request logging, this seems
unnecessary.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[
https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105604#comment-14105604
]
Kihwal Lee commented on HDFS-6888:
--
See the patch in HDFS-3733. When this logging was added, a number of test cases
were added. These test cases need to be modified. open (i.e.
getBlockLocations()) is logged for webhdfs. For example,
{{testAuditWebHdfsOpen()} generates 3 audit log entries: getfileinfo/webhdfs,
open/webhdfs and open/rpc. After this change, it will have the last two
entries.
Remove audit logging of getFIleInfo()
-
Key: HDFS-6888
URL: https://issues.apache.org/jira/browse/HDFS-6888
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Kihwal Lee
Assignee: Chen He
Labels: log
Attachments: HDFS-6888.patch
The audit logging of getFileInfo() was added in HDFS-3733. Since this is a
one of the most called method, users have noticed that audit log is now
filled with this. Since we now have HTTP request logging, this seems
unnecessary.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105623#comment-14105623 ] Gera Shegalov commented on HDFS-6888: - getfileinfo a common RPC but we have identified many times using the audit log an app that needs some help in optimizing their FileSystem API usage. Remove audit logging of getFIleInfo() - Key: HDFS-6888 URL: https://issues.apache.org/jira/browse/HDFS-6888 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.0 Reporter: Kihwal Lee Assignee: Chen He Labels: log Attachments: HDFS-6888.patch The audit logging of getFileInfo() was added in HDFS-3733. Since this is a one of the most called method, users have noticed that audit log is now filled with this. Since we now have HTTP request logging, this seems unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105698#comment-14105698 ] Kihwal Lee commented on HDFS-6888: -- What if we log it at debug level? Remove audit logging of getFIleInfo() - Key: HDFS-6888 URL: https://issues.apache.org/jira/browse/HDFS-6888 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.0 Reporter: Kihwal Lee Assignee: Chen He Labels: log Attachments: HDFS-6888.patch The audit logging of getFileInfo() was added in HDFS-3733. Since this is a one of the most called method, users have noticed that audit log is now filled with this. Since we now have HTTP request logging, this seems unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[
https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105722#comment-14105722
]
Kihwal Lee commented on HDFS-6888:
--
We could have {{logAuditEvent()}} check {{cmd}} against getfileinfo or a
collection of such commands and log at debug level. Otherwise we will have to
change HdfsAuditLogger, which will introduce incompatibility. Users who want
to see this in the log can change the log level at runtime.
Remove audit logging of getFIleInfo()
-
Key: HDFS-6888
URL: https://issues.apache.org/jira/browse/HDFS-6888
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Kihwal Lee
Assignee: Chen He
Labels: log
Attachments: HDFS-6888.patch
The audit logging of getFileInfo() was added in HDFS-3733. Since this is a
one of the most called method, users have noticed that audit log is now
filled with this. Since we now have HTTP request logging, this seems
unnecessary.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[
https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106295#comment-14106295
]
Hadoop QA commented on HDFS-6888:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663508/HDFS-6888-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 3
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
org.apache.hadoop.hdfs.server.namenode.TestAuditLogger
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.TestDecommission
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7703//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7703//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7703//console
This message is automatically generated.
Remove audit logging of getFIleInfo()
-
Key: HDFS-6888
URL: https://issues.apache.org/jira/browse/HDFS-6888
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Kihwal Lee
Assignee: Chen He
Labels: log
Attachments: HDFS-6888-2.patch, HDFS-6888.patch
The audit logging of getFileInfo() was added in HDFS-3733. Since this is a
one of the most called method, users have noticed that audit log is now
filled with this. Since we now have HTTP request logging, this seems
unnecessary.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[
https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106334#comment-14106334
]
Hadoop QA commented on HDFS-6888:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663508/HDFS-6888-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 3
release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-hdfs-project/hadoop-hdfs
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7710//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7710//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7710//console
This message is automatically generated.
Remove audit logging of getFIleInfo()
-
Key: HDFS-6888
URL: https://issues.apache.org/jira/browse/HDFS-6888
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Kihwal Lee
Assignee: Chen He
Labels: log
Attachments: HDFS-6888-2.patch, HDFS-6888.patch
The audit logging of getFileInfo() was added in HDFS-3733. Since this is a
one of the most called method, users have noticed that audit log is now
filled with this. Since we now have HTTP request logging, this seems
unnecessary.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106362#comment-14106362 ] Gera Shegalov commented on HDFS-6888: - [~kihwal], I am +1 for making some commands debug level, so we have an option to capture them in the logs and remove them. [~airbots], how about making the list of DEBUG-level commands configurable via a csv list for conf.getTrimmedStrings instead of hardcoding it as in v2 of the patch. Remove audit logging of getFIleInfo() - Key: HDFS-6888 URL: https://issues.apache.org/jira/browse/HDFS-6888 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.0 Reporter: Kihwal Lee Assignee: Chen He Labels: log Attachments: HDFS-6888-2.patch, HDFS-6888.patch The audit logging of getFileInfo() was added in HDFS-3733. Since this is a one of the most called method, users have noticed that audit log is now filled with this. Since we now have HTTP request logging, this seems unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
