[jira] [Commented] (HADOOP-15278) log s3a at info
[ https://issues.apache.org/jira/browse/HADOOP-15278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522731#comment-16522731 ] Steve Loughran commented on HADOOP-15278: - FWIW, that metrics stuff should be gone soon; Todd moved the metrics off introspection for performance, but I think it should make things quieter too (no BeanHelper, see) > log s3a at info > --- > > Key: HADOOP-15278 > URL: https://issues.apache.org/jira/browse/HADOOP-15278 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.1.0 > > Attachments: HADOOP-15278-001.patch > > > since it was added, hadoop conf/log4j only logs s3a at ERROR, even though in > our test/resources it logs at info. We do actually log lots of stuff useful > when debugging things > Proposed: drop the log level to INFO here -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15278) log s3a at info
[ https://issues.apache.org/jira/browse/HADOOP-15278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399590#comment-16399590 ] Hudson commented on HADOOP-15278: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13839 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13839/]) HADOOP-15278 log s3a at info. Contributed by Steve Loughran. (fabbri: rev e840b4adf42082258ca56c50a858795749d564be) * (edit) hadoop-common-project/hadoop-common/src/main/conf/log4j.properties > log s3a at info > --- > > Key: HADOOP-15278 > URL: https://issues.apache.org/jira/browse/HADOOP-15278 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.1.0, 3.2.0 > > Attachments: HADOOP-15278-001.patch > > > since it was added, hadoop conf/log4j only logs s3a at ERROR, even though in > our test/resources it logs at info. We do actually log lots of stuff useful > when debugging things > Proposed: drop the log level to INFO here -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15278) log s3a at info
[ https://issues.apache.org/jira/browse/HADOOP-15278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399442#comment-16399442 ] Aaron Fabbri commented on HADOOP-15278: --- +1 I think this is a good change. I feel like some of the Metrics stuff could be quieted (i.e. only one INFO log about shutting down, the rest DEBUG). I'm running through tests and CLI commands and if nothing is too obnoxiously noisy I will commit this in an hour or so. > log s3a at info > --- > > Key: HADOOP-15278 > URL: https://issues.apache.org/jira/browse/HADOOP-15278 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15278-001.patch > > > since it was added, hadoop conf/log4j only logs s3a at ERROR, even though in > our test/resources it logs at info. We do actually log lots of stuff useful > when debugging things > Proposed: drop the log level to INFO here -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15278) log s3a at info
[
https://issues.apache.org/jira/browse/HADOOP-15278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390262#comment-16390262
]
genericqa commented on HADOOP-15278:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
28m 41s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
58s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HADOOP-15278 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12913440/HADOOP-15278-001.patch
|
| Optional Tests | asflicense mvnsite unit |
| uname | Linux e4b1ac05f0d3 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 46d29e3 |
| maven | version: Apache Maven 3.3.9 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14274/testReport/ |
| Max. process+thread count | 334 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14274/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> log s3a at info
> ---
>
> Key: HADOOP-15278
> URL: https://issues.apache.org/jira/browse/HADOOP-15278
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15278-001.patch
>
>
> since it was added, hadoop conf/log4j only logs s3a at ERROR, even though in
> our test/resources it logs at info. We do actually log lots of stuff useful
> when debugging things
> Proposed: drop the log level to INFO here
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15278) log s3a at info
[
https://issues.apache.org/jira/browse/HADOOP-15278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390165#comment-16390165
]
Steve Loughran commented on HADOOP-15278:
-
Actually, this is what you see in the default
{code}
2018-03-07 20:20:48,755 INFO beanutils.FluentPropertyBeanIntrospector: Error
when creating PropertyDescriptor for public final void
org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
Ignoring this property.
2018-03-07 20:20:48,780 INFO impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
2018-03-07 20:20:48,844 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot
period at 10 second(s).
2018-03-07 20:20:48,844 INFO impl.MetricsSystemImpl: s3a-file-system metrics
system started
2018-03-07 20:20:50,108 INFO Configuration.deprecation:
fs.s3a.server-side-encryption-key is deprecated. Instead, use
fs.s3a.server-side-encryption.key
Found 1 items
drwxrwxrwx - stevel stevel 0 2018-03-07 20:20
s3a://hwdev-steve-london/Users
2018-03-07 20:20:50,274 INFO impl.MetricsSystemImpl: Stopping s3a-file-system
metrics system...
2018-03-07 20:20:50,274 INFO impl.MetricsSystemImpl: s3a-file-system metrics
system stopped.
2018-03-07 20:20:50,274 INFO impl.MetricsSystemImpl: s3a-file-system metrics
system shutdown complete.
{code}
Something about deprecation I've never got away, and metrics cruft. The fluent
property one is HADOOP-15277; think its actually metrics related.
for reference, azure
{code}
bin/hadoop fs -ls wasb://contr...@contender.blob.core.windows.net
2018-03-07 20:26:16,848 INFO util.log: Logging initialized @1371ms
2018-03-07 20:26:16,968 INFO beanutils.FluentPropertyBeanIntrospector: Error
when creating PropertyDescriptor for public final void
org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
Ignoring this property.
2018-03-07 20:26:17,003 INFO impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
2018-03-07 20:26:17,075 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot
period at 10 second(s).
2018-03-07 20:26:17,075 INFO impl.MetricsSystemImpl: azure-file-system metrics
system started
Found 7 items
-rw-r--r-- 1 stevel supergroup 0 2017-06-30 18:12
wasb://contr...@contender.blob.core.windows.net/user/stevel/file
-rw-r--r-- 1 stevel supergroup513 2017-06-30 18:10
wasb://contr...@contender.blob.core.windows.net/user/stevel/file.dat
-rw-r--r-- 1 stevel supergroup 0 2017-06-30 18:13
wasb://contr...@contender.blob.core.windows.net/user/stevel/fileToRename
drwxr-xr-x - stevel supergroup 0 2017-06-30 18:11
wasb://contr...@contender.blob.core.windows.net/user/stevel/parent
drwxr-xr-x - stevel supergroup 0 2017-06-30 18:11
wasb://contr...@contender.blob.core.windows.net/user/stevel/renamedFolder
drwxr-xr-x - stevel supergroup 0 2016-10-20 22:00
wasb://contr...@contender.blob.core.windows.net/user/stevel/streaming
drwxr-xr-x - stevel supergroup 0 2017-06-30 18:11
wasb://contr...@contender.blob.core.windows.net/user/stevel/testDestFolder
2018-03-07 20:26:17,756 INFO impl.MetricsSystemImpl: Stopping azure-file-system
metrics system...
2018-03-07 20:26:17,756 INFO impl.MetricsSystemImpl: azure-file-system metrics
system stopped.
2018-03-07 20:26:17,756 INFO impl.MetricsSystemImpl: azure-file-system metrics
system shutdown complete.
{code}
so: equivalent noisyness around metrics. Dangerous to turn that off though, as
this is the same log4j used in the servers, which also spin the metrics. wasb
and s3a are just the two clients to use the metrics classes
> log s3a at info
> ---
>
> Key: HADOOP-15278
> URL: https://issues.apache.org/jira/browse/HADOOP-15278
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> since it was added, hadoop conf/log4j only logs s3a at ERROR, even though in
> our test/resources it logs at info. We do actually log lots of stuff useful
> when debugging things
> Proposed: drop the log level to INFO here
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15278) log s3a at info
[
https://issues.apache.org/jira/browse/HADOOP-15278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390131#comment-16390131
]
Steve Loughran commented on HADOOP-15278:
-
Nope, nothing by default
{code}
bin/hadoop fs -ls s3a://hwdev-steve-london/
Found 1 items
drwxrwxrwx - stevel stevel 0 2018-03-07 20:12
s3a://hwdev-steve-london/Users
{code}
you do get more on other ops, like committing work, which is where I'd like it
> log s3a at info
> ---
>
> Key: HADOOP-15278
> URL: https://issues.apache.org/jira/browse/HADOOP-15278
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> since it was added, hadoop conf/log4j only logs s3a at ERROR, even though in
> our test/resources it logs at info. We do actually log lots of stuff useful
> when debugging things
> Proposed: drop the log level to INFO here
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15278) log s3a at info
[
https://issues.apache.org/jira/browse/HADOOP-15278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384338#comment-16384338
]
Mingliang Liu commented on HADOOP-15278:
Oh you mean default logging level. I'm with that generally. Will the command
line be affected? I don't prefer too verbose output when I simply run the {{dfs
ls s3a://bucket/path/}}
> log s3a at info
> ---
>
> Key: HADOOP-15278
> URL: https://issues.apache.org/jira/browse/HADOOP-15278
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> since it was added, hadoop conf/log4j only logs s3a at ERROR, even though in
> our test/resources it logs at info. We do actually log lots of stuff useful
> when debugging things
> Proposed: drop the log level to INFO here
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
