[jira] [Commented] (HADOOP-15549) Upgrade to commons-configuration 2.1 regresses task CPU consumption
[ https://issues.apache.org/jira/browse/HADOOP-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1675#comment-1675 ] Yuming Wang commented on HADOOP-15549: -- Thanks [~ste...@apache.org]. Two new JIRAs have been created: https://issues.apache.org/jira/browse/HADOOP-16086 https://issues.apache.org/jira/browse/HADOOP-16087 > Upgrade to commons-configuration 2.1 regresses task CPU consumption > --- > > Key: HADOOP-15549 > URL: https://issues.apache.org/jira/browse/HADOOP-15549 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 3.0.2 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.2.0 > > Attachments: hadoop-15549.txt > > > HADOOP-13660 upgraded from commons-configuration 1.x to 2.x. > commons-configuration is used when parsing the metrics configuration > properties file. The new builder API used in the new version apparently makes > use of a bunch of very bloated reflection and classloading nonsense to > achieve the same goal, and this results in a regression of >100ms of CPU time > as measured by a program which simply initializes DefaultMetricsSystem. > This isn't a big deal for long-running daemons, but for MR tasks which might > only run a few seconds on poorly-tuned jobs, this can be noticeable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15549) Upgrade to commons-configuration 2.1 regresses task CPU consumption
[ https://issues.apache.org/jira/browse/HADOOP-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755032#comment-16755032 ] Steve Loughran commented on HADOOP-15549: - no reason why not, just needs to go through the patch submission process. Create a new JIRA, "Backport HADOOP-15549 to branch-3.1", submit this pr with a -branch-3.1-001 suffix and see how it goes. Sounds like it makes sense for branch-3.0 too > Upgrade to commons-configuration 2.1 regresses task CPU consumption > --- > > Key: HADOOP-15549 > URL: https://issues.apache.org/jira/browse/HADOOP-15549 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 3.0.2 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.2.0 > > Attachments: hadoop-15549.txt > > > HADOOP-13660 upgraded from commons-configuration 1.x to 2.x. > commons-configuration is used when parsing the metrics configuration > properties file. The new builder API used in the new version apparently makes > use of a bunch of very bloated reflection and classloading nonsense to > achieve the same goal, and this results in a regression of >100ms of CPU time > as measured by a program which simply initializes DefaultMetricsSystem. > This isn't a big deal for long-running daemons, but for MR tasks which might > only run a few seconds on poorly-tuned jobs, this can be noticeable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15549) Upgrade to commons-configuration 2.1 regresses task CPU consumption
[
https://issues.apache.org/jira/browse/HADOOP-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754869#comment-16754869
]
Yuming Wang commented on HADOOP-15549:
--
Cloud we backport this patch to {{branch-3.1}}? I hint
{{IllegalArgumentException}}:
{noformat}
02:44:34.707 ERROR org.apache.hadoop.hive.ql.exec.Task: Job Submission failed
with exception 'java.io.IOException(Cannot initialize Cluster. Please check
your configuration for mapreduce.framework.name and the correspond server
addresses.)'
java.io.IOException: Cannot initialize Cluster. Please check your configuration
for mapreduce.framework.name and the correspond server addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:116)
at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:109)
at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:102)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:475)
at org.apache.hadoop.mapred.JobClient.(JobClient.java:454)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:369)
at
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:151)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
at
org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$runHive$1(HiveClientImpl.scala:730)
at
org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:283)
at
org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:221)
at
org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:220)
at
org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:266)
at
org.apache.spark.sql.hive.client.HiveClientImpl.runHive(HiveClientImpl.scala:719)
at
org.apache.spark.sql.hive.client.HiveClientImpl.runSqlHive(HiveClientImpl.scala:709)
at
org.apache.spark.sql.hive.StatisticsSuite.createNonPartitionedTable(StatisticsSuite.scala:719)
at
org.apache.spark.sql.hive.StatisticsSuite.$anonfun$testAlterTableProperties$2(StatisticsSuite.scala:822)
at
org.apache.spark.sql.test.SQLTestUtilsBase.withTable(SQLTestUtils.scala:284)
at
org.apache.spark.sql.test.SQLTestUtilsBase.withTable$(SQLTestUtils.scala:283)
at
org.apache.spark.sql.StatisticsCollectionTestBase.withTable(StatisticsCollectionTestBase.scala:40)
at
org.apache.spark.sql.hive.StatisticsSuite.$anonfun$testAlterTableProperties$1(StatisticsSuite.scala:821)
at
org.apache.spark.sql.hive.StatisticsSuite.$anonfun$testAlterTableProperties$1$adapted(StatisticsSuite.scala:820)
at scala.collection.immutable.List.foreach(List.scala:392)
at
org.apache.spark.sql.hive.StatisticsSuite.testAlterTableProperties(StatisticsSuite.scala:820)
at
org.apache.spark.sql.hive.StatisticsSuite.$anonfun$new$70(StatisticsSuite.scala:851)
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:104)
at
org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)
at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)
at org.scalatest.FunSuite.runTest(FunSuite.scala:1560)
at
org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229)
at
org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
at
[jira] [Commented] (HADOOP-15549) Upgrade to commons-configuration 2.1 regresses task CPU consumption
[ https://issues.apache.org/jira/browse/HADOOP-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521430#comment-16521430 ] Hudson commented on HADOOP-15549: - FAILURE: Integrated in Jenkins build Hadoop-precommit-ozone-acceptance #20 (See [https://builds.apache.org/job/Hadoop-precommit-ozone-acceptance/20/]) HADOOP-15549. Upgrade to commons-configuration 2.1 regresses task CPU (todd: [https://github.com/apache/hadoop/commit/59de9679540f6d0edfb34cf9f88e52b51d94b4f4]) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java > Upgrade to commons-configuration 2.1 regresses task CPU consumption > --- > > Key: HADOOP-15549 > URL: https://issues.apache.org/jira/browse/HADOOP-15549 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 3.0.2 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.2.0 > > Attachments: hadoop-15549.txt > > > HADOOP-13660 upgraded from commons-configuration 1.x to 2.x. > commons-configuration is used when parsing the metrics configuration > properties file. The new builder API used in the new version apparently makes > use of a bunch of very bloated reflection and classloading nonsense to > achieve the same goal, and this results in a regression of >100ms of CPU time > as measured by a program which simply initializes DefaultMetricsSystem. > This isn't a big deal for long-running daemons, but for MR tasks which might > only run a few seconds on poorly-tuned jobs, this can be noticeable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15549) Upgrade to commons-configuration 2.1 regresses task CPU consumption
[ https://issues.apache.org/jira/browse/HADOOP-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519642#comment-16519642 ] Hudson commented on HADOOP-15549: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14460 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14460/]) HADOOP-15549. Upgrade to commons-configuration 2.1 regresses task CPU (todd: rev 59de9679540f6d0edfb34cf9f88e52b51d94b4f4) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java > Upgrade to commons-configuration 2.1 regresses task CPU consumption > --- > > Key: HADOOP-15549 > URL: https://issues.apache.org/jira/browse/HADOOP-15549 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 3.0.2 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Fix For: 3.2.0 > > Attachments: hadoop-15549.txt > > > HADOOP-13660 upgraded from commons-configuration 1.x to 2.x. > commons-configuration is used when parsing the metrics configuration > properties file. The new builder API used in the new version apparently makes > use of a bunch of very bloated reflection and classloading nonsense to > achieve the same goal, and this results in a regression of >100ms of CPU time > as measured by a program which simply initializes DefaultMetricsSystem. > This isn't a big deal for long-running daemons, but for MR tasks which might > only run a few seconds on poorly-tuned jobs, this can be noticeable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15549) Upgrade to commons-configuration 2.1 regresses task CPU consumption
[ https://issues.apache.org/jira/browse/HADOOP-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519579#comment-16519579 ] Wei-Chiu Chuang commented on HADOOP-15549: -- +1 I reviewed HADOOP-13660 so I take responsibility for that change :) Verified manually properties are loaded properly. This change is more similar to pre-HADOOP-13660 so I feel safe about it. > Upgrade to commons-configuration 2.1 regresses task CPU consumption > --- > > Key: HADOOP-15549 > URL: https://issues.apache.org/jira/browse/HADOOP-15549 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 3.0.2 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Attachments: hadoop-15549.txt > > > HADOOP-13660 upgraded from commons-configuration 1.x to 2.x. > commons-configuration is used when parsing the metrics configuration > properties file. The new builder API used in the new version apparently makes > use of a bunch of very bloated reflection and classloading nonsense to > achieve the same goal, and this results in a regression of >100ms of CPU time > as measured by a program which simply initializes DefaultMetricsSystem. > This isn't a big deal for long-running daemons, but for MR tasks which might > only run a few seconds on poorly-tuned jobs, this can be noticeable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15549) Upgrade to commons-configuration 2.1 regresses task CPU consumption
[ https://issues.apache.org/jira/browse/HADOOP-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518729#comment-16518729 ] Todd Lipcon commented on HADOOP-15549: -- [~mackrorysd] mind taking a look? > Upgrade to commons-configuration 2.1 regresses task CPU consumption > --- > > Key: HADOOP-15549 > URL: https://issues.apache.org/jira/browse/HADOOP-15549 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 3.0.2 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Attachments: hadoop-15549.txt > > > HADOOP-13660 upgraded from commons-configuration 1.x to 2.x. > commons-configuration is used when parsing the metrics configuration > properties file. The new builder API used in the new version apparently makes > use of a bunch of very bloated reflection and classloading nonsense to > achieve the same goal, and this results in a regression of >100ms of CPU time > as measured by a program which simply initializes DefaultMetricsSystem. > This isn't a big deal for long-running daemons, but for MR tasks which might > only run a few seconds on poorly-tuned jobs, this can be noticeable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15549) Upgrade to commons-configuration 2.1 regresses task CPU consumption
[ https://issues.apache.org/jira/browse/HADOOP-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518697#comment-16518697 ] Steve Loughran commented on HADOOP-15549: - LGTM, though I think we should wait for a review from someone who knows the code > Upgrade to commons-configuration 2.1 regresses task CPU consumption > --- > > Key: HADOOP-15549 > URL: https://issues.apache.org/jira/browse/HADOOP-15549 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 3.0.2 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Attachments: hadoop-15549.txt > > > HADOOP-13660 upgraded from commons-configuration 1.x to 2.x. > commons-configuration is used when parsing the metrics configuration > properties file. The new builder API used in the new version apparently makes > use of a bunch of very bloated reflection and classloading nonsense to > achieve the same goal, and this results in a regression of >100ms of CPU time > as measured by a program which simply initializes DefaultMetricsSystem. > This isn't a big deal for long-running daemons, but for MR tasks which might > only run a few seconds on poorly-tuned jobs, this can be noticeable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15549) Upgrade to commons-configuration 2.1 regresses task CPU consumption
[ https://issues.apache.org/jira/browse/HADOOP-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518583#comment-16518583 ] Todd Lipcon commented on HADOOP-15549: -- bq. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. Existing tests cover this code path -- just using a different set of APIs. I'm not 100% sure the behavior didn't change, but if it did change, it probably isn't commonly used functionality (and likely going back to what it was in hadoop 2) > Upgrade to commons-configuration 2.1 regresses task CPU consumption > --- > > Key: HADOOP-15549 > URL: https://issues.apache.org/jira/browse/HADOOP-15549 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 3.0.2 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Attachments: hadoop-15549.txt > > > HADOOP-13660 upgraded from commons-configuration 1.x to 2.x. > commons-configuration is used when parsing the metrics configuration > properties file. The new builder API used in the new version apparently makes > use of a bunch of very bloated reflection and classloading nonsense to > achieve the same goal, and this results in a regression of >100ms of CPU time > as measured by a program which simply initializes DefaultMetricsSystem. > This isn't a big deal for long-running daemons, but for MR tasks which might > only run a few seconds on poorly-tuned jobs, this can be noticeable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15549) Upgrade to commons-configuration 2.1 regresses task CPU consumption
[
https://issues.apache.org/jira/browse/HADOOP-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518552#comment-16518552
]
genericqa commented on HADOOP-15549:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
3s{color} | {color:blue} The patch file was not named according to hadoop's
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute
for instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 33m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 43s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
49s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15549 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928506/hadoop-15549.txt |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux f85ee0007c5d 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9a9e969 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14796/testReport/ |
| Max. process+thread count | 1365 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
[jira] [Commented] (HADOOP-15549) Upgrade to commons-configuration 2.1 regresses task CPU consumption
[ https://issues.apache.org/jira/browse/HADOOP-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518472#comment-16518472 ] Todd Lipcon commented on HADOOP-15549: -- I also noticed there were some weird error logs, but didn't investigate those. Just coming at this from a perf regression angle (trying to figure out various reasons why a sleep job with 1ms tasks regressed noticeably between Hadoop 2 and Hadoop 3) > Upgrade to commons-configuration 2.1 regresses task CPU consumption > --- > > Key: HADOOP-15549 > URL: https://issues.apache.org/jira/browse/HADOOP-15549 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 3.0.2 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Attachments: hadoop-15549.txt > > > HADOOP-13660 upgraded from commons-configuration 1.x to 2.x. > commons-configuration is used when parsing the metrics configuration > properties file. The new builder API used in the new version apparently makes > use of a bunch of very bloated reflection and classloading nonsense to > achieve the same goal, and this results in a regression of >100ms of CPU time > as measured by a program which simply initializes DefaultMetricsSystem. > This isn't a big deal for long-running daemons, but for MR tasks which might > only run a few seconds on poorly-tuned jobs, this can be noticeable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15549) Upgrade to commons-configuration 2.1 regresses task CPU consumption
[ https://issues.apache.org/jira/browse/HADOOP-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518426#comment-16518426 ] Steve Loughran commented on HADOOP-15549: - Is this also why we get those messages in the s3a & wasb connectors about initialization of metrics? > Upgrade to commons-configuration 2.1 regresses task CPU consumption > --- > > Key: HADOOP-15549 > URL: https://issues.apache.org/jira/browse/HADOOP-15549 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 3.0.2 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Attachments: hadoop-15549.txt > > > HADOOP-13660 upgraded from commons-configuration 1.x to 2.x. > commons-configuration is used when parsing the metrics configuration > properties file. The new builder API used in the new version apparently makes > use of a bunch of very bloated reflection and classloading nonsense to > achieve the same goal, and this results in a regression of >100ms of CPU time > as measured by a program which simply initializes DefaultMetricsSystem. > This isn't a big deal for long-running daemons, but for MR tasks which might > only run a few seconds on poorly-tuned jobs, this can be noticeable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15549) Upgrade to commons-configuration 2.1 regresses task CPU consumption
[
https://issues.apache.org/jira/browse/HADOOP-15549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518375#comment-16518375
]
Todd Lipcon commented on HADOOP-15549:
--
I ran a simple program which just calls DefaultMetricsSystem.initialize against
Hadoop 2.8.2 compared to 3.0.0 dist tarballs:
*2.8.2:
{code}
683.416696 task-clock (msec) #1.793 CPUs utilized
( +- 2.32% )
1,790 context-switches #0.003 M/sec
( +- 1.07% )
54 cpu-migrations#0.080 K/sec
( +- 17.64% )
13,688 page-faults #0.020 M/sec
( +- 0.54% )
2,216,866,739 cycles#3.244 GHz
( +- 1.62% )
2,299,332,469 instructions #1.04 insn per cycle
( +- 1.21% )
431,487,977 branches # 631.369 M/sec
( +- 1.17% )
19,346,551 branch-misses #4.48% of all branches
( +- 1.07% )
0.381138028 seconds time elapsed
( +- 2.52% )
{code}
*3.0.0:*
{code}
924.881803 task-clock (msec) #1.902 CPUs utilized
( +- 2.05% )
1,962 context-switches #0.002 M/sec
( +- 0.73% )
44 cpu-migrations#0.047 K/sec
( +- 11.15% )
20,593 page-faults #0.022 M/sec
( +- 0.55% )
3,042,371,457 cycles#3.289 GHz
( +- 1.67% )
3,165,586,053 instructions #1.04 insn per cycle
( +- 1.41% )
592,945,118 branches # 641.104 M/sec
( +- 1.36% )
25,735,278 branch-misses #4.34% of all branches
( +- 1.30% )
0.486354791 seconds time elapsed
( +- 2.04% )
{code}
Not all of the regression is due to the metrics system initialization, but with
a small patch that avoids the "builder" APIs, I can recover some of the
regression.
{code}
885.276567 task-clock (msec) #2.009 CPUs utilized
( +- 1.45% )
1,608 context-switches #0.002 M/sec
( +- 2.02% )
48 cpu-migrations#0.055 K/sec
( +- 12.98% )
18,949 page-faults #0.021 M/sec
( +- 0.88% )
2,908,533,684 cycles#3.285 GHz
( +- 0.46% )
3,045,577,520 instructions #1.05 insn per cycle
( +- 0.66% )
566,661,963 branches # 640.096 M/sec
( +- 0.67% )
24,309,912 branch-misses #4.29% of all branches
( +- 0.77% )
0.440731241 seconds time elapsed
( +- 2.98% )
{code}
It also loads fewer classes (1651 vs 1768) by eliminating usage of 'beanutil'
and a bunch of ancillary classes in commons-configuration.
> Upgrade to commons-configuration 2.1 regresses task CPU consumption
> ---
>
> Key: HADOOP-15549
> URL: https://issues.apache.org/jira/browse/HADOOP-15549
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Affects Versions: 3.0.2
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Major
>
> HADOOP-13660 upgraded from commons-configuration 1.x to 2.x.
> commons-configuration is used when parsing the metrics configuration
> properties file. The new builder API used in the new version apparently makes
> use of a bunch of very bloated reflection and classloading nonsense to
> achieve the same goal, and this results in a regression of >100ms of CPU time
> as measured by a program which simply initializes DefaultMetricsSystem.
> This isn't a big deal for long-running daemons, but for MR tasks which might
> only run a few seconds on poorly-tuned jobs, this can be noticeable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
