[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774618#comment-16774618 ] Hadoop QA commented on HADOOP-11223: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 57s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 52s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m 12s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 53s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch generated 1 new + 146 unchanged - 0 fixed = 147 total (was 146) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 25s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 36s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 37s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 96m 12s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HADOOP-11223 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12959682/HADOOP-11223.005.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 22e866d4ef1d 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct 31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 28d0bf9 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/15961/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/15961/testReport/ | | Max. process+thread count | 1384 (vs. ulimit of 1) | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/15961/console | | Powered
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774560#comment-16774560 ] Michael Miller commented on HADOOP-11223: - I don't quite follow your request for the changes to testAllMethods. Github PRs make feedback like this a lot easier and I pushed my changes to my branch if you want to give your ideas a shot on there: [https://github.com/apache/hadoop/pull/473] > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Michael Miller >Priority: Major > Labels: Performance > Attachments: HADOOP-11223.001.patch, HADOOP-11223.002.patch, > HADOOP-11223.003.patch, HADOOP-11223.004.patch, HADOOP-11223.005.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772966#comment-16772966 ] Steve Loughran commented on HADOOP-11223: - bq. Would it simplify things making this class package private and then adding a static method to Configuration to create the unmodifiable object? yes, but we should still add as @Unstable just to warn people that we are evolving this, and in fact the immutability itself may change > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Michael Miller >Priority: Major > Labels: Performance > Attachments: HADOOP-11223.001.patch, HADOOP-11223.002.patch, > HADOOP-11223.003.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772244#comment-16772244 ] Michael Miller commented on HADOOP-11223: - Would it simplify things making this class package private and then adding a static method to Configuration to create the unmodifiable object? > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Michael Miller >Priority: Major > Labels: Performance > Attachments: HADOOP-11223.001.patch, HADOOP-11223.002.patch, > HADOOP-11223.003.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770994#comment-16770994 ] Steve Loughran commented on HADOOP-11223: - I could probably also modify the config if it was referring to any JCEKS file for passwords, and I just modified that. What you've got is probably useful in a few places: passing on a config & you want to prevent accidental changes. Now, considering patch. * checkstyle is failing. please fix after making the other changes suggested below * Javadocs need to clarify this discussion about unmodifieable vers immutable & list ways where it can be changed underneath. * checkstyle is unhappy * Tag new class as @Public @Unstable. We may remove/change it if there are issues. * check your ordering of importss, in the hadoop code its {code} java. javax. non-asf org.apache static {code} Tests * use {{LambdaTestUtils.intercept()}} for those tests which expect exceptions to be thrown -we're adopting it in all new code. * I had a look at {{testAllMethods}} to see if we could do a parameterized junit test suite from it. I do think it's possible —the reflection code would go into a static {} clause which would create the parameter table of String method, List args, which would then be used to build the test method thanks > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Michael Miller >Priority: Major > Labels: Performance > Attachments: HADOOP-11223.001.patch, HADOOP-11223.002.patch, > HADOOP-11223.003.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769789#comment-16769789 ] Michael Miller commented on HADOOP-11223: - {quote}the "Unmodifiable" one is not immutable (see earlier discussion about addDefaultResource). {quote} Correct. I understand a true immutable object is not possible because of the static methods. This is why we would settle for an "Unmodifiable" object. This would at least prevent changes on the object itself, for example, in our Testing framework: {code:java} public void test(ServerContext context) { Configuration conf = context.getHadoopConf(); conf.set("prop", "value"); } {code} Here we would prefer an error be thrown since the framework expects configuration to be set a certain way. It is a lot easier to make this mistake then to think "I want to change a property so I am going to create another configuration, call this static method and reload it ". > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Michael Miller >Priority: Major > Labels: Performance > Attachments: HADOOP-11223.001.patch, HADOOP-11223.002.patch, > HADOOP-11223.003.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769741#comment-16769741 ] Gopal V commented on HADOOP-11223: -- bq. Mainly for performance and concurrency reasons we don't want the Hadoop Configuration to change or be re-read. This is not guaranteed by this patch, unfortunately - the "Unmodifiable" one is not immutable (see earlier discussion about addDefaultResource). > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Michael Miller >Priority: Major > Labels: Performance > Attachments: HADOOP-11223.001.patch, HADOOP-11223.002.patch, > HADOOP-11223.003.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769739#comment-16769739 ] Michael Miller commented on HADOOP-11223: - {quote} Michael Miller Sorry to join the party so late, just want to understand how Accumulo uses this feature, as this is still not read-only config, But surely benefits you someway. I would like to understand the use case a little better. Thanks {quote} Accumulo currently does not use this feature but it is something that would be beneficial. We previously had a static class that would save the Configuration object the first time it was created and then return that object whenever it was needed. There were a lot of changes internally to eliminate static state like this and only construct some objects once at startup of the server. Mainly for performance and concurrency reasons we don't want the Hadoop Configuration to change or be re-read. > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Michael Miller >Priority: Major > Labels: Performance > Attachments: HADOOP-11223.001.patch, HADOOP-11223.002.patch, > HADOOP-11223.003.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769711#comment-16769711 ] Anu Engineer commented on HADOOP-11223: --- [~milleruntime] Sorry to join the party so late, just want to understand how Accumulo uses this feature, as this is still not read-only config, But surely benefits you someway. I would like to understand the use case a little better. Thanks > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Michael Miller >Priority: Major > Labels: Performance > Attachments: HADOOP-11223.001.patch, HADOOP-11223.002.patch, > HADOOP-11223.003.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765488#comment-16765488 ] Michael Miller commented on HADOOP-11223: - The above failure seems unrelated. The test I added is passing: [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.381 s - in org.apache.hadoop.core.conf.TestUnmodifiableConfiguration > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Michael Miller >Priority: Major > Labels: Performance > Attachments: HADOOP-11223.001.patch, HADOOP-11223.002.patch, > HADOOP-11223.003.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765429#comment-16765429 ] Hadoop QA commented on HADOOP-11223: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 3s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m 44s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 57s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch generated 80 new + 0 unchanged - 0 fixed = 80 total (was 0) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 27s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 35s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 41s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}106m 45s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.ha.TestZKFailoverController | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HADOOP-11223 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12958301/HADOOP-11223.003.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 585a8c81c3e0 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct 31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / ca4e46a | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/15913/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt | | unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/15913/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/15913/testReport/ | | Max. process+thread count | 1380 (vs.
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765334#comment-16765334 ] Hadoop QA commented on HADOOP-11223: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 31s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 40s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m 2s{color} | {color:red} root in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 2s{color} | {color:red} root in the patch failed. {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 40s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch generated 79 new + 0 unchanged - 0 fixed = 79 total (was 0) {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 43s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m 34s{color} | {color:red} patch has errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 31s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 46s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 22s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 57m 44s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HADOOP-11223 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12958287/HADOOP-11223.002.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 3b7ff4e64023 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 0ceb1b7 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | mvninstall | https://builds.apache.org/job/PreCommit-HADOOP-Build/15912/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt | | compile | https://builds.apache.org/job/PreCommit-HADOOP-Build/15912/artifact/out/patch-compile-root.txt | | javac | https://builds.apache.org/job/PreCommit-HADOOP-Build/15912/artifact/out/patch-compile-root.txt | | checkstyle |
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765020#comment-16765020 ] Steve Loughran commented on HADOOP-11223: - you're down as a contributor so can assign issues & submit patches -attach the patch and hit the "submit patch" button for Yetus to trigger the test run > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Michael Miller >Priority: Major > Labels: Performance > Attachments: HADOOP-11223.001.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761893#comment-16761893 ] Michael Miller commented on HADOOP-11223: - [~ste...@apache.org] can you assign this ticket to me? I have a patch but am unable to attach it. > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Varun Saxena >Priority: Major > Labels: Performance > Attachments: HADOOP-11223.001.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756383#comment-16756383 ] Michael Miller commented on HADOOP-11223: - I updated my PR with the better name, UnmodifiableConfiguration: https://github.com/apache/hadoop/pull/473 > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Varun Saxena >Priority: Major > Labels: Performance > Attachments: HADOOP-11223.001.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755371#comment-16755371 ] Michael Miller commented on HADOOP-11223: - The name is wrong since it is not actually immutable. The class I created is closer to the Java unmodifiable collections. I will rename the class. I think an unmodifiable configuration would be good to have at least. A truly immutable configuration, I think would be more complicated and involve changes to Configuration. > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Varun Saxena >Priority: Major > Labels: Performance > Attachments: HADOOP-11223.001.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755268#comment-16755268 ] Steve Loughran commented on HADOOP-11223: - One aspect of config mutability is that if you load a resource through {{addDefaultResource}} it can force in new values underneath the iterators. For real immutability, that is going to have to be locked out. Which isn't a bad thing, it just needs to be addressed to stop someone making changes in something which is now viewed as immutable > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Varun Saxena >Priority: Major > Labels: Performance > Attachments: HADOOP-11223.001.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754347#comment-16754347 ] Michael Miller commented on HADOOP-11223: - I submitted a PR https://github.com/apache/hadoop/pull/473 > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Varun Saxena >Priority: Major > Labels: Performance > Attachments: HADOOP-11223.001.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752809#comment-16752809 ] Michael Miller commented on HADOOP-11223: - I wrote a class that extends Configuration called ImmutableConfiguration and a Test. Please let me know if this can be used anywhere or what branch I can make a pull request against. I have them in my local github [here|https://github.com/milleruntime/accumulo/blob/immutableConf/core/src/main/java/org/apache/accumulo/core/conf/ImmutableConfiguration.java]. > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Varun Saxena >Priority: Major > Labels: Performance > Attachments: HADOOP-11223.001.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752437#comment-16752437 ] Keith Turner commented on HADOOP-11223: --- It would be nice if Configuraiton had a static method like Collections.unmodifiableXXX. {code:java} Configuraiton conf = new Configuration(); //wrap conf with a class that throws exception on set Configuraiton readonlyConf = Configuration.umodifiable(conf); {code} > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Varun Saxena >Priority: Major > Labels: Performance > Attachments: HADOOP-11223.001.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16751720#comment-16751720 ] Hadoop QA commented on HADOOP-11223: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 44s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 48s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 52s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 13s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 18s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 38s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 98m 19s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HADOOP-11223 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12689236/HADOOP-11223.001.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux be3e9287311a 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 4e0aa2c | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/15841/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/15841/testReport/ | | Max. process+thread count | 1347 (vs. ulimit of 1) | | modules | C: hadoop-common-project/hadoop-common U:
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16751630#comment-16751630 ] Michael Miller commented on HADOOP-11223: - Did anything ever come from this ticket? I did work recently for Apache Accumulo that would greatly benefit from having a read only copy of Configuration. > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Varun Saxena >Priority: Major > Labels: Performance > Attachments: HADOOP-11223.001.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160774#comment-16160774 ] Hadoop QA commented on HADOOP-11223: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 19s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 32s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 29s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 31s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 22s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 55m 2s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | HADOOP-11223 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12689236/HADOOP-11223.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux ff2e07e045b6 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 722ee84 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/13227/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/13227/testReport/ | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/13227/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug >
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828293#comment-15828293 ] Jason Lowe commented on HADOOP-11223: - bq. If the real problem is reloading all those XML files all the time, why not change that behavior in Hadoop 3.x? At the very least, we could have some kind of mapping between classpath and default configuration values, and only actually load the XML files when we saw a new classpath which might cause us to load some different files. That's an interesting idea. Tackling the *-default.xml files would get us a long way since hopefully we can not only avoid parsing them for new Configuration objects but also avoid invalidating them in every existing Configuration object every time a new default resource is added. There'd still be the parsing of *-site.xml files which can also be expensive. We'd have to not only snapshot the classpath but also sizes and modification timestamps of the relevant resources located on that classpath if we wanted to apply a similar approach to those. > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Varun Saxena > Labels: Performance > Attachments: HADOOP-11223.001.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795603#comment-15795603 ] Hadoop QA commented on HADOOP-11223: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 11s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 27s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 10s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 28s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 41s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 34s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 51m 0s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:a9ad5d6 | | JIRA Issue | HADOOP-11223 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12689236/HADOOP-11223.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 4380b9fd7d69 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / b31e195 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/11344/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/11344/testReport/ | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/11344/console | | Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 >
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795391#comment-15795391 ] Colin P. McCabe commented on HADOOP-11223: -- If the real problem is reloading all those XML files all the time, why not change that behavior in Hadoop 3.x? At the very least, we could have some kind of mapping between classpath and default configuration values, and only actually load the XML files when we saw a new classpath which might cause us to load some different files. > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Varun Saxena > Labels: Performance > Attachments: HADOOP-11223.001.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452519#comment-15452519 ] Hadoop QA commented on HADOOP-11223: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 57s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 23s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 0s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 21s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 42m 43s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.net.TestDNS | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Issue | HADOOP-11223 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12689236/HADOOP-11223.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 885b3a04a883 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 20ae1fa | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/10427/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt | | unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/10427/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/10427/testReport/ | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/10427/console | | Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Offer a read-only conf
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452379#comment-15452379 ] Jason Lowe commented on HADOOP-11223: - I recently ran across this same issue with a user complaining about unnecessarily slow localization. It takes us at least 3 seconds to localize anything because that's the minimum startup time of the localizer program. Some of this slowness is from loading almost 4000 classes (!!) during localization and needing to lookup every known FileSystem before any FileSystem can be instantiated, but the largest problem is redundant processing in Configuration. The problem with copying a cached configuration object is the add default resource behavior. That invalidates every configuration object and causes each of them to reload the XML files _separately_. So if we have a cached object that never gets directly used but only has copies, we'll always copy a conf that hasn't loaded the XML and each copy will then load the XML the first time a property is accessed in them. Given that addDefaultResource is updating every instantiated conf object whether people like it or not, I don't see why we can't reuse a conf object for the straightforward cases like static code blocks, record readers, codec factories, etc. etc.. We can wrap the singleton instance with a subclass that throws UnsupportedOperationException on set methods. As for addDefaultResource IMHO we can block it or not, since even if we block it on the singleton other conf instances that have this called will update the others. And that's the way it used to work, so arguably we're preserving the old behavior by allowing addDefaultResource to update the shared singleton instance. The only downside I see once we lock down the set methods is that we theoretically can break some use-case where the XML files were updated on disk and the code really needed to see the fresh copy. However I don't think that will be the case for the instances within the Hadoop core code that will be updated to use this new singleton instance. Not sure if there are other scenarios that could break where one plays games with the classloader to switch which XML files will be seen. I'm assuming that if classloading games are being played then we're going to get multiple Configuration classes loaded each with their own singleton instance. Bonus points if we can eliminate the requirement that each conf object has to separately load all resources after addDefaultResource. It'd be nice if one conf object does the load and we poke it into the others like we poke the others when addDefaultResource was called. Also would be nice if we only loaded the newly added resource rather than reloading all resources. However that's really orthogonal to the shared, read-only instance idea and more appropriate for a separate JIRA. > Offer a read-only conf alternative to new Configuration() > - > > Key: HADOOP-11223 > URL: https://issues.apache.org/jira/browse/HADOOP-11223 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Gopal V >Assignee: Varun Saxena > Labels: Performance > Attachments: HADOOP-11223.001.patch > > > new Configuration() is called from several static blocks across Hadoop. > This is incredibly inefficient, since each one of those involves primarily > XML parsing at a point where the JIT won't be triggered & interpreter mode is > essentially forced on the JVM. > The alternate solution would be to offer a {{Configuration::getDefault()}} > alternative which disallows any modifications. > At the very least, such a method would need to be called from > # org.apache.hadoop.io.nativeio.NativeIO::() > # org.apache.hadoop.security.SecurityUtil::() > # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294482#comment-14294482 ] Colin Patrick McCabe commented on HADOOP-11223: --- bq. Gopal wrote: The primary blockage I had trying to implement this is the expected behaviour of addDefaultResource(). The overlays set programmatically is somewhat easier to disable cleanly. Yeah. {{addDefaultResource}} is certainly problematic. We could probably just have a static copy of the configuration for each component (hadoop common, hdfs, yarn, etc.), to get around the fact that different XML files will be parsed for each one. bq. Varun wrote: From the preceding discussion it seemed immutability was not to be considered. And programmers using the class will take care of it. I think if we've learned anything from Configuration.java, it's that the programmers will *not* take care of it :) bq. Can't we keep a simple flag in Configuration class to make it read only. DefaultConfiguration can probably extend Configuration. We can make this flag to be set only from DefaultConfiguration(in constructor) and disallow any operations which involve setting keys or loading resources, if this flag is set. I would rather not add a lot of complexity without benchmarks showing that it's needed. Copying the configuration object is simple and we know it will work. Plus, the real cost is reading the config files from the disk, not creating more Java objects on the heap. Offer a read-only conf alternative to new Configuration() - Key: HADOOP-11223 URL: https://issues.apache.org/jira/browse/HADOOP-11223 Project: Hadoop Common Issue Type: Bug Components: conf Reporter: Gopal V Assignee: Varun Saxena Labels: Performance Attachments: HADOOP-11223.001.patch new Configuration() is called from several static blocks across Hadoop. This is incredibly inefficient, since each one of those involves primarily XML parsing at a point where the JIT won't be triggered interpreter mode is essentially forced on the JVM. The alternate solution would be to offer a {{Configuration::getDefault()}} alternative which disallows any modifications. At the very least, such a method would need to be called from # org.apache.hadoop.io.nativeio.NativeIO::clinit() # org.apache.hadoop.security.SecurityUtil::clinit() # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider::clinit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292452#comment-14292452 ] Gopal V commented on HADOOP-11223: -- [~cmccabe]: Agree, the current patch doesn't fix anything. The primary blockage I had trying to implement this is the expected behaviour of addDefaultResource(). The overlays set programmatically is somewhat easier to disable cleanly. Offer a read-only conf alternative to new Configuration() - Key: HADOOP-11223 URL: https://issues.apache.org/jira/browse/HADOOP-11223 Project: Hadoop Common Issue Type: Bug Components: conf Reporter: Gopal V Assignee: Varun Saxena Labels: Performance Attachments: HADOOP-11223.001.patch new Configuration() is called from several static blocks across Hadoop. This is incredibly inefficient, since each one of those involves primarily XML parsing at a point where the JIT won't be triggered interpreter mode is essentially forced on the JVM. The alternate solution would be to offer a {{Configuration::getDefault()}} alternative which disallows any modifications. At the very least, such a method would need to be called from # org.apache.hadoop.io.nativeio.NativeIO::clinit() # org.apache.hadoop.security.SecurityUtil::clinit() # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider::clinit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292586#comment-14292586 ] Varun Saxena commented on HADOOP-11223: --- Thanks [~cmccabe] and [~gopalv] for your comments. From the preceding discussion it seemed immutability was not to be considered. And programmers using the class will take care of it. Can't we keep a simple flag in Configuration class to make it read only. DefaultConfiguration can probably extend Configuration. We can make this flag to be set only from DefaultConfiguration(in constructor) and disallow any operations which involve setting keys or loading resources, if this flag is set. Offer a read-only conf alternative to new Configuration() - Key: HADOOP-11223 URL: https://issues.apache.org/jira/browse/HADOOP-11223 Project: Hadoop Common Issue Type: Bug Components: conf Reporter: Gopal V Assignee: Varun Saxena Labels: Performance Attachments: HADOOP-11223.001.patch new Configuration() is called from several static blocks across Hadoop. This is incredibly inefficient, since each one of those involves primarily XML parsing at a point where the JIT won't be triggered interpreter mode is essentially forced on the JVM. The alternate solution would be to offer a {{Configuration::getDefault()}} alternative which disallows any modifications. At the very least, such a method would need to be called from # org.apache.hadoop.io.nativeio.NativeIO::clinit() # org.apache.hadoop.security.SecurityUtil::clinit() # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider::clinit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292619#comment-14292619 ] Varun Saxena commented on HADOOP-11223: --- bq. which could lead to race conditions during startup (we have some people setting configuration keys). Are there any pending thread safety issues ? Offer a read-only conf alternative to new Configuration() - Key: HADOOP-11223 URL: https://issues.apache.org/jira/browse/HADOOP-11223 Project: Hadoop Common Issue Type: Bug Components: conf Reporter: Gopal V Assignee: Varun Saxena Labels: Performance Attachments: HADOOP-11223.001.patch new Configuration() is called from several static blocks across Hadoop. This is incredibly inefficient, since each one of those involves primarily XML parsing at a point where the JIT won't be triggered interpreter mode is essentially forced on the JVM. The alternate solution would be to offer a {{Configuration::getDefault()}} alternative which disallows any modifications. At the very least, such a method would need to be called from # org.apache.hadoop.io.nativeio.NativeIO::clinit() # org.apache.hadoop.security.SecurityUtil::clinit() # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider::clinit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292417#comment-14292417 ] Colin Patrick McCabe commented on HADOOP-11223: --- I don't see this as something we can really commit at this point. We would need to see some examples of actually using this. (i.e. changing NativeIO.java, SecurityUtil, etc. to use it). The current patch also doesn't do anything to make the configuration read-only, which could lead to race conditions during startup (we have some people setting configuration keys). Given how many problems we've had with this kind of thing in the past, I think it's important to address this. Maybe we should just have this method hand out copies of the configuration that it loads. Yes, there will be some overhead from the Configuration object copy, but it will be much, much less than the overhead of parsing XML files from the disk. Offer a read-only conf alternative to new Configuration() - Key: HADOOP-11223 URL: https://issues.apache.org/jira/browse/HADOOP-11223 Project: Hadoop Common Issue Type: Bug Components: conf Reporter: Gopal V Assignee: Varun Saxena Labels: Performance Attachments: HADOOP-11223.001.patch new Configuration() is called from several static blocks across Hadoop. This is incredibly inefficient, since each one of those involves primarily XML parsing at a point where the JIT won't be triggered interpreter mode is essentially forced on the JVM. The alternate solution would be to offer a {{Configuration::getDefault()}} alternative which disallows any modifications. At the very least, such a method would need to be called from # org.apache.hadoop.io.nativeio.NativeIO::clinit() # org.apache.hadoop.security.SecurityUtil::clinit() # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider::clinit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14291073#comment-14291073 ] Varun Saxena commented on HADOOP-11223: --- Findbugs unrelated Offer a read-only conf alternative to new Configuration() - Key: HADOOP-11223 URL: https://issues.apache.org/jira/browse/HADOOP-11223 Project: Hadoop Common Issue Type: Bug Components: conf Reporter: Gopal V Assignee: Varun Saxena Labels: Performance Attachments: HADOOP-11223.001.patch new Configuration() is called from several static blocks across Hadoop. This is incredibly inefficient, since each one of those involves primarily XML parsing at a point where the JIT won't be triggered interpreter mode is essentially forced on the JVM. The alternate solution would be to offer a {{Configuration::getDefault()}} alternative which disallows any modifications. At the very least, such a method would need to be called from # org.apache.hadoop.io.nativeio.NativeIO::clinit() # org.apache.hadoop.security.SecurityUtil::clinit() # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider::clinit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14185364#comment-14185364 ] Colin Patrick McCabe commented on HADOOP-11223: --- Interesting idea. Just to clarify, it seems like there are two separate ideas here: 1. Some kind of read-only wrapper for Configuration, to make it impossible to modify a given configuration object. 2. Avoiding re-reading the XML files more than once. It seems like #2 could be accomplished pretty easily by having a DefaultConfiguration.java that just had a public static Configuration object. Then #1 is just a way of making sure that badly behaved programmers don't alter this default configuration when they're using it. Is that an accurate summary? +1 for both ideas from me. I often wish Configuration were an immutable class, perhaps with copy-on-write semantics. It would eliminate a lot of race conditions. Offer a read-only conf alternative to new Configuration() - Key: HADOOP-11223 URL: https://issues.apache.org/jira/browse/HADOOP-11223 Project: Hadoop Common Issue Type: Bug Components: conf Reporter: Gopal V Labels: Performance new Configuration() is called from several static blocks across Hadoop. This is incredibly inefficient, since each one of those involves primarily XML parsing at a point where the JIT won't be triggered interpreter mode is essentially forced on the JVM. The alternate solution would be to offer a {{Configuration::getDefault()}} alternative which disallows any modifications. At the very least, such a method would need to be called from # org.apache.hadoop.io.nativeio.NativeIO::clinit() # org.apache.hadoop.security.SecurityUtil::clinit() # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider::clinit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
[ https://issues.apache.org/jira/browse/HADOOP-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14185990#comment-14185990 ] Gopal V commented on HADOOP-11223: -- Yes, that's the whole idea. This is merely from a JVM startup point of view. I'm not sure it is possible to make it immutable at this point, since it is used both during daemon runtime and job configuration. In the former mode, Configuration has more intimate problems with multiple threads mutability, thanks to addDefaultResources and the internal registry. Offer a read-only conf alternative to new Configuration() - Key: HADOOP-11223 URL: https://issues.apache.org/jira/browse/HADOOP-11223 Project: Hadoop Common Issue Type: Bug Components: conf Reporter: Gopal V Labels: Performance new Configuration() is called from several static blocks across Hadoop. This is incredibly inefficient, since each one of those involves primarily XML parsing at a point where the JIT won't be triggered interpreter mode is essentially forced on the JVM. The alternate solution would be to offer a {{Configuration::getDefault()}} alternative which disallows any modifications. At the very least, such a method would need to be called from # org.apache.hadoop.io.nativeio.NativeIO::clinit() # org.apache.hadoop.security.SecurityUtil::clinit() # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider::clinit -- This message was sent by Atlassian JIRA (v6.3.4#6332)