[jira] [Updated] (HADOOP-14701) Configuration can log misleading warnings about an attempt to override final parameter
[ https://issues.apache.org/jira/browse/HADOOP-14701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-14701: - Resolution: Fixed Fix Version/s: 3.0.0-beta1 2.9.0 Status: Resolved (was: Patch Available) Thanks again for the contribution Andrew! I've committed this to trunk and branch-2. > Configuration can log misleading warnings about an attempt to override final > parameter > -- > > Key: HADOOP-14701 > URL: https://issues.apache.org/jira/browse/HADOOP-14701 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Andrew Sherman >Assignee: Andrew Sherman > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-14701.1.patch, HADOOP-14701.2.patch, > HADOOP-14701.3.patch, HADOOP-14701.4.patch > > > The class Configuration (in hadoop-common-project) is used to hold > configuration properties. > The method loadResources() is used to load properties from a list of default > resources into a parameter called “properties” (which happens to always be > the field called “properties”). > This is done by calling the method loadResource() which is used to load > properties from a single resource into a parameter called “properties” > (which happens to always be the field called “properties”). > This is done by calling loadProperty() which is used to load a single > property into a parameter called “properties” (which is sometimes, but not > always, the field called “properties”). > In loadProperty() there is some logic for dealing with properties that are > marked as final. > (1) a final property is not set into the “properties” parameter. > (2) a warning is printed for a final property if the value of the property > with the same name in the “properties” parameter is not equal to the new > value, and the property name is in the finalParameters field. > When loadResource() is reading properties from a stream it creates a > temporary Properties object that it populates by calling loadProperty(). This > is the case where loadProperty() has a parameter called “properties” which > is not the same as the field called “properties”). > When loadProperty() is called on a final property then because of (1) it is > not set into the “properties” parameter. But sometimes the property name is > already in the finalParameters field and now (2) the warning is triggered > because the property is not present in the “properties” parameter (although > it IS present in the “properties” field). The warning is misleading and > confusing. > h3. Possible Fix > One way of looking at the problem is that loadProperty is using the > finalParameters field in combination with the “properties” parameter. Change > the code to only generate a warning for a final property if the value of the > property with the same name in the “properties” field is not equal to the new > value, and the property name is in the finalParameters field. > I’ve made a patch containing this tiny change. > Note that I have done very minimal testing of this fix > h3. Example warning: > 2017-07-25 12:14:24,639 WARN org.apache.hadoop.conf.Configuration: > [HiveServer2-Background-Pool: Thread-77]: > hdfs://asherman-cdh6-subset-parcel-1.gce.cloudera.com:8020/user/hive/.staging/job_1501008923167_0001/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14701) Configuration can log misleading warnings about an attempt to override final parameter
[ https://issues.apache.org/jira/browse/HADOOP-14701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HADOOP-14701: Attachment: HADOOP-14701.4.patch > Configuration can log misleading warnings about an attempt to override final > parameter > -- > > Key: HADOOP-14701 > URL: https://issues.apache.org/jira/browse/HADOOP-14701 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Andrew Sherman > Attachments: HADOOP-14701.1.patch, HADOOP-14701.2.patch, > HADOOP-14701.3.patch, HADOOP-14701.4.patch > > > The class Configuration (in hadoop-common-project) is used to hold > configuration properties. > The method loadResources() is used to load properties from a list of default > resources into a parameter called “properties” (which happens to always be > the field called “properties”). > This is done by calling the method loadResource() which is used to load > properties from a single resource into a parameter called “properties” > (which happens to always be the field called “properties”). > This is done by calling loadProperty() which is used to load a single > property into a parameter called “properties” (which is sometimes, but not > always, the field called “properties”). > In loadProperty() there is some logic for dealing with properties that are > marked as final. > (1) a final property is not set into the “properties” parameter. > (2) a warning is printed for a final property if the value of the property > with the same name in the “properties” parameter is not equal to the new > value, and the property name is in the finalParameters field. > When loadResource() is reading properties from a stream it creates a > temporary Properties object that it populates by calling loadProperty(). This > is the case where loadProperty() has a parameter called “properties” which > is not the same as the field called “properties”). > When loadProperty() is called on a final property then because of (1) it is > not set into the “properties” parameter. But sometimes the property name is > already in the finalParameters field and now (2) the warning is triggered > because the property is not present in the “properties” parameter (although > it IS present in the “properties” field). The warning is misleading and > confusing. > h3. Possible Fix > One way of looking at the problem is that loadProperty is using the > finalParameters field in combination with the “properties” parameter. Change > the code to only generate a warning for a final property if the value of the > property with the same name in the “properties” field is not equal to the new > value, and the property name is in the finalParameters field. > I’ve made a patch containing this tiny change. > Note that I have done very minimal testing of this fix > h3. Example warning: > 2017-07-25 12:14:24,639 WARN org.apache.hadoop.conf.Configuration: > [HiveServer2-Background-Pool: Thread-77]: > hdfs://asherman-cdh6-subset-parcel-1.gce.cloudera.com:8020/user/hive/.staging/job_1501008923167_0001/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14701) Configuration can log misleading warnings about an attempt to override final parameter
[ https://issues.apache.org/jira/browse/HADOOP-14701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HADOOP-14701: Attachment: HADOOP-14701.3.patch > Configuration can log misleading warnings about an attempt to override final > parameter > -- > > Key: HADOOP-14701 > URL: https://issues.apache.org/jira/browse/HADOOP-14701 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Andrew Sherman > Attachments: HADOOP-14701.1.patch, HADOOP-14701.2.patch, > HADOOP-14701.3.patch > > > The class Configuration (in hadoop-common-project) is used to hold > configuration properties. > The method loadResources() is used to load properties from a list of default > resources into a parameter called “properties” (which happens to always be > the field called “properties”). > This is done by calling the method loadResource() which is used to load > properties from a single resource into a parameter called “properties” > (which happens to always be the field called “properties”). > This is done by calling loadProperty() which is used to load a single > property into a parameter called “properties” (which is sometimes, but not > always, the field called “properties”). > In loadProperty() there is some logic for dealing with properties that are > marked as final. > (1) a final property is not set into the “properties” parameter. > (2) a warning is printed for a final property if the value of the property > with the same name in the “properties” parameter is not equal to the new > value, and the property name is in the finalParameters field. > When loadResource() is reading properties from a stream it creates a > temporary Properties object that it populates by calling loadProperty(). This > is the case where loadProperty() has a parameter called “properties” which > is not the same as the field called “properties”). > When loadProperty() is called on a final property then because of (1) it is > not set into the “properties” parameter. But sometimes the property name is > already in the finalParameters field and now (2) the warning is triggered > because the property is not present in the “properties” parameter (although > it IS present in the “properties” field). The warning is misleading and > confusing. > h3. Possible Fix > One way of looking at the problem is that loadProperty is using the > finalParameters field in combination with the “properties” parameter. Change > the code to only generate a warning for a final property if the value of the > property with the same name in the “properties” field is not equal to the new > value, and the property name is in the finalParameters field. > I’ve made a patch containing this tiny change. > Note that I have done very minimal testing of this fix > h3. Example warning: > 2017-07-25 12:14:24,639 WARN org.apache.hadoop.conf.Configuration: > [HiveServer2-Background-Pool: Thread-77]: > hdfs://asherman-cdh6-subset-parcel-1.gce.cloudera.com:8020/user/hive/.staging/job_1501008923167_0001/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14701) Configuration can log misleading warnings about an attempt to override final parameter
[ https://issues.apache.org/jira/browse/HADOOP-14701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HADOOP-14701: Attachment: HADOOP-14701.2.patch > Configuration can log misleading warnings about an attempt to override final > parameter > -- > > Key: HADOOP-14701 > URL: https://issues.apache.org/jira/browse/HADOOP-14701 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Andrew Sherman > Attachments: HADOOP-14701.1.patch, HADOOP-14701.2.patch > > > The class Configuration (in hadoop-common-project) is used to hold > configuration properties. > The method loadResources() is used to load properties from a list of default > resources into a parameter called “properties” (which happens to always be > the field called “properties”). > This is done by calling the method loadResource() which is used to load > properties from a single resource into a parameter called “properties” > (which happens to always be the field called “properties”). > This is done by calling loadProperty() which is used to load a single > property into a parameter called “properties” (which is sometimes, but not > always, the field called “properties”). > In loadProperty() there is some logic for dealing with properties that are > marked as final. > (1) a final property is not set into the “properties” parameter. > (2) a warning is printed for a final property if the value of the property > with the same name in the “properties” parameter is not equal to the new > value, and the property name is in the finalParameters field. > When loadResource() is reading properties from a stream it creates a > temporary Properties object that it populates by calling loadProperty(). This > is the case where loadProperty() has a parameter called “properties” which > is not the same as the field called “properties”). > When loadProperty() is called on a final property then because of (1) it is > not set into the “properties” parameter. But sometimes the property name is > already in the finalParameters field and now (2) the warning is triggered > because the property is not present in the “properties” parameter (although > it IS present in the “properties” field). The warning is misleading and > confusing. > h3. Possible Fix > One way of looking at the problem is that loadProperty is using the > finalParameters field in combination with the “properties” parameter. Change > the code to only generate a warning for a final property if the value of the > property with the same name in the “properties” field is not equal to the new > value, and the property name is in the finalParameters field. > I’ve made a patch containing this tiny change. > Note that I have done very minimal testing of this fix > h3. Example warning: > 2017-07-25 12:14:24,639 WARN org.apache.hadoop.conf.Configuration: > [HiveServer2-Background-Pool: Thread-77]: > hdfs://asherman-cdh6-subset-parcel-1.gce.cloudera.com:8020/user/hive/.staging/job_1501008923167_0001/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14701) Configuration can log misleading warnings about an attempt to override final parameter
[ https://issues.apache.org/jira/browse/HADOOP-14701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HADOOP-14701: Status: Patch Available (was: Open) > Configuration can log misleading warnings about an attempt to override final > parameter > -- > > Key: HADOOP-14701 > URL: https://issues.apache.org/jira/browse/HADOOP-14701 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Andrew Sherman > Attachments: HADOOP-14701.1.patch > > > The class Configuration (in hadoop-common-project) is used to hold > configuration properties. > The method loadResources() is used to load properties from a list of default > resources into a parameter called “properties” (which happens to always be > the field called “properties”). > This is done by calling the method loadResource() which is used to load > properties from a single resource into a parameter called “properties” > (which happens to always be the field called “properties”). > This is done by calling loadProperty() which is used to load a single > property into a parameter called “properties” (which is sometimes, but not > always, the field called “properties”). > In loadProperty() there is some logic for dealing with properties that are > marked as final. > (1) a final property is not set into the “properties” parameter. > (2) a warning is printed for a final property if the value of the property > with the same name in the “properties” parameter is not equal to the new > value, and the property name is in the finalParameters field. > When loadResource() is reading properties from a stream it creates a > temporary Properties object that it populates by calling loadProperty(). This > is the case where loadProperty() has a parameter called “properties” which > is not the same as the field called “properties”). > When loadProperty() is called on a final property then because of (1) it is > not set into the “properties” parameter. But sometimes the property name is > already in the finalParameters field and now (2) the warning is triggered > because the property is not present in the “properties” parameter (although > it IS present in the “properties” field). The warning is misleading and > confusing. > h3. Possible Fix > One way of looking at the problem is that loadProperty is using the > finalParameters field in combination with the “properties” parameter. Change > the code to only generate a warning for a final property if the value of the > property with the same name in the “properties” field is not equal to the new > value, and the property name is in the finalParameters field. > I’ve made a patch containing this tiny change. > Note that I have done very minimal testing of this fix > h3. Example warning: > 2017-07-25 12:14:24,639 WARN org.apache.hadoop.conf.Configuration: > [HiveServer2-Background-Pool: Thread-77]: > hdfs://asherman-cdh6-subset-parcel-1.gce.cloudera.com:8020/user/hive/.staging/job_1501008923167_0001/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14701) Configuration can log misleading warnings about an attempt to override final parameter
[ https://issues.apache.org/jira/browse/HADOOP-14701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HADOOP-14701: Attachment: HADOOP-14701.1.patch > Configuration can log misleading warnings about an attempt to override final > parameter > -- > > Key: HADOOP-14701 > URL: https://issues.apache.org/jira/browse/HADOOP-14701 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Andrew Sherman > Attachments: HADOOP-14701.1.patch > > > The class Configuration (in hadoop-common-project) is used to hold > configuration properties. > The method loadResources() is used to load properties from a list of default > resources into a parameter called “properties” (which happens to always be > the field called “properties”). > This is done by calling the method loadResource() which is used to load > properties from a single resource into a parameter called “properties” > (which happens to always be the field called “properties”). > This is done by calling loadProperty() which is used to load a single > property into a parameter called “properties” (which is sometimes, but not > always, the field called “properties”). > In loadProperty() there is some logic for dealing with properties that are > marked as final. > (1) a final property is not set into the “properties” parameter. > (2) a warning is printed for a final property if the value of the property > with the same name in the “properties” parameter is not equal to the new > value, and the property name is in the finalParameters field. > When loadResource() is reading properties from a stream it creates a > temporary Properties object that it populates by calling loadProperty(). This > is the case where loadProperty() has a parameter called “properties” which > is not the same as the field called “properties”). > When loadProperty() is called on a final property then because of (1) it is > not set into the “properties” parameter. But sometimes the property name is > already in the finalParameters field and now (2) the warning is triggered > because the property is not present in the “properties” parameter (although > it IS present in the “properties” field). The warning is misleading and > confusing. > h3. Possible Fix > One way of looking at the problem is that loadProperty is using the > finalParameters field in combination with the “properties” parameter. Change > the code to only generate a warning for a final property if the value of the > property with the same name in the “properties” field is not equal to the new > value, and the property name is in the finalParameters field. > I’ve made a patch containing this tiny change. > Note that I have done very minimal testing of this fix > h3. Example warning: > 2017-07-25 12:14:24,639 WARN org.apache.hadoop.conf.Configuration: > [HiveServer2-Background-Pool: Thread-77]: > hdfs://asherman-cdh6-subset-parcel-1.gce.cloudera.com:8020/user/hive/.staging/job_1501008923167_0001/job.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
