[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455139#comment-15455139 ] Hadoop QA commented on HDFS-9847: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 57s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 8s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 38s{color} | {color:green} root: The patch generated 0 new + 1083 unchanged - 3 fixed = 1083 total (was 1086) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 8s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 48s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 13s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 34s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 28s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}150m 40s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.net.TestClusterTopology | | | hadoop.hdfs.server.namenode.ha.TestEditLogTailer | | | hadoop.hdfs.security.TestDelegationTokenForProxyUser | | | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport | | Timed out junit tests | org.apache.hadoop.http.TestHttpServerLifecycle | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Issue | HDFS-9847 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12826586/HDFS-9847.008.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux e48b09749534 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality |
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454185#comment-15454185 ] Hadoop QA commented on HDFS-9847: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 28s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 49s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 12s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 15s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 51s{color} | {color:orange} root: The patch generated 2 new + 1082 unchanged - 3 fixed = 1084 total (was 1085) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 46s{color} | {color:red} hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 30s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 59m 27s{color} | {color:green} hadoop-hdfs in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 26s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}126m 31s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-common-project/hadoop-common | | | Possible null pointer dereference of keyInfo in org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration$DeprecationContext, String) Dereferenced at Configuration.java:keyInfo in org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration$DeprecationContext, String) Dereferenced at Configuration.java:[line 636] | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Issue | HDFS-9847 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12826539/HDFS-9847.007.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux d7298ec90f6a
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454104#comment-15454104 ] Yiqun Lin commented on HDFS-9847: - I have checked the code, it looks good, pending jenkins. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847-nothrow.003.patch, > HDFS-9847-nothrow.004.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, HDFS-9847.007.patch, branch-2-delta.002.txt, > timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454026#comment-15454026 ] Yiqun Lin commented on HDFS-9847: - Thanks a lot for updating the patch, [~chris.douglas]! The latest patch looks good to me. I will make a quick check to see if the method {{conf.getTimeDuration}} of the related configs has been updated in the current code, since the nothrow.004 was based on the old codes. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847-nothrow.003.patch, > HDFS-9847-nothrow.004.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, HDFS-9847.007.patch, branch-2-delta.002.txt, > timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236421#comment-15236421 ] Lin Yiqun commented on HDFS-9847: - Hi, everyone, can we go ahead of this JIRA and do a final review and commit. It seems there are no large disagreement from the latest comments, thanks. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847-nothrow.003.patch, > HDFS-9847-nothrow.004.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221199#comment-15221199 ] Lin Yiqun commented on HDFS-9847: - Update the nothrow-004 patch for addressing comments as [~vinayrpet] suggested. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847-nothrow.003.patch, > HDFS-9847-nothrow.004.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220994#comment-15220994 ] Vinayakumar B commented on HDFS-9847: - bq. Use LOG_DEPRECATION.info() for the warnings. That's what it's there for. As per [~ste...@apache.org]'s suggestion, LOG_DEPRECATION should be used for the logs inside getTimeDuration() instead of LOG, I think [~chris.douglas] also agreed with this. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847-nothrow.003.patch, > HDFS-9847-nothrow.004.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220984#comment-15220984 ] Lin Yiqun commented on HDFS-9847: - Update the patch nothrow-004 with leaving the defaults in hdfs-default.xml. The code in hadoop main source and unit tests should be ok because it has tested in nothow-003 patch which was changing the defaults with time unit. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847-nothrow.003.patch, > HDFS-9847-nothrow.004.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220245#comment-15220245 ] Arpit Agarwal commented on HDFS-9847: - bq. We can just update hdfs-default.xml without changing the default values in v003 patch. Once hdfs-default.xml is updated all existing code that reads the affected parameters is broken and needs to be fixed (as the failed tests demonstrated). This potentially includes test or diagnostics code that is not in the Hadoop source tree. So it sounds like changing the defaults in hdfs-site.xml is incompatible in spirit, even if it does not violate our compatibility guidelines. It sounds like the right thing to do is to fix all the code in the Hadoop source tree to read values with time units (including tests) but leave the defaults as they are for now. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847-nothrow.003.patch, > HDFS-9847.001.patch, HDFS-9847.002.patch, HDFS-9847.003.patch, > HDFS-9847.004.patch, HDFS-9847.005.patch, HDFS-9847.006.patch, > branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219169#comment-15219169 ] Lin Yiqun commented on HDFS-9847: - Hi, [~chris.douglas], the latest v003 patch has fix these errors. The latest jenkins report's failed unit tests is not related to this JIRA. I also agree on this: {quote} IMO, we should go ahead without changing the default values and breaking tests in this Jira. {quote} We can just update hdfs-default.xml without changing the default values in v003 patch. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847-nothrow.003.patch, > HDFS-9847.001.patch, HDFS-9847.002.patch, HDFS-9847.003.patch, > HDFS-9847.004.patch, HDFS-9847.005.patch, HDFS-9847.006.patch, > branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218541#comment-15218541 ] Chris Douglas commented on HDFS-9847: - bq. IMO, we should go ahead without changing the default values and breaking tests in this Jira. Previous versions of the patch were not correct. The unit tests caught this, and trying to start a cluster using a config with units [confirmed|https://issues.apache.org/jira/browse/HDFS-9847?focusedCommentId=15217465=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15217465] it. The handful of tests that do not pass should not be difficult to fix. Please file the JIRA before commit, so deprecation can be tracked. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847-nothrow.003.patch, > HDFS-9847.001.patch, HDFS-9847.002.patch, HDFS-9847.003.patch, > HDFS-9847.004.patch, HDFS-9847.005.patch, HDFS-9847.006.patch, > branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217728#comment-15217728 ] Hadoop QA commented on HDFS-9847: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 49s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 42s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 49s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 45s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 17s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 39s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 6s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 20s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 15s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 0s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 56s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 56s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 48s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 48s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 17s {color} | {color:green} root: patch generated 0 new + 1002 unchanged - 2 fixed = 1002 total (was 1004) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 22s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 40s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 49s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 20s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 11s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 34s {color} | {color:red} hadoop-common in the patch failed with JDK v1.8.0_74. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 53s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK v1.8.0_74. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 56m 56s {color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 57s {color} | {color:red} hadoop-common in the patch failed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 59s {color} | {color:green} hadoop-hdfs-client in the patch passed with
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217546#comment-15217546 ] Vinayakumar B commented on HDFS-9847: - bq. So Chris Douglas, what can we do next for this jira? bq. The deprecation logic should be able to handle migrating variables, but this needs to wrap up. Minimally, all the changes to variables in the main source tree should understand the new type assigned to them here. Please file a followup ticket to complete the work in test classes deferred from this JIRA. IMO, we should go ahead without changing the default values and breaking tests in this Jira. In follow-up, May be we can consider deprecation of current variables, with deprecation logic and corresponding test fixes. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847-nothrow.003.patch, > HDFS-9847.001.patch, HDFS-9847.002.patch, HDFS-9847.003.patch, > HDFS-9847.004.patch, HDFS-9847.005.patch, HDFS-9847.006.patch, > branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217465#comment-15217465 ] Lin Yiqun commented on HDFS-9847: - {quote} Minimally, all the changes to variables in the main source tree should understand the new type assigned to them here. {quote} I have tried to change these variables and bring up the cluster in my local. There were actually some places has not updated to the way {{getTimeDuration}}. Updated a complete patch. I'm not sure the how many unit tests will failed, but it seems that most tests only use method {{setInt}} or {{setLong}} rather than {{getLong}}. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847-nothrow.003.patch, > HDFS-9847.001.patch, HDFS-9847.002.patch, HDFS-9847.003.patch, > HDFS-9847.004.patch, HDFS-9847.005.patch, HDFS-9847.006.patch, > branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217439#comment-15217439 ] Hadoop QA commented on HDFS-9847: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 35s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 48s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 53s {color} | {color:green} trunk passed with JDK v1.8.0_77 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 41s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 16s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 25s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 40s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 8s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 16s {color} | {color:green} trunk passed with JDK v1.8.0_77 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 10s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 57s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 41s {color} | {color:green} the patch passed with JDK v1.8.0_77 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 41s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 38s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 18s {color} | {color:green} root: patch generated 0 new + 1002 unchanged - 2 fixed = 1002 total (was 1004) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 40s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 56s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 16s {color} | {color:green} the patch passed with JDK v1.8.0_77 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 17s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 12s {color} | {color:red} hadoop-common in the patch failed with JDK v1.8.0_77. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 52s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK v1.8.0_77. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 22m 8s {color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 29s {color} | {color:red} hadoop-common in the patch failed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s {color} | {color:green} hadoop-hdfs-client in the patch passed with
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217326#comment-15217326 ] Lin Yiqun commented on HDFS-9847: - Thanks for [~ste...@apache.org] for comments. Update the latest patch v003 to address latest feedback. Two places were updated. # Log level from {{WARN}} to {{INFO}} when missing time unit. # Use StringUtils.toLowerCase() for internationalized string case conversion as Steve suggested. In addition, I am some confused for comments from [~chris.douglas]. Do you means that we should update the unit tests from {{getLong}} to {{getTimeDuration}} which are used these changed variables in anonther JIRA? All the changes to variables in the main source tree should be OK I think because it has been updated in this JIRA. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216629#comment-15216629 ] Steve Loughran commented on HDFS-9847: -- case conversion can surface whenever there's an "i" in the string, "minutes" does, though "m" doesn't. It's good to have it in so the entire codebase can be consistent, which is why it was all moved ove eralier. bq. Because it avoids the TimeUnit import? I disagree that this is useful or desirable, if we're going to bother with this API at all. OK, leave as is. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216396#comment-15216396 ] Chris Douglas commented on HDFS-9847: - bq. StringUtils.toLowerCase() for internationalized string case conversion For units? Is this a problem? bq. Use LOG_DEPRECATION.info() for the warnings. That's what it's there for. Is it? When the config var has been deprecated and replaced ({{blah.blah.seconds}} \-> {{blah.blah}}) that's correct, but this only changes the type. Someone suppressing deprecation warnings probably also wants to suppress these, though. Sure. bq. there's enough use of milliseconds out there, that a long getTimeDurationMillis() would seem useful Because it avoids the {{TimeUnit}} import? I disagree that this is useful or desirable, if we're going to bother with this API at all. bq. GenericTestUtils has test code for validating exceptions and rethrowing on failure. Without that: your assert fails but the stack trace of exception at fault is lost The latest patches do not throw on failure, but good to know this utility method. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15215814#comment-15215814 ] Steve Loughran commented on HDFS-9847: -- Alongside chris's comments, some basic code review # @ Use {{StringUtils.toLowerCase()}} for internationalized string case conversion # Use {{LOG_DEPRECATION.info()}} for the warnings. That's what it's there for. # there's enough use of milliseconds out there, that a {{long getTimeDurationMillis()}} would seem useful. This will also address the issue that some of the multiplications are by 1000L, some by 1000. # GenericTestUtils has test code for validating exceptions and rethrowing on failure. Without that: your assert fails *but the stack trace of exception at fault is lost* {code} try { timeDuration = conf.getTimeDuration("test.time.a", 10, SECONDS); fail("Expected exception, got " + timeDuration); } catch (IllegalArgumentException iae) { assertExceptionContains( "Loss of precision converting " + timeStr + " to " + SECONDS, iae); } {code} > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15215449#comment-15215449 ] Chris Douglas commented on HDFS-9847: - Have you tried to set units on these variables, then format and bring up a cluster? Unless the unit tests are using a separate path, the patch will break this basic functionality... The deprecation logic should be able to handle migrating variables, but this needs to wrap up. Minimally, all the changes to variables in the main source tree should understand the new type assigned to them here. Please file a followup ticket to complete the work in test classes deferred from this JIRA. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15215223#comment-15215223 ] Lin Yiqun commented on HDFS-9847: - So [~chris.douglas], what can we do next for this jira? > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15214540#comment-15214540 ] Chris Douglas commented on HDFS-9847: - bq. Are you okay with making this a INFO ? Sure. bq. Should we just keep the defaults as they are to keep the patch size sane? If it was backwards-compatible test code, then we could file a followup to fix it. But _it is not just test code_. Tests are failing in core classes: {noformat} java.lang.NumberFormatException: For input string: "3s" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Long.parseLong(Long.java:441) at java.lang.Long.parseLong(Long.java:483) at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1375) at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.(DatanodeManager.java:234) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.(BlockManager.java:329) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:738) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:676) {noformat} Even if this constructor is only invoked by test code, it's reasonable to expect that all occurrences have had their type changed in the main classes. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211397#comment-15211397 ] Hadoop QA commented on HDFS-9847: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 41s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 9s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 53s {color} | {color:green} branch-2 passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 35s {color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 17s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 9m 41s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 19s {color} | {color:green} branch-2 passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s {color} | {color:blue} Skipped branch modules with no Java source: . {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 36s {color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in branch-2 has 5 extant Findbugs warnings. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 57s {color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in branch-2 has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 34s {color} | {color:green} branch-2 passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m 41s {color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 9s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 14s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 56s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 56s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 18s {color} | {color:green} root: patch generated 0 new + 766 unchanged - 2 fixed = 766 total (was 768) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 9m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 40s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red} The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s {color} | {color:blue} Skipped patch modules with no Java source: . {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 55s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m 49s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 52m 21s {color} | {color:red} root in the patch failed with JDK v1.8.0_74. {color} | |
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211310#comment-15211310 ] Vinayakumar B commented on HDFS-9847: - {code}LOG.warn("No unit for " + name + "(" + timeStr + ") assuming " + unit);{code} [~arpitagarwal]/[~chris.douglas], Are you okay with making this a INFO ? At least for backward compatibility for the existing config keys, its not mandatory to specify unit in values. In that case, just mentioning about the taken value now would be fine. If its INFO, admins of existing deployments will not be alarmed. (atleast till configs are deprecated. Anyway, once the config is deprecated a WARNING log will come for them). > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211296#comment-15211296 ] Lin Yiqun commented on HDFS-9847: - When we want to change the default config values, we can update its unit tests after or we can create another jira for this. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211288#comment-15211288 ] Lin Yiqun commented on HDFS-9847: - {quote} But we can't change the type of a config variable without updating every instance where it's used. {quote} I also agree on HDFS-9847-nothrow.001.patch for trunk. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211227#comment-15211227 ] Chris Douglas commented on HDFS-9847: - bq. Should we just keep the defaults as they are to keep the patch size sane? Changing the defaults demonstrated that the patch doesn't work. If one were to configure these variables using units, the test results show that most everything would break. Since we've spent a lot of time discussing how to change {{Configuration::getTimeDuration}}, we can repurpose this JIRA to commit that consensus. But we can't change the type of a config variable without updating every instance where it's used. We can accept old values, but calls that set/get that var as a long are invalid. For all variables: {{blah.blah.seconds}}: deprecate, and add a {{blah.blah}} parameter using TimeUnit {{blah.blah}}: verify that it actually works when configured with units AFAICT, the deprecation logic in {{Configuration}} should return (and complain) about {{blah.blah.seconds}} if it's set, and also complain about the missing unit because it should return a long. bq. still think we should eliminate the warning for branch-2 to avoid overwhelming admins with benign warnings I disagree. The warning is not benign, it's informing admins that the type of the var has changed. If there is code that sets these parameters as longs, the deployment should complain when it reads so the config (and any related tooling) can be updated. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211010#comment-15211010 ] Arpit Agarwal commented on HDFS-9847: - Should we just keep the defaults as they are to keep the patch size sane? I'd be +1 on HDFS-9847-nothrow.001.patch for trunk but still think we should eliminate the warning for branch-2 to avoid overwhelming admins with benign warnings (attached a [proposed branch-2 patch here|https://issues.apache.org/jira/secure/attachment/12795307/HDFS-9847-branch-2.002.patch]). We could revisit the defaults in another Jira. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-branch-2.002.patch, HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, branch-2-delta.002.txt, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210982#comment-15210982 ] Hadoop QA commented on HDFS-9847: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 40s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 12s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 29s {color} | {color:green} branch-2 passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 16s {color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 17s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 22s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 44s {color} | {color:green} branch-2 passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 36s {color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in branch-2 has 5 extant Findbugs warnings. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 53s {color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in branch-2 has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 18s {color} | {color:green} branch-2 passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 16s {color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 58s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 16s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 13s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 13s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 14s {color} | {color:green} root: patch generated 0 new + 766 unchanged - 2 fixed = 766 total (was 768) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 39s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red} The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 53s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 19s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 16s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 51s {color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_74. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 49s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK v1.8.0_74. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 50s {color} | {color:red} hadoop-hdfs in the patch failed with JDK
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210853#comment-15210853 ] Chris Douglas commented on HDFS-9847: - The patch needs to change _all_ the uses of the config vars, including tests. Every access using {{getLong}} will fail, as demonstrated when units are actually set in the configuration. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-nothrow.001.patch, HDFS-9847-nothrow.002.patch, > HDFS-9847.001.patch, HDFS-9847.002.patch, HDFS-9847.003.patch, > HDFS-9847.004.patch, HDFS-9847.005.patch, HDFS-9847.006.patch, > timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210836#comment-15210836 ] Chris Douglas commented on HDFS-9847: - Well, that's not promising. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-branch-2.001.patch, > HDFS-9847-nothrow.001.patch, HDFS-9847-nothrow.002.patch, > HDFS-9847.001.patch, HDFS-9847.002.patch, HDFS-9847.003.patch, > HDFS-9847.004.patch, HDFS-9847.005.patch, HDFS-9847.006.patch, > timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210797#comment-15210797 ] Hadoop QA commented on HDFS-9847: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 35s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 49s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 35s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 16s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 18s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 41s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 6s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 16s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 10s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 57s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 45s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 45s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 59s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 59s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 17s {color} | {color:green} root: patch generated 0 new + 1003 unchanged - 2 fixed = 1003 total (was 1005) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 28s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 43s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 14s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 11s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 26s {color} | {color:red} hadoop-common in the patch failed with JDK v1.8.0_74. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 54s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK v1.8.0_74. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 25m 2s {color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 38s {color} | {color:red} hadoop-common in the patch failed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 4s {color} | {color:green} hadoop-hdfs-client in the patch passed with
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210568#comment-15210568 ] Arpit Agarwal commented on HDFS-9847: - +1 for HDFS-9847-nothrow.002.patch. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210542#comment-15210542 ] Chris Douglas commented on HDFS-9847: - Yeah, I agree it's not clear-cut. Attached a patch that includes units for all the vars changed to TimeUnit in {{hdfs-default.xml}}, so it's clearer that this patch changed their type (compatibly) in this release. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-nothrow.001.patch, > HDFS-9847-nothrow.002.patch, HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210509#comment-15210509 ] Arpit Agarwal commented on HDFS-9847: - bq. This is the WARN for missing units? It should warn, shouldn't it? The calls specifying TimeUnit.SECONDS are for backwards compatibility, and would otherwise be in millis. The changes to hdfs-default.xml should include units (changing their type) in this release, then we can change the caller after users have been warned for using values without units. Okay. it will be confusing for administrators to start seeing these warnings post-upgrade if they are not relying on the defaults in {{hdfs-default.xml}}. Not sure what is the right thing to do here, if you feel the warning is warranted then let's keep it. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-nothrow.001.patch, HDFS-9847.001.patch, > HDFS-9847.002.patch, HDFS-9847.003.patch, HDFS-9847.004.patch, > HDFS-9847.005.patch, HDFS-9847.006.patch, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210501#comment-15210501 ] Chris Douglas commented on HDFS-9847: - bq. Remove the warning on line 1651 in Configuration.java. Else I think this change will introduce warnings for all existing deployments. This is the WARN for missing units? It should warn, shouldn't it? The calls specifying TimeUnit.SECONDS are for backwards compatibility, and would otherwise be in millis. The changes to {{hdfs-default.xml}} should include units (changing their type) in this release, then we can change the caller after users have been warned for using values without units. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-nothrow.001.patch, HDFS-9847.001.patch, > HDFS-9847.002.patch, HDFS-9847.003.patch, HDFS-9847.004.patch, > HDFS-9847.005.patch, HDFS-9847.006.patch, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210299#comment-15210299 ] Arpit Agarwal commented on HDFS-9847: - There is one more improvement and I am sorry for not noticing it earlier. Remove the warning on line 1651 in Configuration.java. Else I think this change will introduce warnings for all existing deployments. +1 from me otherwise. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-nothrow.001.patch, HDFS-9847.001.patch, > HDFS-9847.002.patch, HDFS-9847.003.patch, HDFS-9847.004.patch, > HDFS-9847.005.patch, HDFS-9847.006.patch, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208299#comment-15208299 ] Hadoop QA commented on HDFS-9847: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 38s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 44s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 37s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 16s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 19s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 40s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 3s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 14s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 9s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 57s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 44s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 44s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 34s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 34s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 17s {color} | {color:green} root: patch generated 0 new + 1004 unchanged - 2 fixed = 1004 total (was 1006) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 40s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 47s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 15s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 14s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 39s {color} | {color:red} hadoop-common in the patch failed with JDK v1.8.0_74. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 49s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK v1.8.0_74. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 56m 4s {color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 1s {color} | {color:red} hadoop-common in the patch failed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 59s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207745#comment-15207745 ] Vinayakumar B commented on HDFS-9847: - +1. Latest patch looks good to me. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-nothrow.001.patch, HDFS-9847.001.patch, > HDFS-9847.002.patch, HDFS-9847.003.patch, HDFS-9847.004.patch, > HDFS-9847.005.patch, HDFS-9847.006.patch, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207712#comment-15207712 ] Lin Yiqun commented on HDFS-9847: - Update a new patch with nothrow way, and it will print log info when the value are lossing precision. Also I fixed the minor of lossing precision info in my patch. I tested in my local, print like this: {code} Loss of precision converting 7s to MINUTES for test.time.a {code} [~arpitagarwal], [~chris.douglas], I think this is a better way, what do you think, pengding jenkins. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847-nothrow.001.patch, HDFS-9847.001.patch, > HDFS-9847.002.patch, HDFS-9847.003.patch, HDFS-9847.004.patch, > HDFS-9847.005.patch, HDFS-9847.006.patch, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205674#comment-15205674 ] Vinayakumar B commented on HDFS-9847: - bq. Can we use nothrow exception way, but still print out the lossing precision log infos to let users know. This seems to be better. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205644#comment-15205644 ] Lin Yiqun commented on HDFS-9847: - {quote} Yes. Similar to the discussion on casting the result to int, the caller is responsible for precision. {quote} Can we use nothrow exception way, but still print out the lossing precision log infos to let users know. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205369#comment-15205369 ] Chris Douglas commented on HDFS-9847: - bq. Sorry I didn't understand your proposal to handle compatibility in the last para, agreed with everything before that. For params without units we warn, which is consistent with our current strategy for deprecating config knobs. If the impl increases precision, then nothing distinguishes throw/nothrow. If we decrease precision, the nothrow implementation can decide what to do: discard lost precision (default), check the old precision and warn, or throw; if {{getTimeDuration}} throws, then old configs will throw (default) unless the impl checks the old precision and warns, or discards. w.r.t. compatibility, the only difference is the default for decreasing precision. The caller is explicitly saying they want the unit in seconds/hours/days, and following the semantics of {{TimeUnit}} seems preferable to an error. I see your point about our existing cases: we can't switch from {{some.config.seconds}} to {{some.config}} with a default in millis by aliasing the former to the latter. We need to first warn that {{some.config.seconds}} is deprecated in a release, then drop it in the next. Where {{some.config}} has no units in the key, we can't convert the API to use {{getTimeDuration}} and increase precision in the same release. To be fair, this isn't less flexibility than we have now: we don't specify {{some.config\[.seconds\]}} as a float. The opacity of discarded precision creating the possibility for misattribution (e.g., "increasing the hb interval from 1s to 1500ms improved stability" when it's noise) is also a concern, but I'd argue that's not a failure of the API. bq. To continue the example of heartbeat interval do you mean we continue to query it with a unit of seconds for now and discard precision without throwing? Yes. Similar to the discussion on casting the result to int, the caller is responsible for precision. All that said, while I think nothrow is preferable, I won't insist on it and am +0 on the latest patch. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205306#comment-15205306 ] Arpit Agarwal commented on HDFS-9847: - Sorry I didn't understand your proposal to handle compatibility in the last para, agreed with everything before that. To continue the example of heartbeat interval do you mean we continue to query it with a unit of seconds for now and discard precision without throwing? > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205191#comment-15205191 ] Chris Douglas commented on HDFS-9847: - Sorry, I overloaded "precision". By providing another parameter, it might provide semantics like "give me the value of this parameter in this unit, unless it sheds more than (x% | x | whatever) of the configured value". If the caller cares this much, they should ask for the var with high precision and make the decision to throw in that context. bq. Without the precision check Configuration silently changes values without the caller knowing about it. Isn't that the intent behind configuring with units? Throwing for loss of precision requires the operator to know the default precision (no improvement over the current approach, which embeds the unit in the config key), or go through at least one round of errors. To make this less painful for users, the caller should only use {{getTimeDuration}} with high precision, which gets us back to specifying everything but the string value in millis, internally. If {{getTimeDuration}} throws, then why is it better? I'd argue handling backwards compatibility is easier without throwing for loss of precision. This warns for vars without units, which we've decided is sufficient for deprecated config vars. New versions can increase precision changing only the timeunit param, and old client configs will still work. If it wants to decrease precision, then the caller can warn/throw for values that lose precision (whatever's appropriate for its context), and migrate to the new unit in the next version. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205020#comment-15205020 ] Arpit Agarwal commented on HDFS-9847: - The precision check was my idea so I'll try to answer. bq. The Configuration has no context, and providing the expected precision as an argument overengineers the interface. {{getTimeDuration}} already takes a precision parameter so this patch didn't add to the interface. Without the precision check Configuration silently changes values without the caller knowing about it. e.g. this test passes today. {code} conf.set("xyz", "1500ms"); assertEquals(conf.getTimeDuration("xyz", 0, TimeUnit.SECONDS), 1); {code} The alternative is the caller always passes TimeUnit as microseconds (or nanoseconds?) But handling backwards compatibility is harder. How will the caller differentiate {{1us}} from plain {{1}} without looking at the raw value? bq. Why HadoopIllegalArgumentException instead of java.lang.IllegalArgumentException? Added by HADOOP-6537, looks like the intent was to differentiate IllegalArgumentException thrown by Hadoop vs the JDK. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15204945#comment-15204945 ] Chris Douglas commented on HDFS-9847: - The current patch changes {{Configuration::getTimeDuration}} to throw when losing precision: {{noformat}} // Configuration.java +long raw = Long.parseLong(timeStr); +long converted = unit.convert(raw, vUnit.unit()); +if (vUnit.unit().convert(converted, unit) != raw) { + throw new HadoopIllegalArgumentException("Loss of precision converting " + + timeStr + vUnit.suffix() + " to " + unit); } -return unit.convert(Long.parseLong(vStr), vUnit.unit()); +return converted; // TestConfiguration Configuration conf = new Configuration(false); conf.setTimeDuration("test.time.a", 7L, SECONDS); assertEquals("7s", conf.get("test.time.a")); -assertEquals(0L, conf.getTimeDuration("test.time.a", 30, MINUTES)); {{noformat}} This changes the contract. In the current version, the caller determines the precision (as detailed in the javadoc). This is correct; the caller knows reasonable precision, and can perform checks (e.g., unexpected value of 0) in its context. The {{Configuration}} has no context, and providing the expected precision as an argument overengineers the interface. If I want to give a heartbeat interval in microseconds that's my right as a lunatic, but the caller should not throw because it only cares about the value in seconds. (aside) Why {{HadoopIllegalArgumentException}} instead of {{java.lang.IllegalArgumentException}}? Not why is is thrown here, but why does it exist? > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15204613#comment-15204613 ] Arpit Agarwal commented on HDFS-9847: - Thanks [~linyiqun]. I am +1 for the latest patch with a minor issue - the following line should also include the name of the configuration setting. {code} 1660 throw new HadoopIllegalArgumentException("Loss of precision converting " 1661 + timeStr + vUnit.suffix() + " to " + unit); {code} I will hold off committing for now in case [~chris.douglas] or [~ste...@apache.org] have comments. Steve - I think the only open question was whether to add week/year suffixes. I don't have an opinion either way. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15203167#comment-15203167 ] Hadoop QA commented on HDFS-9847: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 1s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 46s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 11s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 25s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 47s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 30s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 39s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 40s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 23s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 57s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 58s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 37s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 42s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 42s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 13s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 13s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 43s {color} | {color:green} root: patch generated 0 new + 1005 unchanged - 2 fixed = 1005 total (was 1007) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 2s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 42s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 8s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 44s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 23s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 29s {color} | {color:red} hadoop-common in the patch failed with JDK v1.8.0_74. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 2s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK v1.8.0_74. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 33s {color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 19s {color} | {color:red} hadoop-common in the patch failed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 4s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15203088#comment-15203088 ] Lin Yiqun commented on HDFS-9847: - Thanks [~arpiagariu] for review. {quote} Up-converting a unit should be an error e.g. if heartbeat interval is specified as 500ms or 2500ms but we expect it to be seconds. {quote} Yes, this is actually a error. The supplied code is good to fix this. Update the latest patch for addressing three comments. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > HDFS-9847.006.patch, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197327#comment-15197327 ] Hadoop QA commented on HDFS-9847: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 42s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 58s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 2s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 53s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 20s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 33s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 42s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 36s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 49s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 32s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 10s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 48s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 48s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 36s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 36s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 16s {color} | {color:red} root: patch generated 6 new + 1014 unchanged - 2 fixed = 1020 total (was 1016) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 30s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 40s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 6s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 45s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 37s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 3s {color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_74. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 3s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK v1.8.0_74. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 114m 25s {color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 58s {color} | {color:red} hadoop-common in the patch failed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 38s {color} | {color:green} hadoop-hdfs-client in the patch passed
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199466#comment-15199466 ] Hadoop QA commented on HDFS-9847: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 35s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 39s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 0s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 35s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 17s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 58s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 18s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 7s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 11s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 53s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 37s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 37s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 30s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 30s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 20s {color} | {color:green} root: patch generated 0 new + 1005 unchanged - 2 fixed = 1005 total (was 1007) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 36s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 42s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 19s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 53s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 39s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 14s {color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_74. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 16s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK v1.8.0_74. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 37s {color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 20s {color} | {color:red} hadoop-common in the patch failed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 20s {color} | {color:green} hadoop-hdfs-client in the
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202559#comment-15202559 ] Arpit Agarwal commented on HDFS-9847: - Thank you for updating the patch [~linyiqun]. Few comments for parseTimeDuration. # parseTimeDuration should not default to zero. This condition can be hit only if defaultValue is null. Let's just throw HadoopIllegalArgumentException. {code} if (timeStr == null) { LOG.warn("No time value str for " + name + ", using default duration value 0 instead."); return 0; } {code} # Up-converting a unit should be an error e.g. if heartbeat interval is specified as 500ms or 2500ms but we expect it to be seconds. I haven't tested it but I think the following check at the end of parseTimeDuration will guard against loss of precision. {code} long raw = Long.parseLong(timeStr); long converted = unit.convert(raw, vUnit.unit()); if (vUnit.unit().convert(converted, unit) != raw) { throw new HadoopIllegalArgumentException("Loss of precision converting " + timeStr + " to " + vUnit); } return converted; {code} # We should add a unit test for (2). The rest of the patch lgtm. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, HDFS-9847.004.patch, HDFS-9847.005.patch, > timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15195658#comment-15195658 ] Arpit Agarwal commented on HDFS-9847: - Thank you [~linyiqun]. I don't think we need getLongTimeSeconds and getLongTimeMillis either. Callers can just use {{getTimeDuration}}. I also suggest adding a getTimeDuration overload that accepts defaultValue as String so defaults can be defined with units. e.g. {code} conf.getTimeDuration(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_DEFAULT, TimeUnit.SECONDS) // seconds for backwards compatibility. {code} where {{DFS_HEARTBEAT_INTERVAL_DEFAULT = "3s"}}. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch, HDFS-9847.002.patch, > HDFS-9847.003.patch, timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15195406#comment-15195406 ] Hadoop QA commented on HDFS-9847: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 49s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 3s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 44s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 21s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 40s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 8s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 19s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 15s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 57s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 57s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 57s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 44s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 44s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 16s {color} | {color:red} root: patch generated 5 new + 1014 unchanged - 1 fixed = 1019 total (was 1015) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 40s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 47s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 13s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 11s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 27s {color} | {color:red} hadoop-common in the patch failed with JDK v1.8.0_74. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 50s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK v1.8.0_74. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 3s {color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_74. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 13s {color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95. {color} | |
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15194511#comment-15194511 ] Arpit Agarwal commented on HDFS-9847: - [~linyiqun], do you want to update the patch to address Chris's feedback? bq. Adding new methods to parameterize getTimeDuration is unnecessary. The caller should cast the result if it's losing precision. We can punt the {{w}}/{{y}} decision to a separate Jira. Adding support for the remaining suffixes will be a step forward. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch, HDFS-9847.002.patch, > timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15170354#comment-15170354 ] Lin Yiqun commented on HDFS-9847: - HI, [~ste...@apache.org], can you please check for this? > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch, HDFS-9847.002.patch, > timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167516#comment-15167516 ] Arpit Agarwal commented on HDFS-9847: - Sounds good to me. Steve recommended the 'y' suffix for consistenty with Spark support so do check with him. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch, HDFS-9847.002.patch, > timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167028#comment-15167028 ] Lin Yiqun commented on HDFS-9847: - That's sound also right, [~chris.douglas]. What do you think, [~arpitagarwal], [~vinayrpet]. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch, HDFS-9847.002.patch, > timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15166981#comment-15166981 ] Chris Douglas commented on HDFS-9847: - bq. Adding week and year in hadoop-common is wanting to support to more differnet units in subclassing(HBase, Hive , etc)rather just HDFS. Is {{21d}} not sufficiently clear? What {{TimeUnit}} provides seems sufficient. bq. There are some places using timeSecondInterval variable in int type, so I define getIntTimeSeconds in Configuration. It nedd not cast type from long to int each time. Adding new methods to parameterize {{getTimeDuration}} is unnecessary. The caller should cast the result if it's losing precision. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch, HDFS-9847.002.patch, > timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15166598#comment-15166598 ] Lin Yiqun commented on HDFS-9847: - Hi, [~chris.douglas], it has case that can use time unit week {code} dfs.datanode.scan.period.hours 504 If this is positive, the DataNode will not scan any individual block more than once in the specified scan period. If this is negative, the block scanner is disabled. If this is set to zero, then the default value of 504 hours or 3 weeks is used. Prior versions of HDFS incorrectly documented that setting this key to zero will disable the block scanner. {code} Adding week and year in hadoop-common is wanting to support to more differnet units in subclassing(HBase, Hive , etc)rather just HDFS. There are some places using timeSecondInterval variable in {{int}} type, so I define {{getIntTimeSeconds}} in Configuration. It nedd not cast type from long to int each time. {code} int interval = conf.getIntTimeSeconds( DFSConfigKeys.DFS_DATANODE_DIRECTORYSCAN_INTERVAL_KEY, DFSConfigKeys.DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT); {code} > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch, HDFS-9847.002.patch, > timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163459#comment-15163459 ] Chris Douglas commented on HDFS-9847: - I'd hesitate to admit years as a unit. We won't handle leap years, though any timer configured at that granularity doesn't require high precision. Weeks will also be accurate, but do we have any use cases? Isn't it sufficient for those config values to be expressed in days? The convenience methods (e.g., {{getIntTimeSeconds}}) added to {{Configuration}} in 002 add to its API unnecessarily. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch, HDFS-9847.002.patch, > timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163231#comment-15163231 ] Hadoop QA commented on HDFS-9847: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 41s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 37s {color} | {color:green} trunk passed with JDK v1.8.0_72 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 54s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 22s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 50s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 43s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 51s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 51s {color} | {color:green} trunk passed with JDK v1.8.0_72 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 42s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 39s {color} | {color:green} the patch passed with JDK v1.8.0_72 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 54s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 54s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 23s {color} | {color:red} root: patch generated 10 new + 1033 unchanged - 1 fixed = 1043 total (was 1034) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 45s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 43s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 36s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 43s {color} | {color:green} the patch passed with JDK v1.8.0_72 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 44s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 31s {color} | {color:red} hadoop-common in the patch failed with JDK v1.8.0_72. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 9s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK v1.8.0_72. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 13s {color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_72. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 48s {color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 19s {color} | {color:green} hadoop-hdfs-client in the patch passed
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160335#comment-15160335 ] Lin Yiqun commented on HDFS-9847: - Thanks everyone for comments and [~vinayrpet]'s patch. I have used in my patch. The latest patch add the w(week) and y(year) in Configuration, and reuse the {{Configuration::getTimeDuration}} API. And besides, as [~ste...@apache.org] point out that there are many places will use milliseconds in testcases, I also design a similar method {{getLongTimeMillis}}. This method can be reused for parsing configuration which default time unit is {{MilliSecond}} in the future, thanks review! > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch, HDFS-9847.002.patch, > timeduration-w-y.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159852#comment-15159852 ] Arpit Agarwal commented on HDFS-9847: - Yes {{Configuration::getTimeDuration}} looks like it does exactly what we want. You may wish to augment it with the additional time units suggested by Steve for completeness. > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159431#comment-15159431 ] Chris Douglas commented on HDFS-9847: - Can't this use the {{Configuration::getTimeDuration}} API? > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159140#comment-15159140 ] Steve Loughran commented on HDFS-9847: -- I agree with the value of this # this is broader than just HDFS; it is changing Hadoop common and will be relevant for hadoop core, YARN, anything else subclassing configuration (Hive, HBase) # so we'd better support the range of units those apps may need, rather than just those of HDFS's current needs. Spark supports time in its configuration; here are some examples {pre} 25ms (milliseconds) 5s (seconds) 10m or 10min (minutes) 3h (hours) 5d (days) 1y (years) {code} I'd like to see the consistent support for the same units —and the ability to ask for time ranges as millis, rather than just seconds. Sometimes you do want minimum intervals measured in ms, not s, especially during testing. That is, something like getTimeMillis(key, defaultMs). > HDFS configuration without time unit name should accept friendly time units > --- > > Key: HDFS-9847 > URL: https://issues.apache.org/jira/browse/HDFS-9847 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Lin Yiqun >Assignee: Lin Yiqun > Attachments: HDFS-9847.001.patch > > > In HDFS-9821, it talks about the issue of leting existing keys use friendly > units e.g. 60s, 5m, 1d, 6w etc. But there are som configuration key names > contain time unit name, like {{dfs.blockreport.intervalMsec}}, so we can make > some other configurations which without time unit name to accept friendly > time units. The time unit {{seconds}} is frequently used in hdfs. We can > updating this configurations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9847) HDFS configuration without time unit name should accept friendly time units
[ https://issues.apache.org/jira/browse/HDFS-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158690#comment-15158690 ] Hadoop QA commented on HDFS-9847: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 21s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 11s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 11s {color} | {color:green} trunk passed with JDK v1.8.0_72 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 20s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 19s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 36s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 42s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 32s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 26s {color} | {color:green} trunk passed with JDK v1.8.0_72 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 20s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 4s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 24s {color} | {color:green} the patch passed with JDK v1.8.0_72 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 10s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 10s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 20s {color} | {color:red} root: patch generated 6 new + 1119 unchanged - 1 fixed = 1125 total (was 1120) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 36s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 41s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 22s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 30s {color} | {color:green} the patch passed with JDK v1.8.0_72 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 25s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 8s {color} | {color:red} hadoop-common in the patch failed with JDK v1.8.0_72. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s {color} | {color:green} hadoop-hdfs-client in the patch passed with JDK v1.8.0_72. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 47s {color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_72. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 45s {color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 1s {color} | {color:green} hadoop-hdfs-client in the patch passed with