[jira] [Created] (HADOOP-13536) Clean up code
Elliott Clark created HADOOP-13536: -- Summary: Clean up code Key: HADOOP-13536 URL: https://issues.apache.org/jira/browse/HADOOP-13536 Project: Hadoop Common Issue Type: Sub-task Reporter: Elliott Clark Some code comments came in while discussing merging. We should clean all those up and get everything that's known done. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11890) Uber-JIRA: Hadoop should support IPv6
[ https://issues.apache.org/jira/browse/HADOOP-11890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431628#comment-15431628 ] Elliott Clark commented on HADOOP-11890: Let me get a patch up with those comments. Thanks bq.Where is HADOOP_ALLOW_IPV6 documented? Where do you think would be the best place to document it ? > Uber-JIRA: Hadoop should support IPv6 > - > > Key: HADOOP-11890 > URL: https://issues.apache.org/jira/browse/HADOOP-11890 > Project: Hadoop Common > Issue Type: Improvement > Components: net >Reporter: Nate Edel >Assignee: Nate Edel > Labels: ipv6 > > Hadoop currently treats IPv6 as unsupported. Track related smaller issues to > support IPv6. > (Current case here is mainly HBase on HDFS, so any suggestions about other > test cases/workload are really appreciated.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13521) SampleQuantile does not perform well under load
[ https://issues.apache.org/jira/browse/HADOOP-13521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428907#comment-15428907 ] Elliott Clark edited comment on HADOOP-13521 at 8/19/16 9:56 PM: - Also related: HBASE-15222 was (Author: eclark): See HBASE-15222 > SampleQuantile does not perform well under load > --- > > Key: HADOOP-13521 > URL: https://issues.apache.org/jira/browse/HADOOP-13521 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 2.3.0 >Reporter: Mark Wagner > Attachments: Screen Shot 2015-07-09 at 12.37.29 PM.png > > > After adding quantile collection to one of our clusters we saw much higher > latency for RPCs. This was traced down to the quantile collection. Samples > are being buffered and inserted in groups of 500. After the buffered samples > are inserted, the entire set of samples for this time period (600 seconds at > the longest for us) is "compressed". > All operations for RPC metrics are synchronized. Usually this isn't an issue > but it seems that this compression operation is taking a significant amount > of time. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13521) SampleQuantile does not perform well under load
[ https://issues.apache.org/jira/browse/HADOOP-13521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428907#comment-15428907 ] Elliott Clark commented on HADOOP-13521: See HBASE-15222 > SampleQuantile does not perform well under load > --- > > Key: HADOOP-13521 > URL: https://issues.apache.org/jira/browse/HADOOP-13521 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 2.3.0 >Reporter: Mark Wagner > Attachments: Screen Shot 2015-07-09 at 12.37.29 PM.png > > > After adding quantile collection to one of our clusters we saw much higher > latency for RPCs. This was traced down to the quantile collection. Samples > are being buffered and inserted in groups of 500. After the buffered samples > are inserted, the entire set of samples for this time period (600 seconds at > the longest for us) is "compressed". > All operations for RPC metrics are synchronized. Usually this isn't an issue > but it seems that this compression operation is taking a significant amount > of time. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11890) Uber-JIRA: Hadoop should support IPv6
[ https://issues.apache.org/jira/browse/HADOOP-11890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427086#comment-15427086 ] Elliott Clark commented on HADOOP-11890: Cool I'll start the discussion on the mailing list then. > Uber-JIRA: Hadoop should support IPv6 > - > > Key: HADOOP-11890 > URL: https://issues.apache.org/jira/browse/HADOOP-11890 > Project: Hadoop Common > Issue Type: Improvement > Components: net >Reporter: Nate Edel >Assignee: Nate Edel > Labels: ipv6 > > Hadoop currently treats IPv6 as unsupported. Track related smaller issues to > support IPv6. > (Current case here is mainly HBase on HDFS, so any suggestions about other > test cases/workload are really appreciated.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11890) Uber-JIRA: Hadoop should support IPv6
[ https://issues.apache.org/jira/browse/HADOOP-11890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406699#comment-15406699 ] Elliott Clark commented on HADOOP-11890: Should we start a merge vote on this? Or what's needed before the community would feel comfortable with the merge? Facebook is running this in production and on test clusters. We have tested MR, HDFS, and HBase extensively on IPv6 clusters, and ipv4 clusters. > Uber-JIRA: Hadoop should support IPv6 > - > > Key: HADOOP-11890 > URL: https://issues.apache.org/jira/browse/HADOOP-11890 > Project: Hadoop Common > Issue Type: Improvement > Components: net >Reporter: Nate Edel >Assignee: Nate Edel > Labels: ipv6 > > Hadoop currently treats IPv6 as unsupported. Track related smaller issues to > support IPv6. > (Current case here is mainly HBase on HDFS, so any suggestions about other > test cases/workload are really appreciated.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12975) Add jitter to CachingGetSpaceUsed's thread
[ https://issues.apache.org/jira/browse/HADOOP-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339947#comment-15339947 ] Elliott Clark commented on HADOOP-12975: Hahah thanks! > Add jitter to CachingGetSpaceUsed's thread > -- > > Key: HADOOP-12975 > URL: https://issues.apache.org/jira/browse/HADOOP-12975 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.8.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.8.0 > > Attachments: HADOOP-12975v0.patch, HADOOP-12975v1.patch, > HADOOP-12975v2.patch, HADOOP-12975v3.patch, HADOOP-12975v4.patch, > HADOOP-12975v5.patch, HADOOP-12975v6.patch > > > Running DU across lots of disks is very expensive and running all of the > processes at the same time creates a noticeable IO spike. We should add some > jitter. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12975) Add jitter to CachingGetSpaceUsed's thread
[ https://issues.apache.org/jira/browse/HADOOP-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334843#comment-15334843 ] Elliott Clark commented on HADOOP-12975: Ping? > Add jitter to CachingGetSpaceUsed's thread > -- > > Key: HADOOP-12975 > URL: https://issues.apache.org/jira/browse/HADOOP-12975 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12975v0.patch, HADOOP-12975v1.patch, > HADOOP-12975v2.patch, HADOOP-12975v3.patch, HADOOP-12975v4.patch, > HADOOP-12975v5.patch, HADOOP-12975v6.patch > > > Running DU across lots of disks is very expensive and running all of the > processes at the same time creates a noticeable IO spike. We should add some > jitter. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12974) Create a CachingGetSpaceUsed implementation that uses df
[ https://issues.apache.org/jira/browse/HADOOP-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334844#comment-15334844 ] Elliott Clark commented on HADOOP-12974: Ping? > Create a CachingGetSpaceUsed implementation that uses df > > > Key: HADOOP-12974 > URL: https://issues.apache.org/jira/browse/HADOOP-12974 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12974v0.patch, HADOOP-12974v1.patch, > HADOOP-12974v2.patch, HADOOP-12974v3.patch, HADOOP-12974v4.patch, > HADOOP-12974v5.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12421) Add jitter to RetryInvocationHandler
[ https://issues.apache.org/jira/browse/HADOOP-12421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12421: --- Resolution: Not A Problem Status: Resolved (was: Patch Available) > Add jitter to RetryInvocationHandler > > > Key: HADOOP-12421 > URL: https://issues.apache.org/jira/browse/HADOOP-12421 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12421-v1.patch, HADOOP-12421-v2.patch, > HADOOP-12421-v3.patch, HADOOP-12421-v4.patch, HADOOP-12421-v5.patch > > > Calls to NN can become synchronized across a cluster during NN failover. This > leads to a spike in requests until things recover. Making an already tricky > time worse. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12974) Create a CachingGetSpaceUsed implementation that uses df
[ https://issues.apache.org/jira/browse/HADOOP-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304817#comment-15304817 ] Elliott Clark commented on HADOOP-12974: Tests are un-related. > Create a CachingGetSpaceUsed implementation that uses df > > > Key: HADOOP-12974 > URL: https://issues.apache.org/jira/browse/HADOOP-12974 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12974v0.patch, HADOOP-12974v1.patch, > HADOOP-12974v2.patch, HADOOP-12974v3.patch, HADOOP-12974v4.patch, > HADOOP-12974v5.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12974) Create a CachingGetSpaceUsed implementation that uses df
[ https://issues.apache.org/jira/browse/HADOOP-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12974: --- Attachment: HADOOP-12974v5.patch > Create a CachingGetSpaceUsed implementation that uses df > > > Key: HADOOP-12974 > URL: https://issues.apache.org/jira/browse/HADOOP-12974 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12974v0.patch, HADOOP-12974v1.patch, > HADOOP-12974v2.patch, HADOOP-12974v3.patch, HADOOP-12974v4.patch, > HADOOP-12974v5.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12974) Create a CachingGetSpaceUsed implementation that uses df
[ https://issues.apache.org/jira/browse/HADOOP-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12974: --- Attachment: HADOOP-12974v4.patch > Create a CachingGetSpaceUsed implementation that uses df > > > Key: HADOOP-12974 > URL: https://issues.apache.org/jira/browse/HADOOP-12974 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12974v0.patch, HADOOP-12974v1.patch, > HADOOP-12974v2.patch, HADOOP-12974v3.patch, HADOOP-12974v4.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12974) Create a CachingGetSpaceUsed implementation that uses df
[ https://issues.apache.org/jira/browse/HADOOP-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304540#comment-15304540 ] Elliott Clark commented on HADOOP-12974: bq.I don't think you need to typecast the instance object. Only caching instances have a stop. > Create a CachingGetSpaceUsed implementation that uses df > > > Key: HADOOP-12974 > URL: https://issues.apache.org/jira/browse/HADOOP-12974 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12974v0.patch, HADOOP-12974v1.patch, > HADOOP-12974v2.patch, HADOOP-12974v3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12975) Add jitter to CachingGetSpaceUsed's thread
[ https://issues.apache.org/jira/browse/HADOOP-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281661#comment-15281661 ] Elliott Clark commented on HADOOP-12975: bq.Can we throw an exception We could but most of the tests do not have a conf. I don't think there's any harm in falling back. Right now getKlass does the same. > Add jitter to CachingGetSpaceUsed's thread > -- > > Key: HADOOP-12975 > URL: https://issues.apache.org/jira/browse/HADOOP-12975 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12975v0.patch, HADOOP-12975v1.patch, > HADOOP-12975v2.patch, HADOOP-12975v3.patch, HADOOP-12975v4.patch, > HADOOP-12975v5.patch, HADOOP-12975v6.patch > > > Running DU across lots of disks is very expensive and running all of the > processes at the same time creates a noticeable IO spike. We should add some > jitter. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12975) Add jitter to CachingGetSpaceUsed's thread
[ https://issues.apache.org/jira/browse/HADOOP-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12975: --- Attachment: HADOOP-12975v6.patch > Add jitter to CachingGetSpaceUsed's thread > -- > > Key: HADOOP-12975 > URL: https://issues.apache.org/jira/browse/HADOOP-12975 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12975v0.patch, HADOOP-12975v1.patch, > HADOOP-12975v2.patch, HADOOP-12975v3.patch, HADOOP-12975v4.patch, > HADOOP-12975v5.patch, HADOOP-12975v6.patch > > > Running DU across lots of disks is very expensive and running all of the > processes at the same time creates a noticeable IO spike. We should add some > jitter. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12975) Add jitter to CachingGetSpaceUsed's thread
[ https://issues.apache.org/jira/browse/HADOOP-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281648#comment-15281648 ] Elliott Clark commented on HADOOP-12975: Yeah that's a typo. > Add jitter to CachingGetSpaceUsed's thread > -- > > Key: HADOOP-12975 > URL: https://issues.apache.org/jira/browse/HADOOP-12975 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12975v0.patch, HADOOP-12975v1.patch, > HADOOP-12975v2.patch, HADOOP-12975v3.patch, HADOOP-12975v4.patch, > HADOOP-12975v5.patch > > > Running DU across lots of disks is very expensive and running all of the > processes at the same time creates a noticeable IO spike. We should add some > jitter. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12975) Add jitter to CachingGetSpaceUsed's thread
[ https://issues.apache.org/jira/browse/HADOOP-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12975: --- Attachment: HADOOP-12975v5.patch Fixed the checkstyle warning. > Add jitter to CachingGetSpaceUsed's thread > -- > > Key: HADOOP-12975 > URL: https://issues.apache.org/jira/browse/HADOOP-12975 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12975v0.patch, HADOOP-12975v1.patch, > HADOOP-12975v2.patch, HADOOP-12975v3.patch, HADOOP-12975v4.patch, > HADOOP-12975v5.patch > > > Running DU across lots of disks is very expensive and running all of the > processes at the same time creates a noticeable IO spike. We should add some > jitter. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12975) Add jitter to CachingGetSpaceUsed's thread
[ https://issues.apache.org/jira/browse/HADOOP-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12975: --- Attachment: HADOOP-12975v4.patch > Add jitter to CachingGetSpaceUsed's thread > -- > > Key: HADOOP-12975 > URL: https://issues.apache.org/jira/browse/HADOOP-12975 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12975v0.patch, HADOOP-12975v1.patch, > HADOOP-12975v2.patch, HADOOP-12975v3.patch, HADOOP-12975v4.patch > > > Running DU across lots of disks is very expensive and running all of the > processes at the same time creates a noticeable IO spike. We should add some > jitter. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12975) Add jitter to CachingGetSpaceUsed's thread
[ https://issues.apache.org/jira/browse/HADOOP-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15274654#comment-15274654 ] Elliott Clark commented on HADOOP-12975: Probably I hadn't pulled in a while. > Add jitter to CachingGetSpaceUsed's thread > -- > > Key: HADOOP-12975 > URL: https://issues.apache.org/jira/browse/HADOOP-12975 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12975v0.patch, HADOOP-12975v1.patch, > HADOOP-12975v2.patch, HADOOP-12975v3.patch > > > Running DU across lots of disks is very expensive and running all of the > processes at the same time creates a noticeable IO spike. We should add some > jitter. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12975) Add jitter to CachingGetSpaceUsed's thread
[ https://issues.apache.org/jira/browse/HADOOP-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12975: --- Attachment: HADOOP-12975v3.patch * Added a default jitter config that's a set time. > Add jitter to CachingGetSpaceUsed's thread > -- > > Key: HADOOP-12975 > URL: https://issues.apache.org/jira/browse/HADOOP-12975 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12975v0.patch, HADOOP-12975v1.patch, > HADOOP-12975v2.patch, HADOOP-12975v3.patch > > > Running DU across lots of disks is very expensive and running all of the > processes at the same time creates a noticeable IO spike. We should add some > jitter. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12974) Create a CachingGetSpaceUsed implementation that uses df
[ https://issues.apache.org/jira/browse/HADOOP-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15259010#comment-15259010 ] Elliott Clark commented on HADOOP-12974: ping ? > Create a CachingGetSpaceUsed implementation that uses df > > > Key: HADOOP-12974 > URL: https://issues.apache.org/jira/browse/HADOOP-12974 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12974v0.patch, HADOOP-12974v1.patch, > HADOOP-12974v2.patch, HADOOP-12974v3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12974) Create a CachingGetSpaceUsed implementation that uses df
[ https://issues.apache.org/jira/browse/HADOOP-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12974: --- Attachment: HADOOP-12974v3.patch Fixed the documentation to correctly describe how to use the class. > Create a CachingGetSpaceUsed implementation that uses df > > > Key: HADOOP-12974 > URL: https://issues.apache.org/jira/browse/HADOOP-12974 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12974v0.patch, HADOOP-12974v1.patch, > HADOOP-12974v2.patch, HADOOP-12974v3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12975) Add jitter to CachingGetSpaceUsed's thread
[ https://issues.apache.org/jira/browse/HADOOP-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12975: --- Affects Version/s: 2.9.0 Status: Patch Available (was: Open) > Add jitter to CachingGetSpaceUsed's thread > -- > > Key: HADOOP-12975 > URL: https://issues.apache.org/jira/browse/HADOOP-12975 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12975v0.patch, HADOOP-12975v1.patch, > HADOOP-12975v2.patch > > > Running DU across lots of disks is very expensive and running all of the > processes at the same time creates a noticeable IO spike. We should add some > jitter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12975) Add jitter to CachingGetSpaceUsed's thread
[ https://issues.apache.org/jira/browse/HADOOP-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12975: --- Attachment: HADOOP-12975v2.patch > Add jitter to CachingGetSpaceUsed's thread > -- > > Key: HADOOP-12975 > URL: https://issues.apache.org/jira/browse/HADOOP-12975 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12975v0.patch, HADOOP-12975v1.patch, > HADOOP-12975v2.patch > > > Running DU across lots of disks is very expensive and running all of the > processes at the same time creates a noticeable IO spike. We should add some > jitter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12975) Add jitter to CachingGetSpaceUsed's thread
[ https://issues.apache.org/jira/browse/HADOOP-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12975: --- Summary: Add jitter to CachingGetSpaceUsed's thread (was: Add jitter to DU's thread) > Add jitter to CachingGetSpaceUsed's thread > -- > > Key: HADOOP-12975 > URL: https://issues.apache.org/jira/browse/HADOOP-12975 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12975v0.patch, HADOOP-12975v1.patch > > > Running DU across lots of disks is very expensive and running all of the > processes at the same time creates a noticeable IO spike. We should add some > jitter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12975) Add jitter to DU's thread
[ https://issues.apache.org/jira/browse/HADOOP-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241536#comment-15241536 ] Elliott Clark commented on HADOOP-12975: bq. What motivates choosing 50% of the full refresh period as the jitter default? 50% is probably too high, I'll change it to 15%. But a percentage is chosen as it makes the jitter scale with anyone who changes du periods. If it's a set number then someone with a refresh period of days won't get any benefit from the jitter. > Add jitter to DU's thread > - > > Key: HADOOP-12975 > URL: https://issues.apache.org/jira/browse/HADOOP-12975 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12975v0.patch, HADOOP-12975v1.patch > > > Running DU across lots of disks is very expensive and running all of the > processes at the same time creates a noticeable IO spike. We should add some > jitter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12974) Create a CachingGetSpaceUsed implementation that uses df
[ https://issues.apache.org/jira/browse/HADOOP-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12974: --- Attachment: HADOOP-12974v2.patch > Create a CachingGetSpaceUsed implementation that uses df > > > Key: HADOOP-12974 > URL: https://issues.apache.org/jira/browse/HADOOP-12974 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12974v0.patch, HADOOP-12974v1.patch, > HADOOP-12974v2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12974) Create a CachingGetSpaceUsed implementation that uses df
[ https://issues.apache.org/jira/browse/HADOOP-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12974: --- Affects Version/s: 2.9.0 Status: Patch Available (was: Open) > Create a CachingGetSpaceUsed implementation that uses df > > > Key: HADOOP-12974 > URL: https://issues.apache.org/jira/browse/HADOOP-12974 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.9.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12974v0.patch, HADOOP-12974v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12974) Create a CachingGetSpaceUsed implementation that uses df
[ https://issues.apache.org/jira/browse/HADOOP-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12974: --- Attachment: HADOOP-12974v1.patch Patch that creates a CachingGetSpaceUsed implementation that uses DF rather than DU to get the used space. I still need to get some more documentation and a test on this. > Create a CachingGetSpaceUsed implementation that uses df > > > Key: HADOOP-12974 > URL: https://issues.apache.org/jira/browse/HADOOP-12974 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12974v0.patch, HADOOP-12974v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12974) Create a CachingGetSpaceUsed implementation that uses df
[ https://issues.apache.org/jira/browse/HADOOP-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12974: --- Summary: Create a CachingGetSpaceUsed implementation that uses df (was: Create a DU implementation that uses df) > Create a CachingGetSpaceUsed implementation that uses df > > > Key: HADOOP-12974 > URL: https://issues.apache.org/jira/browse/HADOOP-12974 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12974v0.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239875#comment-15239875 ] Elliott Clark commented on HADOOP-12973: Thanks for all the reviews [~cmccabe] > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.8.0 > > Attachments: HADOOP-12973v0.patch, HADOOP-12973v1.patch, > HADOOP-12973v10.patch, HADOOP-12973v11.patch, HADOOP-12973v12.patch, > HADOOP-12973v13.patch, HADOOP-12973v2.patch, HADOOP-12973v3.patch, > HADOOP-12973v5.patch, HADOOP-12973v6.patch, HADOOP-12973v7.patch, > HADOOP-12973v8.patch, HADOOP-12973v9.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12973) make DU pluggable
[
https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15238106#comment-15238106
]
Elliott Clark commented on HADOOP-12973:
All test failures seem to be caused by:
{code}
Caused by: ENOENT: No such file or directory
{code}
> make DU pluggable
> -
>
> Key: HADOOP-12973
> URL: https://issues.apache.org/jira/browse/HADOOP-12973
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HADOOP-12973v0.patch, HADOOP-12973v1.patch,
> HADOOP-12973v10.patch, HADOOP-12973v11.patch, HADOOP-12973v12.patch,
> HADOOP-12973v13.patch, HADOOP-12973v2.patch, HADOOP-12973v3.patch,
> HADOOP-12973v5.patch, HADOOP-12973v6.patch, HADOOP-12973v7.patch,
> HADOOP-12973v8.patch, HADOOP-12973v9.patch
>
>
> If people are concerned about replacing the call to DU. Then an easy first
> step is to make it pluggable. Then it's possible to replace it with something
> while leaving the default alone.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12973: --- Attachment: HADOOP-12973v13.patch I had missed the init call in tests. > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12973v0.patch, HADOOP-12973v1.patch, > HADOOP-12973v10.patch, HADOOP-12973v11.patch, HADOOP-12973v12.patch, > HADOOP-12973v13.patch, HADOOP-12973v2.patch, HADOOP-12973v3.patch, > HADOOP-12973v5.patch, HADOOP-12973v6.patch, HADOOP-12973v7.patch, > HADOOP-12973v8.patch, HADOOP-12973v9.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12973: --- Attachment: HADOOP-12973v12.patch * Created CachingGetUsedSpace that does all the threading. * Used IOUtils#cleanup > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12973v0.patch, HADOOP-12973v1.patch, > HADOOP-12973v10.patch, HADOOP-12973v11.patch, HADOOP-12973v12.patch, > HADOOP-12973v2.patch, HADOOP-12973v3.patch, HADOOP-12973v5.patch, > HADOOP-12973v6.patch, HADOOP-12973v7.patch, HADOOP-12973v8.patch, > HADOOP-12973v9.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12973: --- Attachment: HADOOP-12973v11.patch * Moved most of the DU implementation into GetSpaceUsed so that WindowsGetSpaceUsed and the forth coming DFGetSpacedUsed can share all of the implementation. * Created a builder class. * Added more tests > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12973v0.patch, HADOOP-12973v1.patch, > HADOOP-12973v10.patch, HADOOP-12973v11.patch, HADOOP-12973v2.patch, > HADOOP-12973v3.patch, HADOOP-12973v5.patch, HADOOP-12973v6.patch, > HADOOP-12973v7.patch, HADOOP-12973v8.patch, HADOOP-12973v9.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15231286#comment-15231286 ] Elliott Clark commented on HADOOP-12973: bq.but I don't want the WindowsDU to be a subclass of the Linux DU Yeah I can see that. Closable seems good. bq.This can all be avoided by just passing the Builder object to the constructor. Ok sounds good. Let me get that out. > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12973v0.patch, HADOOP-12973v1.patch, > HADOOP-12973v10.patch, HADOOP-12973v2.patch, HADOOP-12973v3.patch, > HADOOP-12973v5.patch, HADOOP-12973v6.patch, HADOOP-12973v7.patch, > HADOOP-12973v8.patch, HADOOP-12973v9.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12973: --- Attachment: HADOOP-12973v10.patch > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12973v0.patch, HADOOP-12973v1.patch, > HADOOP-12973v10.patch, HADOOP-12973v2.patch, HADOOP-12973v3.patch, > HADOOP-12973v5.patch, HADOOP-12973v6.patch, HADOOP-12973v7.patch, > HADOOP-12973v8.patch, HADOOP-12973v9.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15229725#comment-15229725 ] Elliott Clark commented on HADOOP-12973: bq.There's no benefit to changing this class from non-static to static, if it needs to keep a reference to the parent DU object anyway, right? It makes it more obvious when someone overrides the class where things are. bq.There is a non-obvious coupling between DU and WindowsDU, since one is a subclass of the other. It keeps people changing DU from having to support both windows and linux. bq.Any MapReduce code that is using DU directly will now break on Windows, unless it is modified to explicitly use WindowsDU when appropriate This is limited private and there are no other places that I haven't changed. bq.can we make this a builder instead? I don't see the added complexity/ bq.Do we need both decDfsUsed and incDfsUsed? Can't we just add a negative? Or is that too unintuitive? Sure. bq.Is GetSpaceUsed#start really necessary? Shutdown is needed. So it's very strange to have a shutdown without a start. > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12973v0.patch, HADOOP-12973v1.patch, > HADOOP-12973v2.patch, HADOOP-12973v3.patch, HADOOP-12973v5.patch, > HADOOP-12973v6.patch, HADOOP-12973v7.patch, HADOOP-12973v8.patch, > HADOOP-12973v9.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12973: --- Attachment: HADOOP-12973v9.patch Here's a patch that adds on a factory and an interface. > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12973v0.patch, HADOOP-12973v1.patch, > HADOOP-12973v2.patch, HADOOP-12973v3.patch, HADOOP-12973v5.patch, > HADOOP-12973v6.patch, HADOOP-12973v7.patch, HADOOP-12973v8.patch, > HADOOP-12973v9.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12421) Add jitter to RetryInvocationHandler
[ https://issues.apache.org/jira/browse/HADOOP-12421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224855#comment-15224855 ] Elliott Clark commented on HADOOP-12421: Ping? > Add jitter to RetryInvocationHandler > > > Key: HADOOP-12421 > URL: https://issues.apache.org/jira/browse/HADOOP-12421 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12421-v1.patch, HADOOP-12421-v2.patch, > HADOOP-12421-v3.patch, HADOOP-12421-v4.patch, HADOOP-12421-v5.patch > > > Calls to NN can become synchronized across a cluster during NN failover. This > leads to a spike in requests until things recover. Making an already tricky > time worse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224594#comment-15224594 ] Elliott Clark commented on HADOOP-12973: Ping? It would be nice to get this in so that the other two jira's that depend on this can move forward. > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12973v0.patch, HADOOP-12973v1.patch, > HADOOP-12973v2.patch, HADOOP-12973v3.patch, HADOOP-12973v5.patch, > HADOOP-12973v6.patch, HADOOP-12973v7.patch, HADOOP-12973v8.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12973: --- Attachment: HADOOP-12973v8.patch With the annotations on WindwosDU > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12973v0.patch, HADOOP-12973v1.patch, > HADOOP-12973v2.patch, HADOOP-12973v3.patch, HADOOP-12973v5.patch, > HADOOP-12973v6.patch, HADOOP-12973v7.patch, HADOOP-12973v8.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12973: --- Attachment: HADOOP-12973v7.patch * Remove exception re-throwing. * Add logging when we fail to create the asked for class. * Clean up handling of InterruptedException > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12973v0.patch, HADOOP-12973v1.patch, > HADOOP-12973v2.patch, HADOOP-12973v3.patch, HADOOP-12973v5.patch, > HADOOP-12973v6.patch, HADOOP-12973v7.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218747#comment-15218747 ] Elliott Clark commented on HADOOP-12973: bq.Hi Elliott Clark I wonder what was the concern about replacing the call to DU? My understanding is that replacing du with df means that we will lose the ability to tell how large the distributed cache is. So for people who rely on MR/Yarn a lot then it's possible to get into a bad situation. I haven't really looked into that too much. My primary use case is around HBase where a du running and walking all the inodes causes a significant IO spike and latency outliers. bq.With regard to the issues described in HDFS-9923, do you think that your solution also fixes the inconsistent exception handling of DU? I didn't change the exception handling at all, but it is weird and changing it to just log a warn seems like a good idea to me. > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12973v0.patch, HADOOP-12973v1.patch, > HADOOP-12973v2.patch, HADOOP-12973v3.patch, HADOOP-12973v5.patch, > HADOOP-12973v6.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12973: --- Attachment: HADOOP-12973v6.patch > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12973v0.patch, HADOOP-12973v1.patch, > HADOOP-12973v2.patch, HADOOP-12973v3.patch, HADOOP-12973v5.patch, > HADOOP-12973v6.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218676#comment-15218676 ] Elliott Clark commented on HADOOP-12973: Added a test and cleaned up more checkstyle. > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12973v0.patch, HADOOP-12973v1.patch, > HADOOP-12973v2.patch, HADOOP-12973v3.patch, HADOOP-12973v5.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12973: --- Attachment: HADOOP-12973v5.patch > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12973v0.patch, HADOOP-12973v1.patch, > HADOOP-12973v2.patch, HADOOP-12973v3.patch, HADOOP-12973v5.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12975) Add jitter to DU's thread
[ https://issues.apache.org/jira/browse/HADOOP-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12975: --- Attachment: HADOOP-12975v1.patch Rebased patch. > Add jitter to DU's thread > - > > Key: HADOOP-12975 > URL: https://issues.apache.org/jira/browse/HADOOP-12975 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12975v0.patch, HADOOP-12975v1.patch > > > Running DU across lots of disks is very expensive and running all of the > processes at the same time creates a noticeable IO spike. We should add some > jitter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12973: --- Attachment: HADOOP-12973v3.patch Getting checkstyle better. > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12973v0.patch, HADOOP-12973v1.patch, > HADOOP-12973v2.patch, HADOOP-12973v3.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12975) Add jitter to DU's thread
[ https://issues.apache.org/jira/browse/HADOOP-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12975: --- Attachment: HADOOP-12975v0.patch Also based upon HADOOP-12973 > Add jitter to DU's thread > - > > Key: HADOOP-12975 > URL: https://issues.apache.org/jira/browse/HADOOP-12975 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12975v0.patch > > > Running DU across lots of disks is very expensive and running all of the > processes at the same time creates a noticeable IO spike. We should add some > jitter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12975) Add jitter to DU's thread
Elliott Clark created HADOOP-12975: -- Summary: Add jitter to DU's thread Key: HADOOP-12975 URL: https://issues.apache.org/jira/browse/HADOOP-12975 Project: Hadoop Common Issue Type: Sub-task Reporter: Elliott Clark Assignee: Elliott Clark Running DU across lots of disks is very expensive and running all of the processes at the same time creates a noticeable IO spike. We should add some jitter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HADOOP-12974) Create a DU implementation that uses df
[ https://issues.apache.org/jira/browse/HADOOP-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark reassigned HADOOP-12974: -- Assignee: Elliott Clark > Create a DU implementation that uses df > --- > > Key: HADOOP-12974 > URL: https://issues.apache.org/jira/browse/HADOOP-12974 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12974v0.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12974) Create a DU implementation that uses df
[ https://issues.apache.org/jira/browse/HADOOP-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217013#comment-15217013 ] Elliott Clark commented on HADOOP-12974: This patch is based upon the patch in HADOOP-12973 > Create a DU implementation that uses df > --- > > Key: HADOOP-12974 > URL: https://issues.apache.org/jira/browse/HADOOP-12974 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark > Attachments: HADOOP-12974v0.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12974) Create a DU implementation that uses df
[ https://issues.apache.org/jira/browse/HADOOP-12974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12974: --- Attachment: HADOOP-12974v0.patch > Create a DU implementation that uses df > --- > > Key: HADOOP-12974 > URL: https://issues.apache.org/jira/browse/HADOOP-12974 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark > Attachments: HADOOP-12974v0.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12973: --- Attachment: HADOOP-12973v2.patch > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12973v0.patch, HADOOP-12973v1.patch, > HADOOP-12973v2.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12974) Create a DU implementation that uses df
Elliott Clark created HADOOP-12974: -- Summary: Create a DU implementation that uses df Key: HADOOP-12974 URL: https://issues.apache.org/jira/browse/HADOOP-12974 Project: Hadoop Common Issue Type: Sub-task Reporter: Elliott Clark -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12973: --- Attachment: HADOOP-12973v1.patch > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12973v0.patch, HADOOP-12973v1.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12973: --- Status: Patch Available (was: Open) > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12973v0.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12973: --- Attachment: HADOOP-12973v0.patch > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12973v0.patch > > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HADOOP-12973) make DU pluggable
[ https://issues.apache.org/jira/browse/HADOOP-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark reassigned HADOOP-12973: -- Assignee: Elliott Clark > make DU pluggable > - > > Key: HADOOP-12973 > URL: https://issues.apache.org/jira/browse/HADOOP-12973 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > > If people are concerned about replacing the call to DU. Then an easy first > step is to make it pluggable. Then it's possible to replace it with something > while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12973) make DU pluggable
Elliott Clark created HADOOP-12973: -- Summary: make DU pluggable Key: HADOOP-12973 URL: https://issues.apache.org/jira/browse/HADOOP-12973 Project: Hadoop Common Issue Type: Sub-task Reporter: Elliott Clark If people are concerned about replacing the call to DU. Then an easy first step is to make it pluggable. Then it's possible to replace it with something while leaving the default alone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-3619) DNS.getHosts triggers an ArrayIndexOutOfBoundsException in reverseDNS if one of the interfaces is IPv6
[ https://issues.apache.org/jira/browse/HADOOP-3619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-3619: -- Resolution: Not A Problem Status: Resolved (was: Patch Available) > DNS.getHosts triggers an ArrayIndexOutOfBoundsException in reverseDNS if one > of the interfaces is IPv6 > -- > > Key: HADOOP-3619 > URL: https://issues.apache.org/jira/browse/HADOOP-3619 > Project: Hadoop Common > Issue Type: Sub-task > Components: net >Reporter: Steve Loughran >Assignee: Dr. Martin Menzel > Labels: BB2015-05-TBR, ipv6, patch > Attachments: HADOOP-3619-v2.patch > > > reverseDNS tries to split a host address string by ".", and so fails if ":" > is the separator, as it is in IPv6. When it tries to access the parts of the > address, a stack trace is seen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12695) Fix libexec scripts that unconditionally add on preferIPv4
Elliott Clark created HADOOP-12695: -- Summary: Fix libexec scripts that unconditionally add on preferIPv4 Key: HADOOP-12695 URL: https://issues.apache.org/jira/browse/HADOOP-12695 Project: Hadoop Common Issue Type: Sub-task Reporter: Elliott Clark Assignee: Elliott Clark There are other places that add on the prefer ipv4 flag. They aren't needed since the flag is added conditionally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12695) Fix libexec scripts that unconditionally add on preferIPv4
[ https://issues.apache.org/jira/browse/HADOOP-12695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12695: --- Attachment: 0001-HADOOP-12695-Fix-libexec-scripts-that-unconditionall.patch This patch is only needed if running ipv6 on a branch-2 based build. > Fix libexec scripts that unconditionally add on preferIPv4 > -- > > Key: HADOOP-12695 > URL: https://issues.apache.org/jira/browse/HADOOP-12695 > Project: Hadoop Common > Issue Type: Sub-task > Components: net >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: > 0001-HADOOP-12695-Fix-libexec-scripts-that-unconditionall.patch > > > There are other places that add on the prefer ipv4 flag. They aren't needed > since the flag is added conditionally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-12695) Fix libexec scripts that unconditionally add on preferIPv4
[ https://issues.apache.org/jira/browse/HADOOP-12695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark resolved HADOOP-12695. Resolution: Fixed Closing as this isn't needed on the branch that's being built. However if this gets back ported then be sure to pick this patch up. > Fix libexec scripts that unconditionally add on preferIPv4 > -- > > Key: HADOOP-12695 > URL: https://issues.apache.org/jira/browse/HADOOP-12695 > Project: Hadoop Common > Issue Type: Sub-task > Components: net >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: > 0001-HADOOP-12695-Fix-libexec-scripts-that-unconditionall.patch > > > There are other places that add on the prefer ipv4 flag. They aren't needed > since the flag is added conditionally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12485) Don't prefer IPV4 stack on tests.
[ https://issues.apache.org/jira/browse/HADOOP-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12485: --- Resolution: Fixed Status: Resolved (was: Patch Available) Got the non-binding +1 from [~mantonov] Pushed to branch > Don't prefer IPV4 stack on tests. > - > > Key: HADOOP-12485 > URL: https://issues.apache.org/jira/browse/HADOOP-12485 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Affects Versions: HADOOP-11890 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12485-HADOOP-11890.0.patch, > HADOOP-12485-HADOOP-11890.1.patch > > > Don't prefer ipv4 stack and instead use ipv6 if it's there. This should mean > that tests actually bind on ipv6 if it's there. > It should mean that tests running against this branch are more likely to be > representative of what would be running in production. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12670) Fix TestNetUtils and TestSecurityUtil when localhost is ipv6 only
[
https://issues.apache.org/jira/browse/HADOOP-12670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HADOOP-12670:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Got the non-binding +1 from [~mantonov]
Pushed to branch
> Fix TestNetUtils and TestSecurityUtil when localhost is ipv6 only
> -
>
> Key: HADOOP-12670
> URL: https://issues.apache.org/jira/browse/HADOOP-12670
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: net
>Affects Versions: HADOOP-11890
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HADOOP-12670-HADOOP-11890.0.patch,
> HADOOP-12670-HADOOP-11890.2.patch, HADOOP-12670-HADOOP-11890.3.patch
>
>
> {code}
> TestSecurityUtil.testBuildTokenServiceSockAddr:165
> expected:<[127.0.0.]1:123> but was:<[0:0:0:0:0:0:0:]1:123>
> TestSecurityUtil.testBuildDTServiceName:148 expected:<[127.0.0.]1:123> but
> was:<[0:0:0:0:0:0:0:]1:123>
>
> TestSecurityUtil.testSocketAddrWithName:326->verifyServiceAddr:304->verifyAddress:284->verifyValues:251
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
>
> TestSecurityUtil.testSocketAddrWithIP:333->verifyServiceAddr:304->verifyAddress:284->verifyValues:251
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
>
> TestSecurityUtil.testSocketAddrWithNameToStaticName:340->verifyServiceAddr:304->verifyAddress:284->verifyValues:251
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
> TestNetUtils.testNormalizeHostName:639 expected:<[0:0:0:0:0:0:0:]1> but
> was:<[127.0.0.]1>
> TestNetUtils.testResolverLoopback:533->verifyInetAddress:496
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12485) Don't prefer IPV4 stack on tests.
[ https://issues.apache.org/jira/browse/HADOOP-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12485: --- Attachment: HADOOP-12485-HADOOP-11890.1.patch > Don't prefer IPV4 stack on tests. > - > > Key: HADOOP-12485 > URL: https://issues.apache.org/jira/browse/HADOOP-12485 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Affects Versions: HADOOP-11890 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12485-HADOOP-11890.0.patch, > HADOOP-12485-HADOOP-11890.1.patch > > > Don't prefer ipv4 stack and instead use ipv6 if it's there. This should mean > that tests actually bind on ipv6 if it's there. > It should mean that tests running against this branch are more likely to be > representative of what would be running in production. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12670) Fix TestNetUtils and TestSecurityUtil when localhost is ipv6 only
[
https://issues.apache.org/jira/browse/HADOOP-12670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HADOOP-12670:
---
Attachment: HADOOP-12670-HADOOP-11890.3.patch
> Fix TestNetUtils and TestSecurityUtil when localhost is ipv6 only
> -
>
> Key: HADOOP-12670
> URL: https://issues.apache.org/jira/browse/HADOOP-12670
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: net
>Affects Versions: HADOOP-11890
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HADOOP-12670-HADOOP-11890.0.patch,
> HADOOP-12670-HADOOP-11890.2.patch, HADOOP-12670-HADOOP-11890.3.patch
>
>
> {code}
> TestSecurityUtil.testBuildTokenServiceSockAddr:165
> expected:<[127.0.0.]1:123> but was:<[0:0:0:0:0:0:0:]1:123>
> TestSecurityUtil.testBuildDTServiceName:148 expected:<[127.0.0.]1:123> but
> was:<[0:0:0:0:0:0:0:]1:123>
>
> TestSecurityUtil.testSocketAddrWithName:326->verifyServiceAddr:304->verifyAddress:284->verifyValues:251
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
>
> TestSecurityUtil.testSocketAddrWithIP:333->verifyServiceAddr:304->verifyAddress:284->verifyValues:251
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
>
> TestSecurityUtil.testSocketAddrWithNameToStaticName:340->verifyServiceAddr:304->verifyAddress:284->verifyValues:251
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
> TestNetUtils.testNormalizeHostName:639 expected:<[0:0:0:0:0:0:0:]1> but
> was:<[127.0.0.]1>
> TestNetUtils.testResolverLoopback:533->verifyInetAddress:496
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12670) Fix TestNetUtils and TestSecurityUtil when localhost is ipv6 only
Elliott Clark created HADOOP-12670:
--
Summary: Fix TestNetUtils and TestSecurityUtil when localhost is
ipv6 only
Key: HADOOP-12670
URL: https://issues.apache.org/jira/browse/HADOOP-12670
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Elliott Clark
Assignee: Elliott Clark
{code}
TestSecurityUtil.testBuildTokenServiceSockAddr:165 expected:<[127.0.0.]1:123>
but was:<[0:0:0:0:0:0:0:]1:123>
TestSecurityUtil.testBuildDTServiceName:148 expected:<[127.0.0.]1:123> but
was:<[0:0:0:0:0:0:0:]1:123>
TestSecurityUtil.testSocketAddrWithName:326->verifyServiceAddr:304->verifyAddress:284->verifyValues:251
expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
TestSecurityUtil.testSocketAddrWithIP:333->verifyServiceAddr:304->verifyAddress:284->verifyValues:251
expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
TestSecurityUtil.testSocketAddrWithNameToStaticName:340->verifyServiceAddr:304->verifyAddress:284->verifyValues:251
expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
TestNetUtils.testNormalizeHostName:639 expected:<[0:0:0:0:0:0:0:]1> but
was:<[127.0.0.]1>
TestNetUtils.testResolverLoopback:533->verifyInetAddress:496
expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12670) Fix TestNetUtils and TestSecurityUtil when localhost is ipv6 only
[
https://issues.apache.org/jira/browse/HADOOP-12670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HADOOP-12670:
---
Attachment: HADOOP-12670-HADOOP-11890.0.patch
Make sure to use the new NetUtils functions.
> Fix TestNetUtils and TestSecurityUtil when localhost is ipv6 only
> -
>
> Key: HADOOP-12670
> URL: https://issues.apache.org/jira/browse/HADOOP-12670
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: net
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HADOOP-12670-HADOOP-11890.0.patch
>
>
> {code}
> TestSecurityUtil.testBuildTokenServiceSockAddr:165
> expected:<[127.0.0.]1:123> but was:<[0:0:0:0:0:0:0:]1:123>
> TestSecurityUtil.testBuildDTServiceName:148 expected:<[127.0.0.]1:123> but
> was:<[0:0:0:0:0:0:0:]1:123>
>
> TestSecurityUtil.testSocketAddrWithName:326->verifyServiceAddr:304->verifyAddress:284->verifyValues:251
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
>
> TestSecurityUtil.testSocketAddrWithIP:333->verifyServiceAddr:304->verifyAddress:284->verifyValues:251
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
>
> TestSecurityUtil.testSocketAddrWithNameToStaticName:340->verifyServiceAddr:304->verifyAddress:284->verifyValues:251
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
> TestNetUtils.testNormalizeHostName:639 expected:<[0:0:0:0:0:0:0:]1> but
> was:<[127.0.0.]1>
> TestNetUtils.testResolverLoopback:533->verifyInetAddress:496
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12670) Fix TestNetUtils and TestSecurityUtil when localhost is ipv6 only
[
https://issues.apache.org/jira/browse/HADOOP-12670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HADOOP-12670:
---
Affects Version/s: HADOOP-11890
Status: Patch Available (was: Open)
> Fix TestNetUtils and TestSecurityUtil when localhost is ipv6 only
> -
>
> Key: HADOOP-12670
> URL: https://issues.apache.org/jira/browse/HADOOP-12670
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: net
>Affects Versions: HADOOP-11890
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HADOOP-12670-HADOOP-11890.0.patch
>
>
> {code}
> TestSecurityUtil.testBuildTokenServiceSockAddr:165
> expected:<[127.0.0.]1:123> but was:<[0:0:0:0:0:0:0:]1:123>
> TestSecurityUtil.testBuildDTServiceName:148 expected:<[127.0.0.]1:123> but
> was:<[0:0:0:0:0:0:0:]1:123>
>
> TestSecurityUtil.testSocketAddrWithName:326->verifyServiceAddr:304->verifyAddress:284->verifyValues:251
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
>
> TestSecurityUtil.testSocketAddrWithIP:333->verifyServiceAddr:304->verifyAddress:284->verifyValues:251
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
>
> TestSecurityUtil.testSocketAddrWithNameToStaticName:340->verifyServiceAddr:304->verifyAddress:284->verifyValues:251
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
> TestNetUtils.testNormalizeHostName:639 expected:<[0:0:0:0:0:0:0:]1> but
> was:<[127.0.0.]1>
> TestNetUtils.testResolverLoopback:533->verifyInetAddress:496
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11990) DNS#reverseDns fails with a NumberFormatException when using an IPv6 DNS server
[
https://issues.apache.org/jira/browse/HADOOP-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068958#comment-15068958
]
Elliott Clark commented on HADOOP-11990:
8u60 has this and the fix seems to work well for me.
> DNS#reverseDns fails with a NumberFormatException when using an IPv6 DNS
> server
> ---
>
> Key: HADOOP-11990
> URL: https://issues.apache.org/jira/browse/HADOOP-11990
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: net
>Affects Versions: 2.5.1
> Environment: java version "1.7.0_45"
> Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
> Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
>Reporter: Benoit Sigoure
> Labels: ipv6
>
> With this resolv.conf:
> {code}
> nameserver 192.168.1.1
> nameserver 2604:5500:3::3
> nameserver 2604:5500:3:3::3
> {code}
> Starting HBase yields the following:
> {code}
> Caused by: java.lang.NumberFormatException: For input string: "5500:3::3"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:492)
> at java.lang.Integer.parseInt(Integer.java:527)
> at com.sun.jndi.dns.DnsClient.(DnsClient.java:122)
> at com.sun.jndi.dns.Resolver.(Resolver.java:61)
> at com.sun.jndi.dns.DnsContext.getResolver(DnsContext.java:570)
> at com.sun.jndi.dns.DnsContext.c_getAttributes(DnsContext.java:430)
> at
> com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(ComponentDirContext.java:231)
> at
> com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:139)
> at
> com.sun.jndi.toolkit.url.GenericURLDirContext.getAttributes(GenericURLDirContext.java:103)
> at
> javax.naming.directory.InitialDirContext.getAttributes(InitialDirContext.java:142)
> at org.apache.hadoop.net.DNS.reverseDns(DNS.java:84)
> at org.apache.hadoop.net.DNS.getHosts(DNS.java:241)
> at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:344)
> at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:362)
> at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:341)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getHostname(RSRpcServices.java:825)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.(RSRpcServices.java:782)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.(MasterRpcServices.java:195)
> at
> org.apache.hadoop.hbase.master.HMaster.createRpcServices(HMaster.java:477)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:492)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:333)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:276)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-11990) DNS#reverseDns fails with a NumberFormatException when using an IPv6 DNS server
[
https://issues.apache.org/jira/browse/HADOOP-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark resolved HADOOP-11990.
Resolution: Not A Problem
Release Note:
If your resolvers are ipv6 addresses make sure that you use one of the java
versions listed in https://bugs.openjdk.java.net/browse/JDK-6991580
jdk8u60, jdk8u65, or jdk9+
> DNS#reverseDns fails with a NumberFormatException when using an IPv6 DNS
> server
> ---
>
> Key: HADOOP-11990
> URL: https://issues.apache.org/jira/browse/HADOOP-11990
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: net
>Affects Versions: 2.5.1
> Environment: java version "1.7.0_45"
> Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
> Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
>Reporter: Benoit Sigoure
> Labels: ipv6
>
> With this resolv.conf:
> {code}
> nameserver 192.168.1.1
> nameserver 2604:5500:3::3
> nameserver 2604:5500:3:3::3
> {code}
> Starting HBase yields the following:
> {code}
> Caused by: java.lang.NumberFormatException: For input string: "5500:3::3"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:492)
> at java.lang.Integer.parseInt(Integer.java:527)
> at com.sun.jndi.dns.DnsClient.(DnsClient.java:122)
> at com.sun.jndi.dns.Resolver.(Resolver.java:61)
> at com.sun.jndi.dns.DnsContext.getResolver(DnsContext.java:570)
> at com.sun.jndi.dns.DnsContext.c_getAttributes(DnsContext.java:430)
> at
> com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(ComponentDirContext.java:231)
> at
> com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:139)
> at
> com.sun.jndi.toolkit.url.GenericURLDirContext.getAttributes(GenericURLDirContext.java:103)
> at
> javax.naming.directory.InitialDirContext.getAttributes(InitialDirContext.java:142)
> at org.apache.hadoop.net.DNS.reverseDns(DNS.java:84)
> at org.apache.hadoop.net.DNS.getHosts(DNS.java:241)
> at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:344)
> at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:362)
> at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:341)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getHostname(RSRpcServices.java:825)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.(RSRpcServices.java:782)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.(MasterRpcServices.java:195)
> at
> org.apache.hadoop.hbase.master.HMaster.createRpcServices(HMaster.java:477)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:492)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:333)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:276)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-12431) NameNode should bind on both IPv6 and IPv4 if running on dual-stack machine and IPv6 enabled
[ https://issues.apache.org/jira/browse/HADOOP-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark resolved HADOOP-12431. Resolution: Won't Fix Assignee: (was: Nemanja Matkovic) Going to resolve this one as won't fix. We don't want to bind to ipv6 by default. Instead I'm going to open a documentation jira about how to set up a cluster with dual stack. > NameNode should bind on both IPv6 and IPv4 if running on dual-stack machine > and IPv6 enabled > > > Key: HADOOP-12431 > URL: https://issues.apache.org/jira/browse/HADOOP-12431 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.7.1 >Reporter: Nate Edel > Labels: ipv6 > > NameNode works properly on IPv4 or IPv6 single stack (assuming in the latter > case that scripts have been changed to disable preferIPv4Stack, and dependent > on the client/data node fix in HDFS-8078). On dual-stack machines, NameNode > listens only on IPv4 (even ignoring preferIPv6Addresses being set.) > Our initial use case for IPv6 is IPv6-only clusters, but ideally we'd support > binding to both the IPv4 and IPv6 machine addresses so that we can support > heterogenous clusters (some dual-stack and some IPv6-only machines.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12670) Fix TestNetUtils and TestSecurityUtil when localhost is ipv6 only
[
https://issues.apache.org/jira/browse/HADOOP-12670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HADOOP-12670:
---
Attachment: HADOOP-12670-HADOOP-11890.2.patch
Does docker not work on branches ?
> Fix TestNetUtils and TestSecurityUtil when localhost is ipv6 only
> -
>
> Key: HADOOP-12670
> URL: https://issues.apache.org/jira/browse/HADOOP-12670
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: net
>Affects Versions: HADOOP-11890
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HADOOP-12670-HADOOP-11890.0.patch,
> HADOOP-12670-HADOOP-11890.2.patch
>
>
> {code}
> TestSecurityUtil.testBuildTokenServiceSockAddr:165
> expected:<[127.0.0.]1:123> but was:<[0:0:0:0:0:0:0:]1:123>
> TestSecurityUtil.testBuildDTServiceName:148 expected:<[127.0.0.]1:123> but
> was:<[0:0:0:0:0:0:0:]1:123>
>
> TestSecurityUtil.testSocketAddrWithName:326->verifyServiceAddr:304->verifyAddress:284->verifyValues:251
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
>
> TestSecurityUtil.testSocketAddrWithIP:333->verifyServiceAddr:304->verifyAddress:284->verifyValues:251
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
>
> TestSecurityUtil.testSocketAddrWithNameToStaticName:340->verifyServiceAddr:304->verifyAddress:284->verifyValues:251
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
> TestNetUtils.testNormalizeHostName:639 expected:<[0:0:0:0:0:0:0:]1> but
> was:<[127.0.0.]1>
> TestNetUtils.testResolverLoopback:533->verifyInetAddress:496
> expected:<[127.0.0.]1> but was:<[0:0:0:0:0:0:0:]1>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-11582) org.apache.hadoop.net.TestDNS failing with NumberFormatException -IPv6 related?
[
https://issues.apache.org/jira/browse/HADOOP-11582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark resolved HADOOP-11582.
Resolution: Not A Problem
This was fixed in a different jira.
> org.apache.hadoop.net.TestDNS failing with NumberFormatException -IPv6
> related?
> ---
>
> Key: HADOOP-11582
> URL: https://issues.apache.org/jira/browse/HADOOP-11582
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: net
>Affects Versions: 3.0.0
> Environment: OSX yosemite
>Reporter: Steve Loughran
>Assignee: Varun Saxena
> Labels: ipv6
>
> {{org.apache.hadoop.net.TestDNS}} failing {{java.lang.NumberFormatException:
> For input string: ":3246:9aff:fe80:438f"}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12421) Add jitter to RetryInvocationHandler
[ https://issues.apache.org/jira/browse/HADOOP-12421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12421: --- Attachment: HADOOP-12421-v5.patch The tests that Jenkins had a hard time with all pass locally for me. > Add jitter to RetryInvocationHandler > > > Key: HADOOP-12421 > URL: https://issues.apache.org/jira/browse/HADOOP-12421 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12421-v1.patch, HADOOP-12421-v2.patch, > HADOOP-12421-v3.patch, HADOOP-12421-v4.patch, HADOOP-12421-v5.patch > > > Calls to NN can become synchronized across a cluster during NN failover. This > leads to a spike in requests until things recover. Making an already tricky > time worse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12421) Add jitter to RetryInvocationHandler
[ https://issues.apache.org/jira/browse/HADOOP-12421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057193#comment-15057193 ] Elliott Clark commented on HADOOP-12421: Yeah let me get that up. > Add jitter to RetryInvocationHandler > > > Key: HADOOP-12421 > URL: https://issues.apache.org/jira/browse/HADOOP-12421 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12421-v1.patch, HADOOP-12421-v2.patch, > HADOOP-12421-v3.patch, HADOOP-12421-v4.patch > > > Calls to NN can become synchronized across a cluster during NN failover. This > leads to a spike in requests until things recover. Making an already tricky > time worse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12421) Add jitter to RetryInvocationHandler
[ https://issues.apache.org/jira/browse/HADOOP-12421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12421: --- Attachment: HADOOP-12421-v4.patch Rebased on trunk > Add jitter to RetryInvocationHandler > > > Key: HADOOP-12421 > URL: https://issues.apache.org/jira/browse/HADOOP-12421 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12421-v1.patch, HADOOP-12421-v2.patch, > HADOOP-12421-v3.patch, HADOOP-12421-v4.patch > > > Calls to NN can become synchronized across a cluster during NN failover. This > leads to a spike in requests until things recover. Making an already tricky > time worse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12421) Add jitter to RetryInvocationHandler
[ https://issues.apache.org/jira/browse/HADOOP-12421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15052066#comment-15052066 ] Elliott Clark commented on HADOOP-12421: Sure let me do that tomorrow. Thanks > Add jitter to RetryInvocationHandler > > > Key: HADOOP-12421 > URL: https://issues.apache.org/jira/browse/HADOOP-12421 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12421-v1.patch, HADOOP-12421-v2.patch, > HADOOP-12421-v3.patch > > > Calls to NN can become synchronized across a cluster during NN failover. This > leads to a spike in requests until things recover. Making an already tricky > time worse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12421) Add jitter to RetryInvocationHandler
[ https://issues.apache.org/jira/browse/HADOOP-12421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049682#comment-15049682 ] Elliott Clark commented on HADOOP-12421: Ping? > Add jitter to RetryInvocationHandler > > > Key: HADOOP-12421 > URL: https://issues.apache.org/jira/browse/HADOOP-12421 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12421-v1.patch, HADOOP-12421-v2.patch, > HADOOP-12421-v3.patch > > > Calls to NN can become synchronized across a cluster during NN failover. This > leads to a spike in requests until things recover. Making an already tricky > time worse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12491) Hadoop-common - Avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/HADOOP-12491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965609#comment-14965609 ] Elliott Clark commented on HADOOP-12491: +1 on this. Ran the tests and everything looks good. This doesn't change the defaults on flags that Hadoop runs with. > Hadoop-common - Avoid unsafe split and append on fields that might be IPv6 > literals > --- > > Key: HADOOP-12491 > URL: https://issues.apache.org/jira/browse/HADOOP-12491 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-11890 >Reporter: Nemanja Matkovic >Assignee: Nemanja Matkovic > Labels: ipv6 > Fix For: HADOOP-11890 > > Attachments: HADOOP-12491-HADOOP-11890.1.patch, > HADOOP-12491-HADOOP-11890.2.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > Hadoop-common portion of HADOOP-12122 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12491) Hadoop-common - Avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/HADOOP-12491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965638#comment-14965638 ] Elliott Clark commented on HADOOP-12491: Pushed to the branch. Thanks for the review [~ste...@apache.org] > Hadoop-common - Avoid unsafe split and append on fields that might be IPv6 > literals > --- > > Key: HADOOP-12491 > URL: https://issues.apache.org/jira/browse/HADOOP-12491 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-11890 >Reporter: Nemanja Matkovic >Assignee: Nemanja Matkovic > Labels: ipv6 > Fix For: HADOOP-11890 > > Attachments: HADOOP-12491-HADOOP-11890.1.patch, > HADOOP-12491-HADOOP-11890.2.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > Hadoop-common portion of HADOOP-12122 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12122) Fix Hadoop should avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/HADOOP-12122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12122: --- Resolution: Not A Problem Status: Resolved (was: Patch Available) Resolving in favor of the per module issues. > Fix Hadoop should avoid unsafe split and append on fields that might be IPv6 > literals > - > > Key: HADOOP-12122 > URL: https://issues.apache.org/jira/browse/HADOOP-12122 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-11890 >Reporter: Nate Edel >Assignee: Nemanja Matkovic > Attachments: HADOOP-12122-HADOOP-11890.0.patch, > HADOOP-12122-HADOOP-11890.3.patch, HADOOP-12122-HADOOP-11890.4.patch, > HADOOP-12122-HADOOP-11890.5.patch, HADOOP-12122-HADOOP-11890.6.patch, > HADOOP-12122-HADOOP-11890.7.patch, HADOOP-12122-HADOOP-11890.8.patch, > HADOOP-12122-HADOOP-11890.9.patch, HADOOP-12122-HADOOP-12122.2.patch, > HADOOP-12122-HADOOP-12122.3.patch, HADOOP-12122.0.patch, > lets_blow_up_a_lot_of_tests.patch > > > There are a fairly extensive number of locations found via code inspection > which use unsafe methods of handling addresses in a dual-stack or IPv6-only > world: > - splits on the first ":" assuming that delimits a host from a port > - produces a host port pair by appending :port blindly (Java prefers > [ipv6]:port which is the standard for IPv6 URIs) > - depends on the behavior of InetSocketAddress.toString() which produces the > above. > This patch fixes those metaphors that I can find above, and replaces calls to > InetSocketAddress.toString() with a wrapper that properly brackets the IPv6 > address if there is one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12485) Don't prefer IPV4 stack on tests.
[ https://issues.apache.org/jira/browse/HADOOP-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14961527#comment-14961527 ] Elliott Clark commented on HADOOP-12485: Yeah probably. Let me get that in. > Don't prefer IPV4 stack on tests. > - > > Key: HADOOP-12485 > URL: https://issues.apache.org/jira/browse/HADOOP-12485 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Affects Versions: HADOOP-11890 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12485-HADOOP-11890.0.patch > > > Don't prefer ipv4 stack and instead use ipv6 if it's there. This should mean > that tests actually bind on ipv6 if it's there. > It should mean that tests running against this branch are more likely to be > representative of what would be running in production. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12122) Fix Hadoop should avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/HADOOP-12122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958917#comment-14958917 ] Elliott Clark commented on HADOOP-12122: There's a ton of star imports. Can we get some ipv6 ip's in the tests that you had to change? > Fix Hadoop should avoid unsafe split and append on fields that might be IPv6 > literals > - > > Key: HADOOP-12122 > URL: https://issues.apache.org/jira/browse/HADOOP-12122 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-11890 >Reporter: Nate Edel >Assignee: Nemanja Matkovic > Attachments: HADOOP-12122-HADOOP-11890.0.patch, > HADOOP-12122-HADOOP-11890.3.patch, HADOOP-12122-HADOOP-11890.4.patch, > HADOOP-12122-HADOOP-11890.5.patch, HADOOP-12122-HADOOP-11890.6.patch, > HADOOP-12122-HADOOP-11890.7.patch, HADOOP-12122-HADOOP-11890.8.patch, > HADOOP-12122-HADOOP-12122.2.patch, HADOOP-12122-HADOOP-12122.3.patch, > HADOOP-12122.0.patch, lets_blow_up_a_lot_of_tests.patch > > > There are a fairly extensive number of locations found via code inspection > which use unsafe methods of handling addresses in a dual-stack or IPv6-only > world: > - splits on the first ":" assuming that delimits a host from a port > - produces a host port pair by appending :port blindly (Java prefers > [ipv6]:port which is the standard for IPv6 URIs) > - depends on the behavior of InetSocketAddress.toString() which produces the > above. > This patch fixes those metaphors that I can find above, and replaces calls to > InetSocketAddress.toString() with a wrapper that properly brackets the IPv6 > address if there is one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12485) Don't prefer IPV4 stack on tests.
Elliott Clark created HADOOP-12485: -- Summary: Don't prefer IPV4 stack on tests. Key: HADOOP-12485 URL: https://issues.apache.org/jira/browse/HADOOP-12485 Project: Hadoop Common Issue Type: Sub-task Components: test Affects Versions: HADOOP-11890 Reporter: Elliott Clark Assignee: Elliott Clark -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12485) Don't prefer IPV4 stack on tests.
[ https://issues.apache.org/jira/browse/HADOOP-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12485: --- Status: Patch Available (was: Open) > Don't prefer IPV4 stack on tests. > - > > Key: HADOOP-12485 > URL: https://issues.apache.org/jira/browse/HADOOP-12485 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Affects Versions: HADOOP-11890 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12485-HADOOP-11890.0.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12485) Don't prefer IPV4 stack on tests.
[ https://issues.apache.org/jira/browse/HADOOP-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12485: --- Attachment: HADOOP-12485-HADOOP-11890.0.patch > Don't prefer IPV4 stack on tests. > - > > Key: HADOOP-12485 > URL: https://issues.apache.org/jira/browse/HADOOP-12485 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Affects Versions: HADOOP-11890 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12485-HADOOP-11890.0.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12485) Don't prefer IPV4 stack on tests.
[ https://issues.apache.org/jira/browse/HADOOP-12485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12485: --- Description: Don't prefer ipv4 stack and instead use ipv6 if it's there. This should mean that tests actually bind on ipv6 if it's there. It should mean that tests running against this branch are more likely to be representative of what would be running in production. > Don't prefer IPV4 stack on tests. > - > > Key: HADOOP-12485 > URL: https://issues.apache.org/jira/browse/HADOOP-12485 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Affects Versions: HADOOP-11890 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HADOOP-12485-HADOOP-11890.0.patch > > > Don't prefer ipv4 stack and instead use ipv6 if it's there. This should mean > that tests actually bind on ipv6 if it's there. > It should mean that tests running against this branch are more likely to be > representative of what would be running in production. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12122) Fix Hadoop should avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/HADOOP-12122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959762#comment-14959762 ] Elliott Clark commented on HADOOP-12122: Comments on TestNetUtils are just wrong. pom.xml still turns off ipv6 wouldn't allowing ipv6 stack make the tests more representative? It should fully work now correct? Comment IPCLoggerChannel why is it doing the dance of ip instanceof Inet6Address? DatanodeManager talks about BasicInetPeer why not just change that? Checkpointer seems like that's still not very elegant code. Is there something better ? HostUtil creates a logger and never uses it. Remove it if it's not going to be needed. ConverterUtils' logger looks really weird. NetUtils' javadoc seems weird. Why add an extra line that says basically the same thing to a single method? > Fix Hadoop should avoid unsafe split and append on fields that might be IPv6 > literals > - > > Key: HADOOP-12122 > URL: https://issues.apache.org/jira/browse/HADOOP-12122 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-11890 >Reporter: Nate Edel >Assignee: Nemanja Matkovic > Attachments: HADOOP-12122-HADOOP-11890.0.patch, > HADOOP-12122-HADOOP-11890.3.patch, HADOOP-12122-HADOOP-11890.4.patch, > HADOOP-12122-HADOOP-11890.5.patch, HADOOP-12122-HADOOP-11890.6.patch, > HADOOP-12122-HADOOP-11890.7.patch, HADOOP-12122-HADOOP-11890.8.patch, > HADOOP-12122-HADOOP-12122.2.patch, HADOOP-12122-HADOOP-12122.3.patch, > HADOOP-12122.0.patch, lets_blow_up_a_lot_of_tests.patch > > > There are a fairly extensive number of locations found via code inspection > which use unsafe methods of handling addresses in a dual-stack or IPv6-only > world: > - splits on the first ":" assuming that delimits a host from a port > - produces a host port pair by appending :port blindly (Java prefers > [ipv6]:port which is the standard for IPv6 URIs) > - depends on the behavior of InetSocketAddress.toString() which produces the > above. > This patch fixes those metaphors that I can find above, and replaces calls to > InetSocketAddress.toString() with a wrapper that properly brackets the IPv6 > address if there is one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12122) Fix Hadoop should avoid unsafe split and append on fields that might be IPv6 literals
[
https://issues.apache.org/jira/browse/HADOOP-12122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14947470#comment-14947470
]
Elliott Clark commented on HADOOP-12122:
Also while looking at these string splits, we should look at indexOf(":") from
a quick glance there are more places where similar unsafe splits are happening.
> Fix Hadoop should avoid unsafe split and append on fields that might be IPv6
> literals
> -
>
> Key: HADOOP-12122
> URL: https://issues.apache.org/jira/browse/HADOOP-12122
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: HADOOP-11890
>Reporter: Nate Edel
>Assignee: Nemanja Matkovic
> Attachments: HADOOP-12122-HADOOP-11890.0.patch,
> HADOOP-12122-HADOOP-11890.3.patch, HADOOP-12122-HADOOP-11890.4.patch,
> HADOOP-12122-HADOOP-12122.2.patch, HADOOP-12122-HADOOP-12122.3.patch,
> HADOOP-12122.0.patch, lets_blow_up_a_lot_of_tests.patch
>
>
> There are a fairly extensive number of locations found via code inspection
> which use unsafe methods of handling addresses in a dual-stack or IPv6-only
> world:
> - splits on the first ":" assuming that delimits a host from a port
> - produces a host port pair by appending :port blindly (Java prefers
> [ipv6]:port which is the standard for IPv6 URIs)
> - depends on the behavior of InetSocketAddress.toString() which produces the
> above.
> This patch fixes those metaphors that I can find above, and replaces calls to
> InetSocketAddress.toString() with a wrapper that properly brackets the IPv6
> address if there is one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12430) Fix HDFS client gets errors trying to to connect to IPv6 DataNode
[ https://issues.apache.org/jira/browse/HADOOP-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12430: --- Summary: Fix HDFS client gets errors trying to to connect to IPv6 DataNode (was: HDFS client gets errors trying to to connect to IPv6 DataNode) > Fix HDFS client gets errors trying to to connect to IPv6 DataNode > - > > Key: HADOOP-12430 > URL: https://issues.apache.org/jira/browse/HADOOP-12430 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.6.0 >Reporter: Nate Edel >Assignee: Nate Edel > Labels: BB2015-05-TBR, ipv6 > Attachments: HDFS-8078.10.patch, HDFS-8078.11.patch, > HDFS-8078.12.patch, HDFS-8078.13.patch, HDFS-8078.14.patch, > HDFS-8078.15.patch, HDFS-8078.9.patch, dummy.patch > > > 1st exception, on put: > 15/03/23 18:43:18 WARN hdfs.DFSClient: DataStreamer Exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: 2401:db00:1010:70ba:face:0:8:0:50010 > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at > org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1607) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) > Appears to actually stem from code in DataNodeID which assumes it's safe to > append together (ipaddr + ":" + port) -- which is OK for IPv4 and not OK for > IPv6. NetUtils.createSocketAddr( ) assembles a Java URI object, which > requires the format proto://[2401:db00:1010:70ba:face:0:8:0]:50010 > Currently using InetAddress.getByName() to validate IPv6 (guava > InetAddresses.forString has been flaky) but could also use our own parsing. > (From logging this, it seems like a low-enough frequency call that the extra > object creation shouldn't be problematic, and for me the slight risk of > passing in bad input that is not actually an IPv4 or IPv6 address and thus > calling an external DNS lookup is outweighed by getting the address > normalized and avoiding rewriting parsing.) > Alternatively, sun.net.util.IPAddressUtil.isIPv6LiteralAddress() > --- > 2nd exception (on datanode) > 15/04/13 13:18:07 ERROR datanode.DataNode: > dev1903.prn1.facebook.com:50010:DataXceiver error processing unknown > operation src: /2401:db00:20:7013:face:0:7:0:54152 dst: > /2401:db00:11:d010:face:0:2f:0:50010 > java.io.EOFException > at java.io.DataInputStream.readShort(DataInputStream.java:315) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:226) > at java.lang.Thread.run(Thread.java:745) > Which also comes as client error "-get: 2401 is not an IP string literal." > This one has existing parsing logic which needs to shift to the last colon > rather than the first. Should also be a tiny bit faster by using lastIndexOf > rather than split. Could alternatively use the techniques above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12122) Fix Hadoop should avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/HADOOP-12122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12122: --- Attachment: HADOOP-12122-HADOOP-11890.3.patch Correct patch naming > Fix Hadoop should avoid unsafe split and append on fields that might be IPv6 > literals > - > > Key: HADOOP-12122 > URL: https://issues.apache.org/jira/browse/HADOOP-12122 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-11890 >Reporter: Nate Edel >Assignee: Elliott Clark > Attachments: HADOOP-12122-HADOOP-11890.0.patch, > HADOOP-12122-HADOOP-11890.3.patch, HADOOP-12122-HADOOP-12122.2.patch, > HADOOP-12122-HADOOP-12122.3.patch, HADOOP-12122.0.patch, > lets_blow_up_a_lot_of_tests.patch > > > There are a fairly extensive number of locations found via code inspection > which use unsafe methods of handling addresses in a dual-stack or IPv6-only > world: > - splits on the first ":" assuming that delimits a host from a port > - produces a host port pair by appending :port blindly (Java prefers > [ipv6]:port which is the standard for IPv6 URIs) > - depends on the behavior of InetSocketAddress.toString() which produces the > above. > This patch fixes those metaphors that I can find above, and replaces calls to > InetSocketAddress.toString() with a wrapper that properly brackets the IPv6 > address if there is one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HADOOP-12122) Fix Hadoop should avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/HADOOP-12122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark reassigned HADOOP-12122: -- Assignee: Elliott Clark (was: Nate Edel) > Fix Hadoop should avoid unsafe split and append on fields that might be IPv6 > literals > - > > Key: HADOOP-12122 > URL: https://issues.apache.org/jira/browse/HADOOP-12122 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-11890 >Reporter: Nate Edel >Assignee: Elliott Clark > Attachments: HADOOP-12122-HADOOP-11890.0.patch, > HADOOP-12122-HADOOP-12122.2.patch, HADOOP-12122.0.patch, > lets_blow_up_a_lot_of_tests.patch > > > There are a fairly extensive number of locations found via code inspection > which use unsafe methods of handling addresses in a dual-stack or IPv6-only > world: > - splits on the first ":" assuming that delimits a host from a port > - produces a host port pair by appending :port blindly (Java prefers > [ipv6]:port which is the standard for IPv6 URIs) > - depends on the behavior of InetSocketAddress.toString() which produces the > above. > This patch fixes those metaphors that I can find above, and replaces calls to > InetSocketAddress.toString() with a wrapper that properly brackets the IPv6 > address if there is one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12122) Fix Hadoop should avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/HADOOP-12122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12122: --- Attachment: HADOOP-12122-HADOOP-12122.2.patch Rebased patch as the patch no longer applied to trunk > Fix Hadoop should avoid unsafe split and append on fields that might be IPv6 > literals > - > > Key: HADOOP-12122 > URL: https://issues.apache.org/jira/browse/HADOOP-12122 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-11890 >Reporter: Nate Edel >Assignee: Nate Edel > Attachments: HADOOP-12122-HADOOP-11890.0.patch, > HADOOP-12122-HADOOP-12122.2.patch, HADOOP-12122.0.patch, > lets_blow_up_a_lot_of_tests.patch > > > There are a fairly extensive number of locations found via code inspection > which use unsafe methods of handling addresses in a dual-stack or IPv6-only > world: > - splits on the first ":" assuming that delimits a host from a port > - produces a host port pair by appending :port blindly (Java prefers > [ipv6]:port which is the standard for IPv6 URIs) > - depends on the behavior of InetSocketAddress.toString() which produces the > above. > This patch fixes those metaphors that I can find above, and replaces calls to > InetSocketAddress.toString() with a wrapper that properly brackets the IPv6 > address if there is one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12122) Fix Hadoop should avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/HADOOP-12122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941409#comment-14941409 ] Elliott Clark commented on HADOOP-12122: Nate's out of the office for a little bit, I'll grab this and try and clean up his work > Fix Hadoop should avoid unsafe split and append on fields that might be IPv6 > literals > - > > Key: HADOOP-12122 > URL: https://issues.apache.org/jira/browse/HADOOP-12122 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-11890 >Reporter: Nate Edel >Assignee: Elliott Clark > Attachments: HADOOP-12122-HADOOP-11890.0.patch, > HADOOP-12122-HADOOP-12122.2.patch, HADOOP-12122.0.patch, > lets_blow_up_a_lot_of_tests.patch > > > There are a fairly extensive number of locations found via code inspection > which use unsafe methods of handling addresses in a dual-stack or IPv6-only > world: > - splits on the first ":" assuming that delimits a host from a port > - produces a host port pair by appending :port blindly (Java prefers > [ipv6]:port which is the standard for IPv6 URIs) > - depends on the behavior of InetSocketAddress.toString() which produces the > above. > This patch fixes those metaphors that I can find above, and replaces calls to > InetSocketAddress.toString() with a wrapper that properly brackets the IPv6 > address if there is one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12122) Fix Hadoop should avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/HADOOP-12122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HADOOP-12122: --- Summary: Fix Hadoop should avoid unsafe split and append on fields that might be IPv6 literals (was: Hadoop should avoid unsafe split and append on fields that might be IPv6 literals) > Fix Hadoop should avoid unsafe split and append on fields that might be IPv6 > literals > - > > Key: HADOOP-12122 > URL: https://issues.apache.org/jira/browse/HADOOP-12122 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-11890 >Reporter: Nate Edel >Assignee: Nate Edel > Attachments: HADOOP-12122-HADOOP-11890.0.patch, HADOOP-12122.0.patch, > lets_blow_up_a_lot_of_tests.patch > > > There are a fairly extensive number of locations found via code inspection > which use unsafe methods of handling addresses in a dual-stack or IPv6-only > world: > - splits on the first ":" assuming that delimits a host from a port > - produces a host port pair by appending :port blindly (Java prefers > [ipv6]:port which is the standard for IPv6 URIs) > - depends on the behavior of InetSocketAddress.toString() which produces the > above. > This patch fixes those metaphors that I can find above, and replaces calls to > InetSocketAddress.toString() with a wrapper that properly brackets the IPv6 > address if there is one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
