[jira] [Updated] (HADOOP-15275) Incorrect javadoc for return type of RetryPolicy#shouldRetry
[
https://issues.apache.org/jira/browse/HADOOP-15275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-15275:
---
Component/s: documentation
> Incorrect javadoc for return type of RetryPolicy#shouldRetry
>
>
> Key: HADOOP-15275
> URL: https://issues.apache.org/jira/browse/HADOOP-15275
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Minor
> Attachments: HADOOP-15275.000.patch
>
>
> The return type of {{RetryPolicy#shouldRetry}} has been changed from
> {{boolean}} to {{RetryAction}}, but the javadoc is not updated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15274) move hadoop-openstack to slf4j
[ https://issues.apache.org/jira/browse/HADOOP-15274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381764#comment-16381764 ] Akira Ajisaka commented on HADOOP-15274: LGTM, +1 > move hadoop-openstack to slf4j > -- > > Key: HADOOP-15274 > URL: https://issues.apache.org/jira/browse/HADOOP-15274 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/swift >Reporter: Steve Loughran >Assignee: fang zhenyi >Priority: Minor > Attachments: HADOOP-15274.001.patch, HADOOP-15274.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15275) Incorrect javadoc for return type of RetryPolicy#shouldRetry
[
https://issues.apache.org/jira/browse/HADOOP-15275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-15275:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.2
2.9.1
2.10.0
3.1.0
Status: Resolved (was: Patch Available)
Committed this to trunk, branch-3.1, branch-3.0, branch-2, and branch-2.9.
Thanks [~nandakumar131]!
> Incorrect javadoc for return type of RetryPolicy#shouldRetry
>
>
> Key: HADOOP-15275
> URL: https://issues.apache.org/jira/browse/HADOOP-15275
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Minor
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.2
>
> Attachments: HADOOP-15275.000.patch
>
>
> The return type of {{RetryPolicy#shouldRetry}} has been changed from
> {{boolean}} to {{RetryAction}}, but the javadoc is not updated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15275) Incorrect javadoc for return type of RetryPolicy#shouldRetry
[
https://issues.apache.org/jira/browse/HADOOP-15275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381807#comment-16381807
]
Hudson commented on HADOOP-15275:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13748 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13748/])
HADOOP-15275. Incorrect javadoc for return type of (aajisaka: rev
55d04a6db10d0e58cfe0ff82fa646bb6ba57aaac)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicy.java
> Incorrect javadoc for return type of RetryPolicy#shouldRetry
>
>
> Key: HADOOP-15275
> URL: https://issues.apache.org/jira/browse/HADOOP-15275
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Minor
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.2
>
> Attachments: HADOOP-15275.000.patch
>
>
> The return type of {{RetryPolicy#shouldRetry}} has been changed from
> {{boolean}} to {{RetryAction}}, but the javadoc is not updated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15275) Incorrect javadoc for return type of RetryPolicy#shouldRetry
[
https://issues.apache.org/jira/browse/HADOOP-15275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381772#comment-16381772
]
Akira Ajisaka commented on HADOOP-15275:
LGTM, +1
> Incorrect javadoc for return type of RetryPolicy#shouldRetry
>
>
> Key: HADOOP-15275
> URL: https://issues.apache.org/jira/browse/HADOOP-15275
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Minor
> Attachments: HADOOP-15275.000.patch
>
>
> The return type of {{RetryPolicy#shouldRetry}} has been changed from
> {{boolean}} to {{RetryAction}}, but the javadoc is not updated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15272) Update Guava, see what breaks
[ https://issues.apache.org/jira/browse/HADOOP-15272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15272: Parent Issue: HADOOP-14284 (was: HADOOP-9991) > Update Guava, see what breaks > - > > Key: HADOOP-15272 > URL: https://issues.apache.org/jira/browse/HADOOP-15272 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Priority: Major > > We're still on Guava 11; the last attempt at an update (HADOOP-10101) failed > to take > Now we have better shading, we should try again. I suspect that YARN timeline > service is going to be the problem because of its use of HBase. That's the > price of a loop in the DAG. We cannot keep everything frozen just because of > that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14187) Update ZooKeeper dependency to 3.4.9 and Curator dependency to 2.12.0
[ https://issues.apache.org/jira/browse/HADOOP-14187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382063#comment-16382063 ] Steve Loughran commented on HADOOP-14187: - Given trunk is still shipping Guava 11.0; is everything well with curator 2.12? For the 2.7.4 upgrade I had to verify this by doing a build of curator with the guava version marked down > Update ZooKeeper dependency to 3.4.9 and Curator dependency to 2.12.0 > - > > Key: HADOOP-14187 > URL: https://issues.apache.org/jira/browse/HADOOP-14187 > Project: Hadoop Common > Issue Type: Bug >Reporter: Tsuyoshi Ozawa >Assignee: Tsuyoshi Ozawa >Priority: Major > Fix For: 3.0.0-alpha4 > > Attachments: HADOOP-14187.001.patch > > > This is a update for using Apache Curator which shades guava. > Why is Curator updated to 2.12.0 instead of 3.3.0? > It's because Curator 3.x only supports ZooKeeper 3.5.x. ZooKeeper 3.5.x is > still alpha release. Hence, I think we should do conservative choice here. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15272) Update Guava, see what breaks
[ https://issues.apache.org/jira/browse/HADOOP-15272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15272: Description: We're still on Guava 11; the last attempt at an update (HADOOP-10101) failed to take The HBase 2 version of ATS should permit this, at least for its profile. was: We're still on Guava 11; the last attempt at an update (HADOOP-10101) failed to take Now we have better shading, we should try again. I suspect that YARN timeline service is going to be the problem because of its use of HBase. That's the price of a loop in the DAG. We cannot keep everything frozen just because of that. > Update Guava, see what breaks > - > > Key: HADOOP-15272 > URL: https://issues.apache.org/jira/browse/HADOOP-15272 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Priority: Major > > We're still on Guava 11; the last attempt at an update (HADOOP-10101) failed > to take > The HBase 2 version of ATS should permit this, at least for its profile. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15272) Update Guava, see what breaks
[ https://issues.apache.org/jira/browse/HADOOP-15272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381990#comment-16381990 ] Rohith Sharma K S commented on HADOOP-15272: This need to be discussed/tested irrespective of HBase-2 version which will be added in YARN-7346. ATSv2 code compiles by default with HBase-1.2.6 even after YARN-7346. So this JIRA shouldn't depends upon YARN-7346 specifically. > Update Guava, see what breaks > - > > Key: HADOOP-15272 > URL: https://issues.apache.org/jira/browse/HADOOP-15272 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Priority: Major > > We're still on Guava 11; the last attempt at an update (HADOOP-10101) failed > to take > The HBase 2 version of ATS should permit this, at least for its profile. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15269) S3 returning 400 on the directory /test/ GET of getFileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382082#comment-16382082
]
Steve Loughran commented on HADOOP-15269:
-
No, don't think I have versioning.
The issue has "gone away", that is: the prefix is gone, the path is no longer
there. It's like a delete tombstone marker was playing up.
Closing as cannot reproduce for now. Interesting event, makes me wonder if
there is any more diagnostics we can include in that original error message
> S3 returning 400 on the directory /test/ GET of getFileStatus
> -
>
> Key: HADOOP-15269
> URL: https://issues.apache.org/jira/browse/HADOOP-15269
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0, 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Fix For: 3.1.0
>
>
> Since Monday Feb 26, I'm getting intermittent failures of getFileStatus on a
> directory
> # file path: {{/test}} is returning 404, as expected
> # directory path {{//test/}} is returning 400, so failing the entire operation
> S3 Ireland.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15269) S3 returning 400 on the directory /test/ GET of getFileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-15269.
-
Resolution: Cannot Reproduce
Fix Version/s: 3.1.0
> S3 returning 400 on the directory /test/ GET of getFileStatus
> -
>
> Key: HADOOP-15269
> URL: https://issues.apache.org/jira/browse/HADOOP-15269
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0, 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Fix For: 3.1.0
>
>
> Since Monday Feb 26, I'm getting intermittent failures of getFileStatus on a
> directory
> # file path: {{/test}} is returning 404, as expected
> # directory path {{//test/}} is returning 400, so failing the entire operation
> S3 Ireland.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15269) S3 returning 400 on the directory /test/ GET of getFileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381902#comment-16381902
]
Steve Loughran commented on HADOOP-15269:
-
happens on the aws cli too. That's the good news: not our code
> S3 returning 400 on the directory /test/ GET of getFileStatus
> -
>
> Key: HADOOP-15269
> URL: https://issues.apache.org/jira/browse/HADOOP-15269
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0, 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
>
> Since Monday Feb 26, I'm getting intermittent failures of getFileStatus on a
> directory
> # file path: {{/test}} is returning 404, as expected
> # directory path {{//test/}} is returning 400, so failing the entire operation
> S3 Ireland.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15279) increase maven heap size recommendations
[ https://issues.apache.org/jira/browse/HADOOP-15279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382635#comment-16382635 ] genericqa commented on HADOOP-15279: (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/14236/console in case of problems. > increase maven heap size recommendations > > > Key: HADOOP-15279 > URL: https://issues.apache.org/jira/browse/HADOOP-15279 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Attachments: HADOOP-15279.00.patch > > > 1G is just a bit too low for JDK8+surefire 2.20+hdfs unit tests running in > parallel. Bump it up a bit more. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15279) increase maven heap size recommendations
Allen Wittenauer created HADOOP-15279: - Summary: increase maven heap size recommendations Key: HADOOP-15279 URL: https://issues.apache.org/jira/browse/HADOOP-15279 Project: Hadoop Common Issue Type: Bug Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer 1G is just a bit too low for JDK8+surefire 2.20+hdfs unit tests running in parallel. Bump it up a bit more. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15279) increase maven heap size recommendations
[ https://issues.apache.org/jira/browse/HADOOP-15279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-15279: -- Attachment: HADOOP-15279.00.patch > increase maven heap size recommendations > > > Key: HADOOP-15279 > URL: https://issues.apache.org/jira/browse/HADOOP-15279 > Project: Hadoop Common > Issue Type: Bug > Components: build, test >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Major > Attachments: HADOOP-15279.00.patch > > > 1G is just a bit too low for JDK8+surefire 2.20+hdfs unit tests running in > parallel. Bump it up a bit more. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15279) increase maven heap size recommendations
[ https://issues.apache.org/jira/browse/HADOOP-15279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-15279: -- Priority: Minor (was: Major) Issue Type: Improvement (was: Bug) > increase maven heap size recommendations > > > Key: HADOOP-15279 > URL: https://issues.apache.org/jira/browse/HADOOP-15279 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Attachments: HADOOP-15279.00.patch > > > 1G is just a bit too low for JDK8+surefire 2.20+hdfs unit tests running in > parallel. Bump it up a bit more. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15279) increase maven heap size recommendations
[ https://issues.apache.org/jira/browse/HADOOP-15279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-15279: -- Status: Patch Available (was: Open) > increase maven heap size recommendations > > > Key: HADOOP-15279 > URL: https://issues.apache.org/jira/browse/HADOOP-15279 > Project: Hadoop Common > Issue Type: Bug > Components: build, test >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Major > Attachments: HADOOP-15279.00.patch > > > 1G is just a bit too low for JDK8+surefire 2.20+hdfs unit tests running in > parallel. Bump it up a bit more. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15274) Move hadoop-openstack to slf4j
[ https://issues.apache.org/jira/browse/HADOOP-15274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382861#comment-16382861 ] Hudson commented on HADOOP-15274: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13753 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13753/]) HADOOP-15274. Move hadoop-openstack to slf4j (aajisaka: rev 29233c3c4a5ef6c7829177fef454c0052d852cc5) * (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeOutputStream.java * (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/util/SwiftTestUtils.java * (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeInputStream.java * (edit) hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestLogResources.java * (edit) hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftObjectPath.java * (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java * (edit) hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/http/TestSwiftRestClient.java * (edit) hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemBasicOps.java * (edit) hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemConcurrency.java * (edit) hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/SwiftFileSystemBaseTest.java * (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/ExceptionDiags.java * (edit) hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestReadPastBuffer.java * (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/util/SwiftUtils.java * (edit) hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSeek.java * (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/RestClientBindings.java * (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java * (edit) hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/TestSwiftFileSystemContract.java * (edit) hadoop-tools/hadoop-openstack/src/test/java/org/apache/hadoop/fs/swift/scale/TestWriteManySmallFiles.java * (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystemStore.java > Move hadoop-openstack to slf4j > -- > > Key: HADOOP-15274 > URL: https://issues.apache.org/jira/browse/HADOOP-15274 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/swift >Reporter: Steve Loughran >Assignee: fang zhenyi >Priority: Minor > Fix For: 3.2.0 > > Attachments: HADOOP-15274.001.patch, HADOOP-15274.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15279) increase maven heap size recommendations
[ https://issues.apache.org/jira/browse/HADOOP-15279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382804#comment-16382804 ] Chris Douglas commented on HADOOP-15279: +1 > increase maven heap size recommendations > > > Key: HADOOP-15279 > URL: https://issues.apache.org/jira/browse/HADOOP-15279 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Attachments: HADOOP-15279.00.patch > > > 1G is just a bit too low for JDK8+surefire 2.20+hdfs unit tests running in > parallel. Bump it up a bit more. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15261) Upgrade commons-io from 2.4 to 2.5
[ https://issues.apache.org/jira/browse/HADOOP-15261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-15261: --- Attachment: 347.patch > Upgrade commons-io from 2.4 to 2.5 > -- > > Key: HADOOP-15261 > URL: https://issues.apache.org/jira/browse/HADOOP-15261 > Project: Hadoop Common > Issue Type: Bug > Components: minikdc >Affects Versions: 3.0.0-alpha3 >Reporter: PandaMonkey >Priority: Major > Attachments: 347.patch, hadoop.txt > > > Hi, after analyzing hadoop-common-project\hadoop-minikdc\pom.xml, we found > that Hadoop depends on org.apache.kerby:kerb-simplekdc 1.0.1, which > transitivity introduced commons-io:2.5. > At the same time, hadoop directly depends on a older version of > commons-io:2.4. By further look into the source code, these two versions of > commons-io have many different features. The dependency conflict problem > brings high risks of "NotClassDefFoundError:" or "NoSuchMethodError" issues > at runtime. Please notice this problem. Maybe upgrading commons-io from 2.4 > to 2.5 is a good choice. Hope this report can help you. Thanks! > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15279) increase maven heap size recommendations
[ https://issues.apache.org/jira/browse/HADOOP-15279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-15279: -- Resolution: Fixed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) thanks for the review. committed. > increase maven heap size recommendations > > > Key: HADOOP-15279 > URL: https://issues.apache.org/jira/browse/HADOOP-15279 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Fix For: 3.2.0 > > Attachments: HADOOP-15279.00.patch > > > 1G is just a bit too low for JDK8+surefire 2.20+hdfs unit tests running in > parallel. Bump it up a bit more. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15279) increase maven heap size recommendations
[ https://issues.apache.org/jira/browse/HADOOP-15279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-15279: -- Component/s: documentation > increase maven heap size recommendations > > > Key: HADOOP-15279 > URL: https://issues.apache.org/jira/browse/HADOOP-15279 > Project: Hadoop Common > Issue Type: Improvement > Components: build, documentation, test >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Fix For: 3.2.0 > > Attachments: HADOOP-15279.00.patch > > > 1G is just a bit too low for JDK8+surefire 2.20+hdfs unit tests running in > parallel. Bump it up a bit more. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15277) remove .FluentPropertyBeanIntrospector from CLI operations
Steve Loughran created HADOOP-15277:
---
Summary: remove .FluentPropertyBeanIntrospector from CLI operations
Key: HADOOP-15277
URL: https://issues.apache.org/jira/browse/HADOOP-15277
Project: Hadoop Common
Issue Type: Improvement
Components: conf
Affects Versions: 3.1.0
Reporter: Steve Loughran
when using the default logs, I get told off by beanutils
{code}
18/03/01 18:43:54 INFO beanutils.FluentPropertyBeanIntrospector: Error when
creating PropertyDescriptor for public final void
org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
Ignoring this property.
{code}
This is a distraction.
I propose to raise the log level to ERROR for that class in log4j.properties
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382580#comment-16382580 ] Wangda Tan commented on HADOOP-13761: - [~fabbri]/[~ste...@apache.org], Given the other blockers are close to finish, how much more time needed for this JIRA? Asking this because I want to cut RC0 of 3.1.0 earlier if possible. > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, > HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch, > HADOOP-13761-006.patch, HADOOP-13761-007.patch, HADOOP-13761-008.patch, > HADOOP-13761-009.patch, HADOOP-13761-010.patch, HADOOP-13761-010.patch, > HADOOP-13761-011.patch, HADOOP-13761.001.patch, HADOOP-13761.002.patch, > HADOOP-13761.003.patch, HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15279) increase maven heap size recommendations
[
https://issues.apache.org/jira/browse/HADOOP-15279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382667#comment-16382667
]
genericqa commented on HADOOP-15279:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
12s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 40s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15279 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912650/HADOOP-15279.00.patch
|
| Optional Tests | asflicense shellcheck shelldocs |
| uname | Linux 40e7e9fb64eb 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 22928c0 |
| maven | version: Apache Maven 3.3.9 |
| shellcheck | v0.4.6 |
| Max. process+thread count | 333 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14236/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> increase maven heap size recommendations
>
>
> Key: HADOOP-15279
> URL: https://issues.apache.org/jira/browse/HADOOP-15279
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, test
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>Priority: Minor
> Attachments: HADOOP-15279.00.patch
>
>
> 1G is just a bit too low for JDK8+surefire 2.20+hdfs unit tests running in
> parallel. Bump it up a bit more.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15261) Upgrade commons-io from 2.4 to 2.5
[
https://issues.apache.org/jira/browse/HADOOP-15261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382838#comment-16382838
]
Akira Ajisaka commented on HADOOP-15261:
[https://builds.apache.org/job/PreCommit-HADOOP-Build/14237/console]
{noformat}
Processing: HADOOP-15261
HADOOP-15261 patch is being downloaded at Thu Mar 1 23:13:05 UTC 2018 from
https://issues.apache.org/jira/secure/attachment/12911995/hadoop.txt ->
Downloaded
ERROR: Unsure how to process HADOOP-15261.
{noformat}
GitHub integration seems to be failing. Attached the same patch as GitHub PR to
run precommit job.
> Upgrade commons-io from 2.4 to 2.5
> --
>
> Key: HADOOP-15261
> URL: https://issues.apache.org/jira/browse/HADOOP-15261
> Project: Hadoop Common
> Issue Type: Bug
> Components: minikdc
>Affects Versions: 3.0.0-alpha3
>Reporter: PandaMonkey
>Priority: Major
> Attachments: 347.patch, hadoop.txt
>
>
> Hi, after analyzing hadoop-common-project\hadoop-minikdc\pom.xml, we found
> that Hadoop depends on org.apache.kerby:kerb-simplekdc 1.0.1, which
> transitivity introduced commons-io:2.5.
> At the same time, hadoop directly depends on a older version of
> commons-io:2.4. By further look into the source code, these two versions of
> commons-io have many different features. The dependency conflict problem
> brings high risks of "NotClassDefFoundError:" or "NoSuchMethodError" issues
> at runtime. Please notice this problem. Maybe upgrading commons-io from 2.4
> to 2.5 is a good choice. Hope this report can help you. Thanks!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15279) increase maven heap size recommendations
[ https://issues.apache.org/jira/browse/HADOOP-15279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382687#comment-16382687 ] Jason Lowe commented on HADOOP-15279: - Thanks, Allen! +1 lgtm. > increase maven heap size recommendations > > > Key: HADOOP-15279 > URL: https://issues.apache.org/jira/browse/HADOOP-15279 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Attachments: HADOOP-15279.00.patch > > > 1G is just a bit too low for JDK8+surefire 2.20+hdfs unit tests running in > parallel. Bump it up a bit more. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15274) Move hadoop-openstack to slf4j
[ https://issues.apache.org/jira/browse/HADOOP-15274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-15274: --- Resolution: Fixed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) Committed this to trunk. Thanks [~zhenyi] for the contribution. > Move hadoop-openstack to slf4j > -- > > Key: HADOOP-15274 > URL: https://issues.apache.org/jira/browse/HADOOP-15274 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/swift >Reporter: Steve Loughran >Assignee: fang zhenyi >Priority: Minor > Fix For: 3.2.0 > > Attachments: HADOOP-15274.001.patch, HADOOP-15274.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15274) Move hadoop-openstack to slf4j
[ https://issues.apache.org/jira/browse/HADOOP-15274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-15274: --- Hadoop Flags: Reviewed Summary: Move hadoop-openstack to slf4j (was: move hadoop-openstack to slf4j) > Move hadoop-openstack to slf4j > -- > > Key: HADOOP-15274 > URL: https://issues.apache.org/jira/browse/HADOOP-15274 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/swift >Reporter: Steve Loughran >Assignee: fang zhenyi >Priority: Minor > Fix For: 3.2.0 > > Attachments: HADOOP-15274.001.patch, HADOOP-15274.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14652) Update metrics-core version to 3.2.4
[
https://issues.apache.org/jira/browse/HADOOP-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382739#comment-16382739
]
genericqa commented on HADOOP-14652:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
55m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
10s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 56m 21s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}181m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.webapp.TestContainerLogsPage |
| | hadoop.crypto.key.kms.server.TestKMS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14652 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912634/HADOOP-14652.006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 5bcda28b6985 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 22928c0 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14235/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14235/testReport/ |
| Max. process+thread count | 1351 (vs. ulimit of 1) |
| modules | C: hadoop-project hadoop-common-project/hadoop-kms
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-tools/hadoop-sls . U: . |
| Console output |
[jira] [Created] (HADOOP-15280) TestKMS.testWebHDFSProxyUserKerb and TestKMS.testWebHDFSProxyUserSimple fail in trunk
Ray Chiang created HADOOP-15280:
---
Summary: TestKMS.testWebHDFSProxyUserKerb and
TestKMS.testWebHDFSProxyUserSimple fail in trunk
Key: HADOOP-15280
URL: https://issues.apache.org/jira/browse/HADOOP-15280
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.2.0
Reporter: Ray Chiang
I'm seeing these messages on OS X and on Linux.
{noformat}
[ERROR] Failures:
[ERROR]
TestKMS.testWebHDFSProxyUserKerb:2526->doWebHDFSProxyUserTest:2625->runServer:158->runServer:176
org.apache.hadoop.security.authentication.client.AuthenticationException:
Error while authenticating with endpoint:
http://localhost:56112/kms/v1/keys?doAs=foo1
[ERROR]
TestKMS.testWebHDFSProxyUserSimple:2531->doWebHDFSProxyUserTest:2625->runServer:158->runServer:176
org.apache.hadoop.security.authentication.client.AuthenticationException:
Error while authenticating with endpoint:
http://localhost:56206/kms/v1/keys?doAs=foo1
{noformat}
as well as a [recent PreCommit-HADOOP-Build
job|https://builds.apache.org/job/PreCommit-HADOOP-Build/14235/].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14652) Update metrics-core version to 3.2.4
[ https://issues.apache.org/jira/browse/HADOOP-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382862#comment-16382862 ] Ray Chiang commented on HADOOP-14652: - Filed HADOOP-15280 for TestKMS unit test failures. Same tests fail for me in OS X and Linux, with and without the patch. TestContainerLogsPage passes for me in OS X and Linux, with and without the patch. > Update metrics-core version to 3.2.4 > > > Key: HADOOP-14652 > URL: https://issues.apache.org/jira/browse/HADOOP-14652 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0-beta1 >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Major > Attachments: HADOOP-14652.001.patch, HADOOP-14652.002.patch, > HADOOP-14652.003.patch, HADOOP-14652.004.patch, HADOOP-14652.005.patch, > HADOOP-14652.006.patch > > > The current artifact is: > com.codehale.metrics:metrics-core:3.0.1 > That version could either be bumped to 3.0.2 (the latest of that line), or > use the latest artifact: > io.dropwizard.metrics:metrics-core:3.2.4 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14652) Update metrics-core version to 3.2.4
[ https://issues.apache.org/jira/browse/HADOOP-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382476#comment-16382476 ] Ray Chiang commented on HADOOP-14652: - I had to rebase the patch. I'll get this checked in if Jenkins is clean. > Update metrics-core version to 3.2.4 > > > Key: HADOOP-14652 > URL: https://issues.apache.org/jira/browse/HADOOP-14652 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0-beta1 >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Major > Attachments: HADOOP-14652.001.patch, HADOOP-14652.002.patch, > HADOOP-14652.003.patch, HADOOP-14652.004.patch, HADOOP-14652.005.patch, > HADOOP-14652.006.patch > > > The current artifact is: > com.codehale.metrics:metrics-core:3.0.1 > That version could either be bumped to 3.0.2 (the latest of that line), or > use the latest artifact: > io.dropwizard.metrics:metrics-core:3.2.4 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15278) log s3a at info
Steve Loughran created HADOOP-15278: --- Summary: log s3a at info Key: HADOOP-15278 URL: https://issues.apache.org/jira/browse/HADOOP-15278 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.0.0 Reporter: Steve Loughran Assignee: Steve Loughran since it was added, hadoop conf/log4j only logs s3a at ERROR, even though in our test/resources it logs at info. We do actually log lots of stuff useful when debugging things Proposed: drop the log level to INFO here -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14652) Update metrics-core version to 3.2.4
[ https://issues.apache.org/jira/browse/HADOOP-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated HADOOP-14652: Attachment: HADOOP-14652.006.patch > Update metrics-core version to 3.2.4 > > > Key: HADOOP-14652 > URL: https://issues.apache.org/jira/browse/HADOOP-14652 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0-beta1 >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Major > Attachments: HADOOP-14652.001.patch, HADOOP-14652.002.patch, > HADOOP-14652.003.patch, HADOOP-14652.004.patch, HADOOP-14652.005.patch, > HADOOP-14652.006.patch > > > The current artifact is: > com.codehale.metrics:metrics-core:3.0.1 > That version could either be bumped to 3.0.2 (the latest of that line), or > use the latest artifact: > io.dropwizard.metrics:metrics-core:3.2.4 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14652) Update metrics-core version to 3.2.4
[ https://issues.apache.org/jira/browse/HADOOP-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382914#comment-16382914 ] Hudson commented on HADOOP-14652: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13755 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13755/]) HADOOP-14652. Update metrics-core version to 3.2.4. (rchiang) (rchiang: rev 923e1773e181ce125855639b882dc5a1a58e80ee) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * (edit) hadoop-tools/hadoop-sls/pom.xml * (edit) NOTICE.txt * (edit) hadoop-common-project/hadoop-kms/pom.xml * (edit) hadoop-project/pom.xml > Update metrics-core version to 3.2.4 > > > Key: HADOOP-14652 > URL: https://issues.apache.org/jira/browse/HADOOP-14652 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0-beta1 >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-14652.001.patch, HADOOP-14652.002.patch, > HADOOP-14652.003.patch, HADOOP-14652.004.patch, HADOOP-14652.005.patch, > HADOOP-14652.006.patch > > > The current artifact is: > com.codehale.metrics:metrics-core:3.0.1 > That version could either be bumped to 3.0.2 (the latest of that line), or > use the latest artifact: > io.dropwizard.metrics:metrics-core:3.2.4 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15261) Upgrade commons-io from 2.4 to 2.5
[
https://issues.apache.org/jira/browse/HADOOP-15261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382924#comment-16382924
]
genericqa commented on HADOOP-15261:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
47s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
28m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
11s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15261 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912684/347.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 3b0fb7997897 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d4cc50f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14238/testReport/ |
| Max. process+thread count | 340 (vs. ulimit of 1) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14238/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Upgrade commons-io from 2.4 to 2.5
> --
>
> Key: HADOOP-15261
> URL: https://issues.apache.org/jira/browse/HADOOP-15261
> Project: Hadoop Common
> Issue Type: Bug
> Components: minikdc
>Affects Versions: 3.0.0-alpha3
>Reporter: PandaMonkey
>Priority: Major
> Attachments: 347.patch, hadoop.txt
>
>
> Hi, after analyzing hadoop-common-project\hadoop-minikdc\pom.xml, we found
[jira] [Updated] (HADOOP-14652) Update metrics-core version to 3.2.4
[ https://issues.apache.org/jira/browse/HADOOP-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated HADOOP-14652: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks [~andrew.wang] and [~ste...@apache.org] for reviewing! > Update metrics-core version to 3.2.4 > > > Key: HADOOP-14652 > URL: https://issues.apache.org/jira/browse/HADOOP-14652 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0-beta1 >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-14652.001.patch, HADOOP-14652.002.patch, > HADOOP-14652.003.patch, HADOOP-14652.004.patch, HADOOP-14652.005.patch, > HADOOP-14652.006.patch > > > The current artifact is: > com.codehale.metrics:metrics-core:3.0.1 > That version could either be bumped to 3.0.2 (the latest of that line), or > use the latest artifact: > io.dropwizard.metrics:metrics-core:3.2.4 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14898) Create official Docker images for development and testing features
[
https://issues.apache.org/jira/browse/HADOOP-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382897#comment-16382897
]
Chris Douglas commented on HADOOP-14898:
lgtm. Do we need to change our [release
process|https://wiki.apache.org/hadoop/HowToRelease/] to update these branches?
Or will that happen implicitly? The patch in HADOOP-15256 looks like the
version is hard-coded in the Dockerfile, so I'm not sure I'm following how we
maintain the set of valid release targets from the ASF repo.
I'd expect that downstream, one could specify 2 (implicitly the latest 2.x.y),
2.8 (implicitly the latest 2.8.y) or 2.8.3 (exact). Is that the case?
> Create official Docker images for development and testing features
> ---
>
> Key: HADOOP-14898
> URL: https://issues.apache.org/jira/browse/HADOOP-14898
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HADOOP-14898.001.tar.gz, HADOOP-14898.002.tar.gz,
> HADOOP-14898.003.tgz, docker_design.pdf
>
>
> This is the original mail from the mailing list:
> {code}
> TL;DR: I propose to create official hadoop images and upload them to the
> dockerhub.
> GOAL/SCOPE: I would like improve the existing documentation with easy-to-use
> docker based recipes to start hadoop clusters with various configuration.
> The images also could be used to test experimental features. For example
> ozone could be tested easily with these compose file and configuration:
> https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> Or even the configuration could be included in the compose file:
> https://github.com/elek/hadoop/blob/docker-2.8.0/example/docker-compose.yaml
> I would like to create separated example compose files for federation, ha,
> metrics usage, etc. to make it easier to try out and understand the features.
> CONTEXT: There is an existing Jira
> https://issues.apache.org/jira/browse/HADOOP-13397
> But it’s about a tool to generate production quality docker images (multiple
> types, in a flexible way). If no objections, I will create a separated issue
> to create simplified docker images for rapid prototyping and investigating
> new features. And register the branch to the dockerhub to create the images
> automatically.
> MY BACKGROUND: I am working with docker based hadoop/spark clusters quite a
> while and run them succesfully in different environments (kubernetes,
> docker-swarm, nomad-based scheduling, etc.) My work is available from here:
> https://github.com/flokkr but they could handle more complex use cases (eg.
> instrumenting java processes with btrace, or read/reload configuration from
> consul).
> And IMHO in the official hadoop documentation it’s better to suggest to use
> official apache docker images and not external ones (which could be changed).
> {code}
> The next list will enumerate the key decision points regarding to docker
> image creating
> A. automated dockerhub build / jenkins build
> Docker images could be built on the dockerhub (a branch pattern should be
> defined for a github repository and the location of the Docker files) or
> could be built on a CI server and pushed.
> The second one is more flexible (it's more easy to create matrix build, for
> example)
> The first one had the advantage that we can get an additional flag on the
> dockerhub that the build is automated (and built from the source by the
> dockerhub).
> The decision is easy as ASF supports the first approach: (see
> https://issues.apache.org/jira/browse/INFRA-12781?focusedCommentId=15824096=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15824096)
> B. source: binary distribution or source build
> The second question is about creating the docker image. One option is to
> build the software on the fly during the creation of the docker image the
> other one is to use the binary releases.
> I suggest to use the second approach as:
> 1. In that case the hadoop:2.7.3 could contain exactly the same hadoop
> distrubution as the downloadable one
> 2. We don't need to add development tools to the image, the image could be
> more smaller (which is important as the goal for this image to getting
> started as fast as possible)
> 3. The docker definition will be more simple (and more easy to maintain)
> Usually this approach is used in other projects (I checked Apache Zeppelin
> and Apache Nutch)
> C. branch usage
> Other question is the location of the Docker file. It could be on the
> official source-code branches (branch-2, trunk, etc.) or we can create
> separated branches for the dockerhub (eg. docker/2.7 docker/2.8 docker/3.0)
> For the first approach it's easier to find the docker images, but it's less
>
[jira] [Commented] (HADOOP-15279) increase maven heap size recommendations
[ https://issues.apache.org/jira/browse/HADOOP-15279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382896#comment-16382896 ] Hudson commented on HADOOP-15279: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13754 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13754/]) HADOOP-15279. increase maven heap size recommendations (aw: rev d4cc50f62279dc4d607cd05c16de480a76f05880) * (edit) BUILDING.txt * (edit) dev-support/docker/Dockerfile > increase maven heap size recommendations > > > Key: HADOOP-15279 > URL: https://issues.apache.org/jira/browse/HADOOP-15279 > Project: Hadoop Common > Issue Type: Improvement > Components: build, documentation, test >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Fix For: 3.2.0 > > Attachments: HADOOP-15279.00.patch > > > 1G is just a bit too low for JDK8+surefire 2.20+hdfs unit tests running in > parallel. Bump it up a bit more. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15090) Add ADL troubleshooting doc
[ https://issues.apache.org/jira/browse/HADOOP-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HADOOP-15090: -- Attachment: HADOOP-15090-branch-2.002.patch > Add ADL troubleshooting doc > --- > > Key: HADOOP-15090 > URL: https://issues.apache.org/jira/browse/HADOOP-15090 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/adl >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.1.0, 3.0.1 > > Attachments: HADOOP-15090-001.patch, HADOOP-15090-branch-2.002.patch > > > Add a troubleshooting section/doc to the ADL docs based on our experiences. > this should not be a substitute for improving the diagnostics/fixing the > error messages. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15090) Add ADL troubleshooting doc
[ https://issues.apache.org/jira/browse/HADOOP-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382907#comment-16382907 ] Masatake Iwasaki commented on HADOOP-15090: --- Leading whitespace in 56 seems to be the cause. Removing it fixed the issue on my env. I attached HADOOP-15090-branch-2.002.patch. > Add ADL troubleshooting doc > --- > > Key: HADOOP-15090 > URL: https://issues.apache.org/jira/browse/HADOOP-15090 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/adl >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.1.0, 3.0.1 > > Attachments: HADOOP-15090-001.patch, HADOOP-15090-branch-2.002.patch > > > Add a troubleshooting section/doc to the ADL docs based on our experiences. > this should not be a substitute for improving the diagnostics/fixing the > error messages. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10584) ActiveStandbyElector goes down if ZK quorum become unavailable
[ https://issues.apache.org/jira/browse/HADOOP-10584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383045#comment-16383045 ] SammiChen commented on HADOOP-10584: Hi [~templedf], does this still target for 2.9.1? If not, can we push this out to next 2.9.2 release? > ActiveStandbyElector goes down if ZK quorum become unavailable > -- > > Key: HADOOP-10584 > URL: https://issues.apache.org/jira/browse/HADOOP-10584 > Project: Hadoop Common > Issue Type: Bug > Components: ha >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Assignee: Daniel Templeton >Priority: Major > Attachments: HADOOP-10584.prelim.patch, hadoop-10584-prelim.patch, > rm.log > > > ActiveStandbyElector retries operations for a few times. If the ZK quorum > itself is down, it goes down and the daemons will have to be brought up > again. > Instead, it should log the fact that it is unable to talk to ZK, call > becomeStandby on its client, and continue to attempt connecting to ZK. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15099) YARN Federation Link not working
[ https://issues.apache.org/jira/browse/HADOOP-15099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383077#comment-16383077 ] SammiChen commented on HADOOP-15099: Hi [~animenon], thanks for reporting this. Would you create a patch using Git and upload to this Jira through more->attach files. I can help you commit into the upstream. > YARN Federation Link not working > > > Key: HADOOP-15099 > URL: https://issues.apache.org/jira/browse/HADOOP-15099 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 2.9.0 >Reporter: Anirudh >Priority: Trivial > Labels: documentation, easyfix, newbie > Original Estimate: 1m > Remaining Estimate: 1m > > YARN federation(in the last paragraph on the page) link isn't working on > [http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383066#comment-16383066 ] Aaron Fabbri commented on HADOOP-13761: --- Thanks [~leftnoteasy] I've been travelling week, so sorry for the delay. I should be able to get out a new patch by Friday morning pacific time. Not sure how quick we can turn around a code review and testing. When do you want to cut RC0? > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, > HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch, > HADOOP-13761-006.patch, HADOOP-13761-007.patch, HADOOP-13761-008.patch, > HADOOP-13761-009.patch, HADOOP-13761-010.patch, HADOOP-13761-010.patch, > HADOOP-13761-011.patch, HADOOP-13761.001.patch, HADOOP-13761.002.patch, > HADOOP-13761.003.patch, HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15262) AliyunOSS: rename() to move files in a directory in parallel
[ https://issues.apache.org/jira/browse/HADOOP-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15262: - Issue Type: Sub-task (was: Improvement) Parent: HADOOP-13377 > AliyunOSS: rename() to move files in a directory in parallel > > > Key: HADOOP-15262 > URL: https://issues.apache.org/jira/browse/HADOOP-15262 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 3.1.0, 2.9.1, 3.0.1 > > Attachments: HADOOP-15262.001.patch, HADOOP-15262.002.patch, > HADOOP-15262.003.patch > > > Currently, rename() operation renames files in series. This will be slow if a > directory contains many files. So we can improve this by rename files in > parallel. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15099) YARN Federation Link not working
[ https://issues.apache.org/jira/browse/HADOOP-15099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HADOOP-15099: --- Target Version/s: 3.0.0 (was: 2.9.1) > YARN Federation Link not working > > > Key: HADOOP-15099 > URL: https://issues.apache.org/jira/browse/HADOOP-15099 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 2.9.0 >Reporter: Anirudh >Priority: Trivial > Labels: documentation, easyfix, newbie > Original Estimate: 1m > Remaining Estimate: 1m > > YARN federation(in the last paragraph on the page) link isn't working on > [http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15099) YARN Federation Link not working
[ https://issues.apache.org/jira/browse/HADOOP-15099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383188#comment-16383188 ] Anirudh edited comment on HADOOP-15099 at 3/2/18 5:22 AM: -- [~Sammi] Thanks, looks like somebody pushed the change. Fixed: [https://github.com/apache/hadoop/commit/fbbbf59c82658e18dad7e0e256613187b5b75d0f] was (Author: animenon): Fixed: https://github.com/apache/hadoop/commit/fbbbf59c82658e18dad7e0e256613187b5b75d0f > YARN Federation Link not working > > > Key: HADOOP-15099 > URL: https://issues.apache.org/jira/browse/HADOOP-15099 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 2.9.0 >Reporter: Anirudh >Priority: Trivial > Labels: documentation, easyfix, newbie > Original Estimate: 1m > Remaining Estimate: 1m > > YARN federation(in the last paragraph on the page) link isn't working on > [http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15099) YARN Federation Link not working
[ https://issues.apache.org/jira/browse/HADOOP-15099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383186#comment-16383186 ] ASF GitHub Bot commented on HADOOP-15099: - Github user animenon closed the pull request at: https://github.com/apache/hadoop/pull/300 > YARN Federation Link not working > > > Key: HADOOP-15099 > URL: https://issues.apache.org/jira/browse/HADOOP-15099 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 2.9.0 >Reporter: Anirudh >Priority: Trivial > Labels: documentation, easyfix, newbie > Original Estimate: 1m > Remaining Estimate: 1m > > YARN federation(in the last paragraph on the page) link isn't working on > [http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15099) YARN Federation Link not working
[ https://issues.apache.org/jira/browse/HADOOP-15099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383185#comment-16383185 ] ASF GitHub Bot commented on HADOOP-15099: - Github user animenon commented on the issue: https://github.com/apache/hadoop/pull/300 Fixed: https://github.com/apache/hadoop/commit/fbbbf59c82658e18dad7e0e256613187b5b75d0f > YARN Federation Link not working > > > Key: HADOOP-15099 > URL: https://issues.apache.org/jira/browse/HADOOP-15099 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 2.9.0 >Reporter: Anirudh >Priority: Trivial > Labels: documentation, easyfix, newbie > Original Estimate: 1m > Remaining Estimate: 1m > > YARN federation(in the last paragraph on the page) link isn't working on > [http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15099) YARN Federation Link not working
[ https://issues.apache.org/jira/browse/HADOOP-15099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383187#comment-16383187 ] Anirudh commented on HADOOP-15099: -- Fixed: https://github.com/apache/hadoop/commit/fbbbf59c82658e18dad7e0e256613187b5b75d0f > YARN Federation Link not working > > > Key: HADOOP-15099 > URL: https://issues.apache.org/jira/browse/HADOOP-15099 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 2.9.0 >Reporter: Anirudh >Priority: Trivial > Labels: documentation, easyfix, newbie > Original Estimate: 1m > Remaining Estimate: 1m > > YARN federation(in the last paragraph on the page) link isn't working on > [http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: HADOOP-13761-012.patch > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, > HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch, > HADOOP-13761-006.patch, HADOOP-13761-007.patch, HADOOP-13761-008.patch, > HADOOP-13761-009.patch, HADOOP-13761-010.patch, HADOOP-13761-010.patch, > HADOOP-13761-011.patch, HADOOP-13761-012.patch, HADOOP-13761.001.patch, > HADOOP-13761.002.patch, HADOOP-13761.003.patch, HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15099) YARN Federation Link not working
[ https://issues.apache.org/jira/browse/HADOOP-15099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh updated HADOOP-15099: - Resolution: Fixed Status: Resolved (was: Patch Available) Fixed: https://github.com/apache/hadoop/commit/fbbbf59c82658e18dad7e0e256613187b5b75d0f > YARN Federation Link not working > > > Key: HADOOP-15099 > URL: https://issues.apache.org/jira/browse/HADOOP-15099 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 2.9.0 >Reporter: Anirudh >Priority: Trivial > Labels: documentation, easyfix, newbie > Original Estimate: 1m > Remaining Estimate: 1m > > YARN federation(in the last paragraph on the page) link isn't working on > [http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-15099) YARN Federation Link not working
[ https://issues.apache.org/jira/browse/HADOOP-15099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh updated HADOOP-15099: - Comment: was deleted (was: Fixed: https://github.com/apache/hadoop/commit/fbbbf59c82658e18dad7e0e256613187b5b75d0f) > YARN Federation Link not working > > > Key: HADOOP-15099 > URL: https://issues.apache.org/jira/browse/HADOOP-15099 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 2.9.0 >Reporter: Anirudh >Priority: Trivial > Labels: documentation, easyfix, newbie > Original Estimate: 1m > Remaining Estimate: 1m > > YARN federation(in the last paragraph on the page) link isn't working on > [http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383163#comment-16383163 ] Wangda Tan commented on HADOOP-13761: - Thanks [~fabbri] for help, we have last two blockers, this is one of them, if everything goes very well, we can get RC0 by end of this week or early next week. Just want to make sure this is not the last one :) > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, > HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch, > HADOOP-13761-006.patch, HADOOP-13761-007.patch, HADOOP-13761-008.patch, > HADOOP-13761-009.patch, HADOOP-13761-010.patch, HADOOP-13761-010.patch, > HADOOP-13761-011.patch, HADOOP-13761-012.patch, HADOOP-13761.001.patch, > HADOOP-13761.002.patch, HADOOP-13761.003.patch, HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[
https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383164#comment-16383164
]
Aaron Fabbri commented on HADOOP-13761:
---
v12 patch attached which eliminates nested retries. Changes from v11:
- reopen() RetryTranslated -> OnceTranslated. Use once() instead of retry().
All callers (lazySeek() and callers of onReadFailure()) have their own
higher-level retries.
- onReadFailure() RetryTranslated -> OnceTranslated, because reopen() no longer
has internal retries.
- Move lazySeek() in read() above the retry'd lambda; eliminates nested retry
and matches what read(b, off, len) does.
- Remove reopen() from read(b, off, len) after wrapped stream throws EOF.
*This was existing code I changed*: I don't understand why you'd reopen() on
EOF, [~ste...@apache.org]?
Tested in us-west-2. I was able to reproduce the test timeout for
ITestS3AFailureHandling, which is now passing for me.
Running on little sleep but wanted to get a patch posted to give a chance for
European Friday code review.
Here is the delta from v11:
{noformat}
diff -ur ./src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
/Users/fabbri/Code/hadoop/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
--- ./src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
2018-03-01 20:34:33.0 -0800
+++
/Users/fabbri/Code/hadoop/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
2018-03-01 20:09:05.0 -0800
@@ -154,7 +154,7 @@
* @param length length requested
* @throws IOException on any failure to open the object
*/
- @Retries.RetryTranslated
+ @Retries.OnceTranslated
private synchronized void reopen(String reason, long targetPos, long length)
throws IOException {
@@ -178,7 +178,7 @@
}
String text = String.format("Failed to %s %s at %d",
(opencount == 0 ? "open" : "re-open"), uri, targetPos);
-S3Object object = context.invoker.retry(text, uri, true,
+S3Object object = context.getReadInvoker().once(text, uri,
() -> client.getObject(request));
wrappedStream = object.getObjectContent();
contentRangeStart = targetPos;
@@ -349,6 +349,12 @@
return -1;
}
+try {
+ lazySeek(nextReadPos, 1);
+} catch (EOFException e) {
+ return -1;
+}
+
// With S3Guard, the metadatastore gave us metadata for the file in
// open(), so we use a slightly different retry policy.
// read() may not be likely to fail, but reopen() does a GET which
@@ -358,7 +364,6 @@
() -> {
int b;
try {
-lazySeek(nextReadPos, 1);
b = wrappedStream.read();
} catch (EOFException e) {
return -1;
@@ -387,7 +392,7 @@
* @param length length of data being attempted to read
* @throws IOException any exception thrown on the re-open attempt.
*/
- @Retries.RetryTranslated
+ @Retries.OnceTranslated
private void onReadFailure(IOException ioe, int length) throws IOException {
LOG.info("Got exception while trying to read from stream {}" +
@@ -439,7 +444,6 @@
try {
bytes = wrappedStream.read(buf, off, len);
} catch (EOFException e) {
-onReadFailure(e, len);
// the base implementation swallows EOFs.
return -1;
} catch (IOException e) {
{noformat}
> S3Guard: implement retries for DDB failures and throttling; translate
> exceptions
>
>
> Key: HADOOP-13761
> URL: https://issues.apache.org/jira/browse/HADOOP-13761
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
>Priority: Blocker
> Attachments: HADOOP-13761-004-to-005.patch,
> HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch,
> HADOOP-13761-006.patch, HADOOP-13761-007.patch, HADOOP-13761-008.patch,
> HADOOP-13761-009.patch, HADOOP-13761-010.patch, HADOOP-13761-010.patch,
> HADOOP-13761-011.patch, HADOOP-13761-012.patch, HADOOP-13761.001.patch,
> HADOOP-13761.002.patch, HADOOP-13761.003.patch, HADOOP-13761.004.patch
>
>
> Following the S3AFileSystem integration patch in HADOOP-13651, we need to add
> retry logic.
> In HADOOP-13651, I added TODO comments in most of the places retry loops are
> needed, including:
> - open(path). If MetadataStore reflects recent create/move of file path, but
> we fail to read it from S3, retry.
> - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the
> file exists, retry.
> - rename(src,dest). If source path is not visible in S3 yet, retry.
> - listFiles(). Skip for now. Not
[jira] [Commented] (HADOOP-14898) Create official Docker images for development and testing features
[
https://issues.apache.org/jira/browse/HADOOP-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382311#comment-16382311
]
Elek, Marton commented on HADOOP-14898:
---
Just some minor clarification:
* Images are built by dockerhub. INFRA can register the branches on dockerhub
and dockerhub will automatically fetch the latest branch and build the images
on change. (From the docker-* branches not the real branches which contain only
the Dockerfile files)
* I proposed to use the binary releases for the image creation which are voted
by PMC members. The image will contain exactly the same Hadoop. There *won't be
and mvn build * during the docker image creation just some packaging (download
hadoop and extract it).
* The hadoop-runner images won't be released with every hadoop release.
Ideally we need to creat it once. As it contains only a startup script I think
we can handle it outside of the release structure (this is how it is handled by
other Apache projects).
* HADOOP-15259 will provide a Dockerfile for developer images with snapshot
version of Hadoop *locally*. They won't be uploaded to the dockerhub.
Summary:
* apache/hadoop-runner:latest -- contains only the base OS + a very simple
starter script . Only latest version is available, no releases.
* apache/hadoop:2 -- contains the latest stable hadoop 2 release (official
binary release, approved by the PMC) on top of hadoop-runner:latest
* apache/hadoop:3 -- same, but with the latest hadoop 3 release. (It should be
updated *after* the release vote with the official download link)
I would be more than happy to participate on a call if you are interested about
how it could be used or discuss the implementation. Please propose a time
period or we can also continue the discussion here.
> Create official Docker images for development and testing features
> ---
>
> Key: HADOOP-14898
> URL: https://issues.apache.org/jira/browse/HADOOP-14898
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HADOOP-14898.001.tar.gz, HADOOP-14898.002.tar.gz,
> HADOOP-14898.003.tgz, docker_design.pdf
>
>
> This is the original mail from the mailing list:
> {code}
> TL;DR: I propose to create official hadoop images and upload them to the
> dockerhub.
> GOAL/SCOPE: I would like improve the existing documentation with easy-to-use
> docker based recipes to start hadoop clusters with various configuration.
> The images also could be used to test experimental features. For example
> ozone could be tested easily with these compose file and configuration:
> https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> Or even the configuration could be included in the compose file:
> https://github.com/elek/hadoop/blob/docker-2.8.0/example/docker-compose.yaml
> I would like to create separated example compose files for federation, ha,
> metrics usage, etc. to make it easier to try out and understand the features.
> CONTEXT: There is an existing Jira
> https://issues.apache.org/jira/browse/HADOOP-13397
> But it’s about a tool to generate production quality docker images (multiple
> types, in a flexible way). If no objections, I will create a separated issue
> to create simplified docker images for rapid prototyping and investigating
> new features. And register the branch to the dockerhub to create the images
> automatically.
> MY BACKGROUND: I am working with docker based hadoop/spark clusters quite a
> while and run them succesfully in different environments (kubernetes,
> docker-swarm, nomad-based scheduling, etc.) My work is available from here:
> https://github.com/flokkr but they could handle more complex use cases (eg.
> instrumenting java processes with btrace, or read/reload configuration from
> consul).
> And IMHO in the official hadoop documentation it’s better to suggest to use
> official apache docker images and not external ones (which could be changed).
> {code}
> The next list will enumerate the key decision points regarding to docker
> image creating
> A. automated dockerhub build / jenkins build
> Docker images could be built on the dockerhub (a branch pattern should be
> defined for a github repository and the location of the Docker files) or
> could be built on a CI server and pushed.
> The second one is more flexible (it's more easy to create matrix build, for
> example)
> The first one had the advantage that we can get an additional flag on the
> dockerhub that the build is automated (and built from the source by the
> dockerhub).
> The decision is easy as ASF supports the first approach: (see
>
[jira] [Updated] (HADOOP-15277) remove .FluentPropertyBeanIntrospector from CLI operation log output
[
https://issues.apache.org/jira/browse/HADOOP-15277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15277:
Summary: remove .FluentPropertyBeanIntrospector from CLI operation log
output (was: remove .FluentPropertyBeanIntrospector from CLI operations)
> remove .FluentPropertyBeanIntrospector from CLI operation log output
>
>
> Key: HADOOP-15277
> URL: https://issues.apache.org/jira/browse/HADOOP-15277
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Major
>
> when using the default logs, I get told off by beanutils
> {code}
> 18/03/01 18:43:54 INFO beanutils.FluentPropertyBeanIntrospector: Error when
> creating PropertyDescriptor for public final void
> org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
> Ignoring this property.
> {code}
> This is a distraction.
> I propose to raise the log level to ERROR for that class in log4j.properties
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
