[jira] [Work logged] (HADOOP-18065) ExecutorHelper.logThrowableFromAfterExecute() is too noisy.
[
https://issues.apache.org/jira/browse/HADOOP-18065?focusedWorklogId=704414=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-704414
]
ASF GitHub Bot logged work on HADOOP-18065:
---
Author: ASF GitHub Bot
Created on: 06/Jan/22 05:25
Start Date: 06/Jan/22 05:25
Worklog Time Spent: 10m
Work Description: mukund-thakur commented on pull request #3860:
URL: https://github.com/apache/hadoop/pull/3860#issuecomment-1006294876
Yetus failure is just because no new tests are added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 704414)
Time Spent: 40m (was: 0.5h)
> ExecutorHelper.logThrowableFromAfterExecute() is too noisy.
>
>
> Key: HADOOP-18065
> URL: https://issues.apache.org/jira/browse/HADOOP-18065
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.1
>Reporter: Mukund Thakur
>Assignee: Mukund Thakur
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> {code:java}
> if (t == null && r instanceof Future && ((Future) r).isDone()) {
> try {
> ((Future) r).get();
> } catch (ExecutionException ee) {
> LOG.warn(
> "Execution exception when running task in " + Thread.currentThread()
> .getName());
> t = ee.getCause();
> } catch (InterruptedException ie) {
> LOG.warn("Thread (" + Thread.currentThread() + ") interrupted: ", ie);
> Thread.currentThread().interrupt();
> } catch (Throwable throwable) {
> t = throwable;
> }
> }
> if (t != null) {
> LOG.warn("Caught exception in thread " + Thread
> .currentThread().getName() + ": ", t);
> } {code}
> We should downgrade the logging here from warn to debug.
>
> CC [~ste...@apache.org] [~mehakmeetSingh]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on pull request #3860: HADOOP-18065 ExecutorHelper.logThrowableFromAfterExecute() is too noisy.
mukund-thakur commented on pull request #3860: URL: https://github.com/apache/hadoop/pull/3860#issuecomment-1006294876 Yetus failure is just because no new tests are added. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18065) ExecutorHelper.logThrowableFromAfterExecute() is too noisy.
[
https://issues.apache.org/jira/browse/HADOOP-18065?focusedWorklogId=704413=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-704413
]
ASF GitHub Bot logged work on HADOOP-18065:
---
Author: ASF GitHub Bot
Created on: 06/Jan/22 05:24
Start Date: 06/Jan/22 05:24
Worklog Time Spent: 10m
Work Description: mukund-thakur merged pull request #3860:
URL: https://github.com/apache/hadoop/pull/3860
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 704413)
Time Spent: 0.5h (was: 20m)
> ExecutorHelper.logThrowableFromAfterExecute() is too noisy.
>
>
> Key: HADOOP-18065
> URL: https://issues.apache.org/jira/browse/HADOOP-18065
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.1
>Reporter: Mukund Thakur
>Assignee: Mukund Thakur
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:java}
> if (t == null && r instanceof Future && ((Future) r).isDone()) {
> try {
> ((Future) r).get();
> } catch (ExecutionException ee) {
> LOG.warn(
> "Execution exception when running task in " + Thread.currentThread()
> .getName());
> t = ee.getCause();
> } catch (InterruptedException ie) {
> LOG.warn("Thread (" + Thread.currentThread() + ") interrupted: ", ie);
> Thread.currentThread().interrupt();
> } catch (Throwable throwable) {
> t = throwable;
> }
> }
> if (t != null) {
> LOG.warn("Caught exception in thread " + Thread
> .currentThread().getName() + ": ", t);
> } {code}
> We should downgrade the logging here from warn to debug.
>
> CC [~ste...@apache.org] [~mehakmeetSingh]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur merged pull request #3860: HADOOP-18065 ExecutorHelper.logThrowableFromAfterExecute() is too noisy.
mukund-thakur merged pull request #3860: URL: https://github.com/apache/hadoop/pull/3860 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3155: HDFS-16095. Add lsQuotaList command and getQuotaListing api for hdfs …
hadoop-yetus commented on pull request #3155:
URL: https://github.com/apache/hadoop/pull/3155#issuecomment-1006289619
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 27s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 42s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 9s | | trunk passed |
| +1 :green_heart: | compile | 24m 3s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 30s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 2s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 51s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 5m 10s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 10m 3s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 42s | | the patch passed |
| +1 :green_heart: | compile | 23m 13s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | cc | 23m 13s |
[/results-compile-cc-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3155/1/artifact/out/results-compile-cc-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
| root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 23 new + 300 unchanged - 23
fixed = 323 total (was 323) |
| -1 :x: | javac | 23m 13s |
[/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3155/1/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt)
| root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 1 new + 1933 unchanged - 0
fixed = 1934 total (was 1933) |
| +1 :green_heart: | compile | 20m 26s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | cc | 20m 26s |
[/results-compile-cc-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3155/1/artifact/out/results-compile-cc-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 322
unchanged - 1 fixed = 323 total (was 323) |
| -1 :x: | javac | 20m 26s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3155/1/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 1807
unchanged - 0 fixed = 1808 total (was 1807) |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3155/1/artifact/out/blanks-eol.txt)
| The patch has 3 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 3m 58s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3155/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 13 new + 565 unchanged - 0 fixed = 578 total (was
565) |
| +1 :green_heart: | mvnsite | 5m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 50s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 5m 5s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1
[jira] [Created] (HADOOP-18071) ABFS: Set driver global timeout for ITestAzureBlobFileSystemBasics
Sumangala Patki created HADOOP-18071: Summary: ABFS: Set driver global timeout for ITestAzureBlobFileSystemBasics Key: HADOOP-18071 URL: https://issues.apache.org/jira/browse/HADOOP-18071 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.1 Reporter: Sumangala Patki Assignee: Sumangala Patki Unlike all other ABFS driver tests that have a timeout of 15min, ITestAzureBlobFileSystemBasics times out after 30s due to the global timeout inherited from FileSystemContractBaseTest. Setting a 15min timeout for the test will ensure sufficient time to allow retries and avoid transient failures. Example failure: testListOnFolderWithNoChildren(org.apache.hadoop.fs.azurebfs.contract.ITestAzureBlobFileSystemBasics) Time elapsed: 34.655 s <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 3 milliseconds -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18070) libhdfspp authentication failed
wondertx created HADOOP-18070: - Summary: libhdfspp authentication failed Key: HADOOP-18070 URL: https://issues.apache.org/jira/browse/HADOOP-18070 Project: Hadoop Common Issue Type: Bug Components: hdfs-client Affects Versions: 3.3.0 Reporter: wondertx I build the example lihdfspp client code in [https://github.com/apache/hadoop/blob/rel/release-3.3.0/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cc/cat/cat.cc] When running, the following error happened. While I can successfully connect to the same cluster using example code from C API from [https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/LibHdfs.html] [WARN ][RPC ][Thu Jan 6 11:47:12 2022][Thread id = 139810402694976][/tmp/orc/build/libhdfspp_ep-prefix/src/libhdfspp_ep/lib/rpc/namenode_tracker.cc:50] Nameservice declares more than two nodes. Some won't be used. [ERROR ][RPC ][Thu Jan 6 11:47:12 2022][Thread id = 139810357286656][/tmp/orc/build/libhdfspp_ep-prefix/src/libhdfspp_ep/lib/rpc/rpc_engine.cc:191] RpcEngine::AsyncRpcCommsError called; status="AuthenticationFailed" conn=0x13ab6e0 reqs=1 [WARN ][RPC ][Thu Jan 6 11:47:12 2022][Thread id = 139810357286656][/tmp/orc/build/libhdfspp_ep-prefix/src/libhdfspp_ep/lib/rpc/rpc_engine.cc:202] RpcEngine::RpcCommsError called; status="AuthenticationFailed" conn=0x13ab6e0 reqs=1 [WARN ][RPC ][Thu Jan 6 11:47:12 2022][Thread id = 139810357286656][/tmp/orc/build/libhdfspp_ep-prefix/src/libhdfspp_ep/lib/rpc/rpc_connection_impl.h:387] Network error during RPC read: Bad file descriptor [ERROR ][RPC ][Thu Jan 6 11:47:12 2022][Thread id = 139810357286656][/tmp/orc/build/libhdfspp_ep-prefix/src/libhdfspp_ep/lib/rpc/rpc_engine.cc:191] RpcEngine::AsyncRpcCommsError called; status="Bad file descriptor" conn=0x13ab6e0 reqs=0 [WARN ][RPC ][Thu Jan 6 11:47:12 2022][Thread id = 139810357286656][/tmp/orc/build/libhdfspp_ep-prefix/src/libhdfspp_ep/lib/rpc/rpc_engine.cc:202] RpcEngine::RpcCommsError called; status="Bad file descriptor" conn=0x13ab6e0 reqs=0 Could not connect to jssz-bigdata-proxy-ns1:. AuthenticationFailed -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] (HADOOP-17971) Exclude IBM Java security classes from being shaded/relocated
[ https://issues.apache.org/jira/browse/HADOOP-17971 ]
Michail Giannakopoulos deleted comment on HADOOP-17971:
-
was (Author: miccagiann):
Can we backport this fix to hadoop-client 2.10.0 stream. I have the same issue
when trying to use Apache Spark 3.2.0 and hadoop 2.10.1.
> Exclude IBM Java security classes from being shaded/relocated
> -
>
> Key: HADOOP-17971
> URL: https://issues.apache.org/jira/browse/HADOOP-17971
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.3.1
>Reporter: Nicholas Marion
>Assignee: Nicholas Marion
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.3, 3.3.2
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> IBM Java classes are shaded in Hadoop libraries, e.g. hadoop-client-api. When
> loaded by Spark, UserGroupInformation has exception:
> {noformat}
> org.apache.hadoop.security.KerberosAuthException: failure to login:
> javax.security.auth.login.LoginException: unable to find LoginModule class:
> org.apache.hadoop.shaded.com.ibm.security.auth.module.JAASLoginModule
> at
> org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1986)
> at
> org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:719)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:669)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:579)
> at
> org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2609)
> at
> org.apache.spark.util.Utils$$$Lambda$1388/0x338e9c30.apply(Unknown
> Source)
> at scala.Option.getOrElse(Option.scala:189)
> {noformat}
> When I manually compile UserGroupInformation.java without maven (aka
> relocation) and inject the class files into hadoop-client-api jar; it works.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17971) Exclude IBM Java security classes from being shaded/relocated
[
https://issues.apache.org/jira/browse/HADOOP-17971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469651#comment-17469651
]
Michail Giannakopoulos commented on HADOOP-17971:
-
Can we backport this fix to hadoop-client 2.10.0 stream. I have the same issue
when trying to use Apache Spark 3.2.0 and hadoop 2.10.1.
> Exclude IBM Java security classes from being shaded/relocated
> -
>
> Key: HADOOP-17971
> URL: https://issues.apache.org/jira/browse/HADOOP-17971
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.3.1
>Reporter: Nicholas Marion
>Assignee: Nicholas Marion
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.3, 3.3.2
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> IBM Java classes are shaded in Hadoop libraries, e.g. hadoop-client-api. When
> loaded by Spark, UserGroupInformation has exception:
> {noformat}
> org.apache.hadoop.security.KerberosAuthException: failure to login:
> javax.security.auth.login.LoginException: unable to find LoginModule class:
> org.apache.hadoop.shaded.com.ibm.security.auth.module.JAASLoginModule
> at
> org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1986)
> at
> org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:719)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:669)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:579)
> at
> org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2609)
> at
> org.apache.spark.util.Utils$$$Lambda$1388/0x338e9c30.apply(Unknown
> Source)
> at scala.Option.getOrElse(Option.scala:189)
> {noformat}
> When I manually compile UserGroupInformation.java without maven (aka
> relocation) and inject the class files into hadoop-client-api jar; it works.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] cndaimin commented on pull request #3842: HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background
cndaimin commented on pull request #3842: URL: https://github.com/apache/hadoop/pull/3842#issuecomment-1006249942 @sodonnel I have removed the unrelated code, could you please take a look again? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3861: HDFS-16316.Improve DirectoryScanner: add regular file check related block.
hadoop-yetus commented on pull request #3861:
URL: https://github.com/apache/hadoop/pull/3861#issuecomment-1006241356
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 56s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 41s | | trunk passed |
| +1 :green_heart: | compile | 23m 33s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 41s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 3s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 24s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 26s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 17s | | the patch passed |
| +1 :green_heart: | compile | 22m 24s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 22m 24s | | the patch passed |
| +1 :green_heart: | compile | 20m 36s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 20m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 27s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 1m 45s |
[/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3861/2/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
|
hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new
+ 0 unchanged - 0 fixed = 1 total (was 0) |
| -1 :x: | spotbugs | 3m 50s |
[/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3861/2/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html)
| hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) |
| +1 :green_heart: | shadedclient | 23m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 17m 44s |
[/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3861/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 232m 43s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 13s | | The patch does not
generate ASF License warnings. |
| | | 464m 32s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Dead store to corruptBlock in
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.checkAndUpdate(String,
FsVolumeSpi$ScanInfo) At
FsDatasetImpl.java:org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.checkAndUpdate(String,
FsVolumeSpi$ScanInfo) At FsDatasetImpl.java:[line 2814] |
| Failed junit tests | hadoop.ipc.TestIPC |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3861/2/artifact/out/Dockerfile
|
| GITHUB PR |
[GitHub] [hadoop] hadoop-yetus commented on pull request #3859: YARN-10632. Make auto queue creation maximum allowed depth configurable
hadoop-yetus commented on pull request #3859:
URL: https://github.com/apache/hadoop/pull/3859#issuecomment-1006237897
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 19s | | trunk passed |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 1s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 4s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 15s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 0m 58s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 58s | | the patch passed |
| +1 :green_heart: | compile | 0m 49s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 38s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3859/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 34 unchanged - 0 fixed = 35 total (was 34) |
| +1 :green_heart: | mvnsite | 0m 53s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 40s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 37s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 96m 27s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 186m 35s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3859/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3859 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux c93546a21df2 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5aa48d8391f4fb5d44213d1bc3af9fdf289c9d5e |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3859/2/testReport/ |
| Max. process+thread count | 945 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3859/2/console
[GitHub] [hadoop] tomscut commented on a change in pull request #3827: HDFS-16396. Reconfig slow peer parameters for datanode
tomscut commented on a change in pull request #3827:
URL: https://github.com/apache/hadoop/pull/3827#discussion_r779258490
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
##
@@ -642,13 +656,76 @@ public String reconfigurePropertyImpl(String property,
String newVal)
}
break;
}
+case DFS_DATANODE_PEER_STATS_ENABLED_KEY:
+case DFS_DATANODE_MIN_OUTLIER_DETECTION_NODES_KEY:
+case DFS_DATANODE_SLOWPEER_LOW_THRESHOLD_MS_KEY:
+case DFS_DATANODE_PEER_METRICS_MIN_OUTLIER_DETECTION_SAMPLES_KEY:
+ return reconfSlowPeerParameters(property, newVal);
default:
break;
}
throw new ReconfigurationException(
property, newVal, getConf().get(property));
}
+ private String reconfSlowPeerParameters(String property, String newVal)
+ throws ReconfigurationException {
+String result;
+try {
+ LOG.info("Reconfiguring {} to {}", property, newVal);
+ if (property.equals(DFS_DATANODE_PEER_STATS_ENABLED_KEY)) {
+checkNotNull(dnConf, "DNConf has not been initialized.");
+if (newVal != null && !newVal.equalsIgnoreCase("true")
+&& !newVal.equalsIgnoreCase("false")) {
+ throw new IllegalArgumentException("Not a valid Boolean value for "
+ property +
+ " in reconfSlowPeerParameters");
+}
+boolean enable = (newVal == null ?
DFS_DATANODE_PEER_STATS_ENABLED_DEFAULT :
+Boolean.parseBoolean(newVal));
+result = Boolean.toString(enable);
+dnConf.setPeerStatsEnabled(enable);
+if (enable) {
+ if (peerMetrics == null) {
+peerMetrics = DataNodePeerMetrics.create(getDisplayName(),
getConf());
+ }
+} else {
+ peerMetrics = null;
Review comment:
> Thanks @ayushtkn for your comments.
>
> To avoid NPE, I set `peerStatsEnabled`(volatile) first, and then I add
judgment `dnConf.peerStatsEnabled && peerMetrics != null` before using
`peerMetrics`. Do you think this is ok?
Hi @ayushtkn , what do you think of this solution? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3863: HDFS-16413. Reconfig dfs usage parameters for datanode
hadoop-yetus commented on pull request #3863:
URL: https://github.com/apache/hadoop/pull/3863#issuecomment-1006223517
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 14s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 15s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 56s | | trunk passed |
| +1 :green_heart: | compile | 30m 58s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 26m 19s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 12s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 21s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 55s | | trunk passed |
| +1 :green_heart: | shadedclient | 28m 38s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 22s | | the patch passed |
| +1 :green_heart: | compile | 23m 50s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 23m 50s | | the patch passed |
| +1 :green_heart: | compile | 20m 37s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 54s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3863/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 3 new + 212 unchanged - 0 fixed = 215 total (was
212) |
| +1 :green_heart: | mvnsite | 3m 11s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 13s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 19s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 28m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 37s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 348m 13s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 10s | | The patch does not
generate ASF License warnings. |
| | | 611m 12s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3863/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3863 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 8aefeb409351 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5
11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b0b27f4c11baf9360b9071c871835a20b65439af |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3863/1/testReport/ |
| Max. process+thread count | 2004 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3863/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT
[GitHub] [hadoop] tomscut commented on pull request #3753: HDFS-16371. Exclude slow disks when choosing volume
tomscut commented on pull request #3753: URL: https://github.com/apache/hadoop/pull/3753#issuecomment-1006187746 Thanks @tasanuma for your review and merging this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tasanuma commented on pull request #3753: HDFS-16371. Exclude slow disks when choosing volume
tasanuma commented on pull request #3753: URL: https://github.com/apache/hadoop/pull/3753#issuecomment-1006186602 Merged it. Thanks for your contribution, @tomscut! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tasanuma merged pull request #3753: HDFS-16371. Exclude slow disks when choosing volume
tasanuma merged pull request #3753: URL: https://github.com/apache/hadoop/pull/3753 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3753: HDFS-16371. Exclude slow disks when choosing volume
hadoop-yetus commented on pull request #3753:
URL: https://github.com/apache/hadoop/pull/3753#issuecomment-1006166618
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 9s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 45s | | trunk passed |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 34s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 30s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 21s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 24s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 14s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 19s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 372m 24s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 472m 19s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3753/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3753 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 3f6445cd199f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / eb505749011297893ed44edbcd691387e0acf5da |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3753/6/testReport/ |
| Max. process+thread count | 2856 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3753/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact
[jira] [Commented] (HADOOP-18026) Fix default value of Magic committer
[ https://issues.apache.org/jira/browse/HADOOP-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469596#comment-17469596 ] Steve Loughran commented on HADOOP-18026: - done. > Fix default value of Magic committer > > > Key: HADOOP-18026 > URL: https://issues.apache.org/jira/browse/HADOOP-18026 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.3.1 >Reporter: guophilipse >Assignee: guophilipse >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.2 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > `fs.s3a.committer.magic.enabled` was set to true by default after > HADOOP-17483, we can improve the doc -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18026) Fix default value of Magic committer
[ https://issues.apache.org/jira/browse/HADOOP-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-18026: --- Assignee: guophilipse > Fix default value of Magic committer > > > Key: HADOOP-18026 > URL: https://issues.apache.org/jira/browse/HADOOP-18026 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.3.1 >Reporter: guophilipse >Assignee: guophilipse >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.2 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > `fs.s3a.committer.magic.enabled` was set to true by default after > HADOOP-17483, we can improve the doc -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3862: HDFS-11107.TestStartup#testStorageBlockContentsStaleAfterNNRestart fails intermittently.
hadoop-yetus commented on pull request #3862:
URL: https://github.com/apache/hadoop/pull/3862#issuecomment-1006055417
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 42s | | trunk passed |
| +1 :green_heart: | compile | 1m 30s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 11s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 11s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 51s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 228m 14s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 330m 6s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3862/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3862 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 13c72faf5be9 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 67c64a0598c4f485aa53a8dc732f99853a27eea2 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3862/1/testReport/ |
| Max. process+thread count | 3027 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3862/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HADOOP-18024) SocketChannel is not closed when IOException happens in Server$Listener.doAccept
[
https://issues.apache.org/jira/browse/HADOOP-18024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469527#comment-17469527
]
Haoze Wu commented on HADOOP-18024:
---
[~ayushtkn]
Sorry for the delay. I should have fixed the issue earlier.
Now I think the root cause is
`org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1922)` which
may throw IOException (Connection reset by peer) from time to time. In the test
I only consider EOFException, which happens in most cases.
IOException (Connection reset by peer) should also be an expected exception
caught. I will raise a pr in github today.
> SocketChannel is not closed when IOException happens in
> Server$Listener.doAccept
>
>
> Key: HADOOP-18024
> URL: https://issues.apache.org/jira/browse/HADOOP-18024
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 3.2.2
>Reporter: Haoze Wu
>Assignee: Haoze Wu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> This is a follow-up of HADOOP-17552.
> When the symptom described in HADOOP-17552 happens, the client may time out
> in 2min, according to the default RPC timeout configuration specified in
> HADOOP-17552. Before this timeout, the client just waits, and does not know
> this issue happens.
> However, we recently found that actually the client doesn’t need to waste
> this 2min, and the server’s availability can be also improved. If the
> IOException happens in line 1402 or 1403 or 1404, we can just close this
> problematic `SocketChannel` and continue to accept new socket connections.
> The client side can also be aware of the close socket immediately, instead of
> waiting 2min.
> The old implementation:
> {code:java}
> //hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
> public void run() {
> while (running) {
> // ...
> try {
> // ...
> while (iter.hasNext()) {
> // ...
> try {
> if (key.isValid()) {
> if (key.isAcceptable())
> doAccept(key); // line 1348
> }
> } catch (IOException e) { // line 1350
> }
> // ...
> }
> } catch (OutOfMemoryError e) {
> // ...
> } catch (Exception e) {
> // ...
> }
> }
> } {code}
> {code:java}
> //hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
> void doAccept(SelectionKey key) throws InterruptedException, IOException,
> OutOfMemoryError {
> ServerSocketChannel server = (ServerSocketChannel) key.channel();
> SocketChannel channel;
> while ((channel = server.accept()) != null) { // line 1400
> channel.configureBlocking(false); // line 1402
> channel.socket().setTcpNoDelay(tcpNoDelay); // line 1403
> channel.socket().setKeepAlive(true); // line 1404
> Reader reader = getReader();
> Connection c = connectionManager.register(channel,
> this.listenPort, this.isOnAuxiliaryPort);
> // If the connectionManager can't take it, close the connection.
> if (c == null) {
> if (channel.isOpen()) {
> IOUtils.cleanup(null, channel);
> }
> connectionManager.droppedConnections.getAndIncrement();
> continue;
> }
> key.attach(c); // so closeCurrentConnection can get the object
> reader.addConnection(c);
> }
> } {code}
>
> We propose that the following implementation is better:
> {code:java}
> void doAccept(SelectionKey key) throws InterruptedException, IOException,
> OutOfMemoryError {
> ServerSocketChannel server = (ServerSocketChannel) key.channel();
> SocketChannel channel;
> while ((channel = server.accept()) != null) { // line 1400
> try {
> channel.configureBlocking(false); // line 1402
> channel.socket().setTcpNoDelay(tcpNoDelay); // line 1403
> channel.socket().setKeepAlive(true); // line 1404
> } catch (IOException e) {
> LOG.warn(...);
> try {
> channel.socket().close();
> channel.close();
> } catch (IOException ignored) { }
> continue;
> }
> // ...
> }
> }{code}
> The advantages include:
> # {*}In the old implementation{*}, the `ServerSocketChannel` was abandoned
> due to the single
[jira] [Created] (HADOOP-18069) CVE-2021-0341 in okhttp@2.7.5 detected in hdfs-client
Eugene Shinn (Truveta) created HADOOP-18069: --- Summary: CVE-2021-0341 in okhttp@2.7.5 detected in hdfs-client Key: HADOOP-18069 URL: https://issues.apache.org/jira/browse/HADOOP-18069 Project: Hadoop Common Issue Type: Bug Components: hdfs-client Affects Versions: 3.3.1 Reporter: Eugene Shinn (Truveta) Our static vulnerability scanner (Fortify On Demand) detected [NVD - CVE-2021-0341 (nist.gov)|https://nvd.nist.gov/vuln/detail/CVE-2021-0341#VulnChangeHistorySection] in our application. We traced the vulnerability to a transitive dependency coming from hadoop-hdfs-client, which depends on okhttp@2.7.5 ([hadoop/pom.xml at trunk · apache/hadoop (github.com)|https://github.com/apache/hadoop/blob/trunk/hadoop-project/pom.xml#L137]). To resolve this issue, okhttp should be upgraded to 4.9.2+ (ref: [CVE-2021-0341 · Issue #6724 · square/okhttp (github.com)|https://github.com/square/okhttp/issues/6724]). -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18068) upgrade AWS SDK to 1.12.x
[ https://issues.apache.org/jira/browse/HADOOP-18068?focusedWorklogId=704168=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-704168 ] ASF GitHub Bot logged work on HADOOP-18068: --- Author: ASF GitHub Bot Created on: 05/Jan/22 19:17 Start Date: 05/Jan/22 19:17 Worklog Time Spent: 10m Work Description: steveloughran opened a new pull request #3864: URL: https://github.com/apache/hadoop/pull/3864 ### Description of PR move to latest AWS SDK ### How was this patch tested? itests without s3guard ``` -Dparallel-tests -DtestsThreadCount=6 -Dmarkers=keep -Dscale ``` full manual qualification as covered in testing doc ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [x] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 704168) Remaining Estimate: 0h Time Spent: 10m > upgrade AWS SDK to 1.12.x > - > > Key: HADOOP-18068 > URL: https://issues.apache.org/jira/browse/HADOOP-18068 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 3.3.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > upgrade to the latest SDK -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18068) upgrade AWS SDK to 1.12.x
[ https://issues.apache.org/jira/browse/HADOOP-18068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18068: Labels: pull-request-available (was: ) > upgrade AWS SDK to 1.12.x > - > > Key: HADOOP-18068 > URL: https://issues.apache.org/jira/browse/HADOOP-18068 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 3.3.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > upgrade to the latest SDK -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran opened a new pull request #3864: HADOOP-18068. upgrade AWS SDK to 1.12.132
steveloughran opened a new pull request #3864: URL: https://github.com/apache/hadoop/pull/3864 ### Description of PR move to latest AWS SDK ### How was this patch tested? itests without s3guard ``` -Dparallel-tests -DtestsThreadCount=6 -Dmarkers=keep -Dscale ``` full manual qualification as covered in testing doc ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [x] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18024) SocketChannel is not closed when IOException happens in Server$Listener.doAccept
[
https://issues.apache.org/jira/browse/HADOOP-18024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469486#comment-17469486
]
Ayush Saxena commented on HADOOP-18024:
---
The introduced test is failing in the daily build. Was able to repro that
locally as well,
[~functioner] Any plans to chase HADOOP-18046 which is raised for the test?
> SocketChannel is not closed when IOException happens in
> Server$Listener.doAccept
>
>
> Key: HADOOP-18024
> URL: https://issues.apache.org/jira/browse/HADOOP-18024
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 3.2.2
>Reporter: Haoze Wu
>Assignee: Haoze Wu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> This is a follow-up of HADOOP-17552.
> When the symptom described in HADOOP-17552 happens, the client may time out
> in 2min, according to the default RPC timeout configuration specified in
> HADOOP-17552. Before this timeout, the client just waits, and does not know
> this issue happens.
> However, we recently found that actually the client doesn’t need to waste
> this 2min, and the server’s availability can be also improved. If the
> IOException happens in line 1402 or 1403 or 1404, we can just close this
> problematic `SocketChannel` and continue to accept new socket connections.
> The client side can also be aware of the close socket immediately, instead of
> waiting 2min.
> The old implementation:
> {code:java}
> //hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
> public void run() {
> while (running) {
> // ...
> try {
> // ...
> while (iter.hasNext()) {
> // ...
> try {
> if (key.isValid()) {
> if (key.isAcceptable())
> doAccept(key); // line 1348
> }
> } catch (IOException e) { // line 1350
> }
> // ...
> }
> } catch (OutOfMemoryError e) {
> // ...
> } catch (Exception e) {
> // ...
> }
> }
> } {code}
> {code:java}
> //hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
> void doAccept(SelectionKey key) throws InterruptedException, IOException,
> OutOfMemoryError {
> ServerSocketChannel server = (ServerSocketChannel) key.channel();
> SocketChannel channel;
> while ((channel = server.accept()) != null) { // line 1400
> channel.configureBlocking(false); // line 1402
> channel.socket().setTcpNoDelay(tcpNoDelay); // line 1403
> channel.socket().setKeepAlive(true); // line 1404
> Reader reader = getReader();
> Connection c = connectionManager.register(channel,
> this.listenPort, this.isOnAuxiliaryPort);
> // If the connectionManager can't take it, close the connection.
> if (c == null) {
> if (channel.isOpen()) {
> IOUtils.cleanup(null, channel);
> }
> connectionManager.droppedConnections.getAndIncrement();
> continue;
> }
> key.attach(c); // so closeCurrentConnection can get the object
> reader.addConnection(c);
> }
> } {code}
>
> We propose that the following implementation is better:
> {code:java}
> void doAccept(SelectionKey key) throws InterruptedException, IOException,
> OutOfMemoryError {
> ServerSocketChannel server = (ServerSocketChannel) key.channel();
> SocketChannel channel;
> while ((channel = server.accept()) != null) { // line 1400
> try {
> channel.configureBlocking(false); // line 1402
> channel.socket().setTcpNoDelay(tcpNoDelay); // line 1403
> channel.socket().setKeepAlive(true); // line 1404
> } catch (IOException e) {
> LOG.warn(...);
> try {
> channel.socket().close();
> channel.close();
> } catch (IOException ignored) { }
> continue;
> }
> // ...
> }
> }{code}
> The advantages include:
> # {*}In the old implementation{*}, the `ServerSocketChannel` was abandoned
> due to the single exception in this single `SocketChannel`, because the
> exception handler is in line 1350. {*}In the new implementation{*}, we use a
> try-catch to handle the exception in line 1402 or 1403 or 1404, then the
> `ServerSocketChannel` can continue to accept new
[jira] [Updated] (HADOOP-18031) Build arm64 (aarch64) and x86_64 image with the same Dockerfile
[
https://issues.apache.org/jira/browse/HADOOP-18031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-18031:
--
Target Version/s: 3.4.0, 3.3.3 (was: 3.4.0, 3.3.2, 3.3.3)
> Build arm64 (aarch64) and x86_64 image with the same Dockerfile
> ---
>
> Key: HADOOP-18031
> URL: https://issues.apache.org/jira/browse/HADOOP-18031
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HADOOP-18031.branch-3.3.1.001.testing.patch
>
>
> -Support building Linux arm64 (aarch64) Docker images. And bump up some
> dependency versions.-
> -Note: This only provides a arm64 *runtime* environment for Hadoop 3. But not
> a full environment for compiling Hadoop 3 in arm64 yet. For the latter, gRPC
> may well need to be compiled from source because it hasn't started
> distributing Linux arm64 binaries yet.-
> -The patch for branch-3.3 is ready. I developed this patch on branch-3.3.1
> when I was trying to build arm64 Linux Hadoop Docker image. For trunk
> (3.4.0), due to HADOOP-17727, I need to post a different PR.-
> Just realized we already had {{Dockerfile_aarch64}}. Will try it out.
> My approach builds the Docker images for both architectures (x86_64 and
> aarch64) with the same {{Dockerfile}}.
> We should push the built arm64 image to Docker Hub. I only see amd64
> [there|https://hub.docker.com/r/apache/hadoop/tags] so I assumed we didn't
> have arm64 Docker image, hmm.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18065) ExecutorHelper.logThrowableFromAfterExecute() is too noisy.
[
https://issues.apache.org/jira/browse/HADOOP-18065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HADOOP-18065:
--
Target Version/s: 3.3.3 (was: 3.3.2)
> ExecutorHelper.logThrowableFromAfterExecute() is too noisy.
>
>
> Key: HADOOP-18065
> URL: https://issues.apache.org/jira/browse/HADOOP-18065
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.1
>Reporter: Mukund Thakur
>Assignee: Mukund Thakur
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {code:java}
> if (t == null && r instanceof Future && ((Future) r).isDone()) {
> try {
> ((Future) r).get();
> } catch (ExecutionException ee) {
> LOG.warn(
> "Execution exception when running task in " + Thread.currentThread()
> .getName());
> t = ee.getCause();
> } catch (InterruptedException ie) {
> LOG.warn("Thread (" + Thread.currentThread() + ") interrupted: ", ie);
> Thread.currentThread().interrupt();
> } catch (Throwable throwable) {
> t = throwable;
> }
> }
> if (t != null) {
> LOG.warn("Caught exception in thread " + Thread
> .currentThread().getName() + ": ", t);
> } {code}
> We should downgrade the logging here from warn to debug.
>
> CC [~ste...@apache.org] [~mehakmeetSingh]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3842: HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background
hadoop-yetus commented on pull request #3842:
URL: https://github.com/apache/hadoop/pull/3842#issuecomment-1005977842
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 24m 37s | | trunk passed |
| +1 :green_heart: | compile | 3m 32s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 3m 28s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 16s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 18s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 18s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 19s | |
branch/hadoop-hdfs-project/hadoop-hdfs-native-client no spotbugs output file
(spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 22m 34s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 14s | | the patch passed |
| +1 :green_heart: | compile | 3m 24s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | cc | 3m 24s | | the patch passed |
| +1 :green_heart: | golang | 3m 24s | | the patch passed |
| +1 :green_heart: | javac | 3m 24s | | the patch passed |
| +1 :green_heart: | compile | 3m 27s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | cc | 3m 27s | | the patch passed |
| +1 :green_heart: | golang | 3m 27s | | the patch passed |
| +1 :green_heart: | javac | 3m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 10s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 15s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 0m 13s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 13s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 14s | |
hadoop-hdfs-project/hadoop-hdfs-native-client has no data from spotbugs |
| +1 :green_heart: | shadedclient | 22m 11s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 76m 21s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 166m 54s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell shellcheck shelldocs cc golang spotbugs
checkstyle |
| uname | Linux 8f72fd1d301e 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5
11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ae8786d87220bc2ef8eb1725fdab63d19f20c6c0 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3842/3/testReport/ |
| Max. process+thread count | 519 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
[jira] [Commented] (HADOOP-18056) DistCp: Filter duplicates in the source paths
[ https://issues.apache.org/jira/browse/HADOOP-18056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469472#comment-17469472 ] Ayush Saxena commented on HADOOP-18056: --- Committed to trunk and branch-3.3 Thanx Everyone for the reviews!!! > DistCp: Filter duplicates in the source paths > - > > Key: HADOOP-18056 > URL: https://issues.apache.org/jira/browse/HADOOP-18056 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > Add a basic filtering to remove the exact duplicate paths exposed for copying. > In case two same srcPath say /tmp/file1 is passed in the list twice. DistCp > fails with DuplicateFileException, post building the listing. > Would be better if we do a basic filtering of duplicate paths. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18056) DistCp: Filter duplicates in the source paths
[ https://issues.apache.org/jira/browse/HADOOP-18056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18056. --- Fix Version/s: 3.4.0 3.3.3 Hadoop Flags: Reviewed Resolution: Fixed > DistCp: Filter duplicates in the source paths > - > > Key: HADOOP-18056 > URL: https://issues.apache.org/jira/browse/HADOOP-18056 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.3 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Add a basic filtering to remove the exact duplicate paths exposed for copying. > In case two same srcPath say /tmp/file1 is passed in the list twice. DistCp > fails with DuplicateFileException, post building the listing. > Would be better if we do a basic filtering of duplicate paths. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18056) DistCp: Filter duplicates in the source paths
[ https://issues.apache.org/jira/browse/HADOOP-18056?focusedWorklogId=704150=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-704150 ] ASF GitHub Bot logged work on HADOOP-18056: --- Author: ASF GitHub Bot Created on: 05/Jan/22 18:23 Start Date: 05/Jan/22 18:23 Worklog Time Spent: 10m Work Description: ayushtkn merged pull request #3825: URL: https://github.com/apache/hadoop/pull/3825 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 704150) Time Spent: 2h 10m (was: 2h) > DistCp: Filter duplicates in the source paths > - > > Key: HADOOP-18056 > URL: https://issues.apache.org/jira/browse/HADOOP-18056 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > Add a basic filtering to remove the exact duplicate paths exposed for copying. > In case two same srcPath say /tmp/file1 is passed in the list twice. DistCp > fails with DuplicateFileException, post building the listing. > Would be better if we do a basic filtering of duplicate paths. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn merged pull request #3825: HADOOP-18056. DistCp: Filter duplicates in the source paths.
ayushtkn merged pull request #3825: URL: https://github.com/apache/hadoop/pull/3825 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #3854: HDFS-16410. Fixing Insecure Xml parsing in OfflineEditsXmlLoader
sunchao commented on pull request #3854: URL: https://github.com/apache/hadoop/pull/3854#issuecomment-1005952115 @steveloughran sure, I've cherry-picked it to branch-3.3 and branch-3.3.2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3861: HDFS-16316.Improve DirectoryScanner: add regular file check related block.
hadoop-yetus commented on pull request #3861:
URL: https://github.com/apache/hadoop/pull/3861#issuecomment-1005940570
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 11m 39s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 0s | | trunk passed |
| +1 :green_heart: | compile | 28m 25s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | compile | 8m 33s |
[/branch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3861/1/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root in trunk failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -0 :warning: | checkstyle | 0m 39s |
[/buildtool-branch-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3861/1/artifact/out/buildtool-branch-checkstyle-root.txt)
| The patch fails to run checkstyle in root |
| -1 :x: | mvnsite | 0m 40s |
[/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3861/1/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| -1 :x: | mvnsite | 0m 40s |
[/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3861/1/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in trunk failed. |
| -1 :x: | javadoc | 0m 41s |
[/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3861/1/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-common in trunk failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| +1 :green_heart: | javadoc | 3m 38s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 54s | | trunk passed |
| -1 :x: | shadedclient | 8m 47s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 40s | | the patch passed |
| +1 :green_heart: | compile | 23m 57s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 57s | | the patch passed |
| +1 :green_heart: | compile | 19m 27s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 19m 27s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3861/1/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 811 new + 995
unchanged - 0 fixed = 1806 total (was 995) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 44s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3861/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 148 new + 0 unchanged - 0 fixed = 148 total (was
0) |
| +1 :green_heart: | mvnsite | 3m 18s | | the patch passed |
| -1 :x: | javadoc | 1m 11s |
[/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3861/1/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
|
hadoop-common-project_hadoop-common-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 99 new + 0 unchanged
- 0 fixed = 99 total (was 0) |
| -1 :x: | javadoc | 1m 44s |
[jira] [Updated] (HADOOP-17077) S3A delegation token binding to support secondary binding list
[ https://issues.apache.org/jira/browse/HADOOP-17077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17077: Parent: HADOOP-18067 (was: HADOOP-17566) > S3A delegation token binding to support secondary binding list > -- > > Key: HADOOP-17077 > URL: https://issues.apache.org/jira/browse/HADOOP-17077 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > (followon from HADOOP-17050) > Add the ability of an S3A FS instance to support multiple instances of > delegation token bindings. > The property "fs.s3a.delegation.token.secondary.bindings" will list the > classnames of all secondary bindings. > for each one, an instance shall be created with the canonical service name > being: fs URI + [ tokenKind ]. This is to ensure that the URIs are unique for > each FS instance -but also that a single fs instance can have multiple tokens > in the credential list. > the instance is just a AbstractDelegationTokenBinding provider of an AWS > credential provider chain, with the normal lifecycle and operations to bind > to a DT, issue tokens, etc > * the final list of AWS Credential providers will be built by appending those > provided by each binding in turn. > Token binding at launch > If the primary token binding binds to a delegation token, then the whole > binding is changed such that all secondary tokens MUST also bind. That is: it > will be an error if one cannot be found. This is possibly overstrict-but it > avoids situations where an incomplete set of tokens are retrieved and This > does not surface until later. > Only the encryption secrets in the primary DT will be used for FS encryption > settings. > Testing: yes. > Probably also by adding a test-only DT provider which doesn't actually issue > any real credentials and so which can be deployed in both ITests and staging > tests where we can verify that the chained instantiation works. > Compatibility: the goal is to be backwards compatible with any already > released token provider plugin. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13294) Test hadoop fs shell against s3a; fix problems
[
https://issues.apache.org/jira/browse/HADOOP-13294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13294:
Parent: HADOOP-18067 (was: HADOOP-17566)
> Test hadoop fs shell against s3a; fix problems
> --
>
> Key: HADOOP-13294
> URL: https://issues.apache.org/jira/browse/HADOOP-13294
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Major
>
> There's no tests of {{hadoop -fs}} commands against s3a; add some. Ideally,
> generic to all object stores.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15479) s3guard bucket-info command to add a verify-property =
[
https://issues.apache.org/jira/browse/HADOOP-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15479:
Parent: HADOOP-18067 (was: HADOOP-17566)
> s3guard bucket-info command to add a verify-property =
> ---
>
> Key: HADOOP-15479
> URL: https://issues.apache.org/jira/browse/HADOOP-15479
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Minor
>
> This is driven by me trying to test o whether a bucket has fault injection
> enabled. You can see from the logs, but not from a shell script.
> I want to be able to go
> {code}
> hadoop s3guard verify-property
> fs.s3a.s3.client.factory.impl=org.apache.hadoop.fs.s3a.InconsistentS3ClientFactory
> s3a://test-bucket/
> {code}
> and have the command return -1 if the property doesn't equal this value. This
> lets me check that per-bucket and test run settings are propagating down. As
> it is you need to look @ the logs.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16800) ITestDynamoDBMetadataStore.testTableVersioning failure -DDB deleteItem consistency?
[ https://issues.apache.org/jira/browse/HADOOP-16800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16800: Parent: HADOOP-18067 (was: HADOOP-17566) > ITestDynamoDBMetadataStore.testTableVersioning failure -DDB deleteItem > consistency? > --- > > Key: HADOOP-16800 > URL: https://issues.apache.org/jira/browse/HADOOP-16800 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Minor > > transient failure of ITestDynamoDBMetadataStore.testTableVersioning; the > deleted table version record was still there. > Looks like the delete is EC; finally I hit a consistency event > Proposed: the test iterates a bit awaiting the exception -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14707) AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade
[
https://issues.apache.org/jira/browse/HADOOP-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14707:
Parent: HADOOP-18067 (was: HADOOP-17566)
> AbstractContractDistCpTest to test attr preservation with -p, verify
> blobstores downgrade
> -
>
> Key: HADOOP-14707
> URL: https://issues.apache.org/jira/browse/HADOOP-14707
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/azure, fs/s3, test, tools/distcp
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14707-001.patch, HADOOP-14707-002.patch,
> HADOOP-14707-003.patch
>
>
> It *may* be that trying to use {{distcp -p}} with S3a triggers a stack trace
> {code}
> java.lang.UnsupportedOperationException: S3AFileSystem doesn't support
> getXAttrs
> at org.apache.hadoop.fs.FileSystem.getXAttrs(FileSystem.java:2559)
> at
> org.apache.hadoop.tools.util.DistCpUtils.toCopyListingFileStatus(DistCpUtils.java:322)
>
> {code}
> Add a test to {{AbstractContractDistCpTest}} to verify that this is handled
> better. What is "handle better" here? Either ignore the option or fail with
> "don't do that" text
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18068) upgrade AWS SDK to 1.12.x
[ https://issues.apache.org/jira/browse/HADOOP-18068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469441#comment-17469441 ] Steve Loughran commented on HADOOP-18068: - https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk/1.12.132 > upgrade AWS SDK to 1.12.x > - > > Key: HADOOP-18068 > URL: https://issues.apache.org/jira/browse/HADOOP-18068 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 3.3.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > upgrade to the latest SDK -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18068) upgrade AWS SDK to 1.12.x
Steve Loughran created HADOOP-18068: --- Summary: upgrade AWS SDK to 1.12.x Key: HADOOP-18068 URL: https://issues.apache.org/jira/browse/HADOOP-18068 Project: Hadoop Common Issue Type: Sub-task Components: build, fs/s3 Affects Versions: 3.3.2 Reporter: Steve Loughran Assignee: Steve Loughran upgrade to the latest SDK -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15057) s3guard bucket-info command to include default bucket encryption info
[ https://issues.apache.org/jira/browse/HADOOP-15057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15057: Parent: HADOOP-18067 (was: HADOOP-17566) > s3guard bucket-info command to include default bucket encryption info > - > > Key: HADOOP-15057 > URL: https://issues.apache.org/jira/browse/HADOOP-15057 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran >Priority: Minor > > AWS S3 now has the notion of default bucket encryption > [http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETencryption.html] > Once set, all data uploaded is automatically encrypted, without needing to > set any client options > We should provide that info in the s3guard bucket-info command, so you can > see that data being uploaded really is encrypted. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15870: Parent: HADOOP-18067 (was: HADOOP-17566) > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch, > HADOOP-15870-004.patch, HADOOP-15870-005.patch, HADOOP-15870-006.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16017) Add some S3A-specific create file options
[ https://issues.apache.org/jira/browse/HADOOP-16017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16017: Parent: HADOOP-18067 (was: HADOOP-17566) > Add some S3A-specific create file options > - > > Key: HADOOP-16017 > URL: https://issues.apache.org/jira/browse/HADOOP-16017 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran >Priority: Minor > > Add some options in createFile() for S3A specifically. > I think we need something for "put-only" where none of the existence checks > nor cleanup checks are made when s3guard == off. That is: no HEAD/LIST first, > no DELETE parents after. Only the PUT is done. > This is > * faster > * lower cost > * avoids 404's being cached in load balancers and so create inconsistency > It does rely on the caller knowing what they are doing else you end up in a > mess, but since the s3a committers use WriteOperationsHelper for this exact > operation, we should open it up to others who also know what they are doing. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15503) strip s3.amazonaws.com off hostnames before making s3a calls
[
https://issues.apache.org/jira/browse/HADOOP-15503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15503:
Parent: HADOOP-18067 (was: HADOOP-17566)
> strip s3.amazonaws.com off hostnames before making s3a calls
>
>
> Key: HADOOP-15503
> URL: https://issues.apache.org/jira/browse/HADOOP-15503
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Minor
>
> If you copy an http URL https://bucketname.s3.amazonaws.com/ and convert to
> an s3a one by replacing the schema, you can try to list
> {code}
> bin/hadoop fs -ls -r s3a://hwdev-steve-new.s3.amazonaws.com/
> {code}
> But do that and you get told there's no such bucket
> {code}
> ls: Bucket hwdev-steve-new.s3.amazonaws.com does not exist
> {code}
> This is non-intuitive, and catches me out.
> We could strip this automatically during initialization to produce the actual
> bucket, which would need to be done before any per-bucket init is done.
> I do worry about what could break though
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14132) Filesystem discovery to stop loading implementation classes
[ https://issues.apache.org/jira/browse/HADOOP-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14132: Parent: HADOOP-18067 (was: HADOOP-17566) > Filesystem discovery to stop loading implementation classes > --- > > Key: HADOOP-14132 > URL: https://issues.apache.org/jira/browse/HADOOP-14132 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/adl, fs/azure, fs/oss, fs/s3, fs/swift >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Priority: Major > > Integration testing of Hadoop with the HADOOP-14040 has shown up that the > move to a shaded AWS JAR is slowing all hadoop client code down. > I believe this is due to how we use service discovery to identify FS > implementations: the implementation classes themselves are instantiated. > This has known problems today with classloading, but clearly impacts > performance too, especially with complex transitive dependencies unique to > the loaded class. > Proposed: have lightweight service declaration classes which implement an > interface declaring > # schema > # classname of FileSystem impl > # classname of AbstractFS impl > # homepage (for third party code, support, etc) > These are what we register and scan in the FS to look for services. > This will leave the question about what to do for existing filesystems? I > think we'll need to retain the old code for external ones, while moving the > hadoop modules to the new ones -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17566) Über-jira: S3A Hadoop 3.3.2 features
[ https://issues.apache.org/jira/browse/HADOOP-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17566. - Fix Version/s: 3.3.2 Resolution: Done > Über-jira: S3A Hadoop 3.3.2 features > > > Key: HADOOP-17566 > URL: https://issues.apache.org/jira/browse/HADOOP-17566 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.3.2 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14639) test YARN log collection works to s3a
[ https://issues.apache.org/jira/browse/HADOOP-14639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14639: Parent: HADOOP-18067 (was: HADOOP-17566) > test YARN log collection works to s3a > - > > Key: HADOOP-14639 > URL: https://issues.apache.org/jira/browse/HADOOP-14639 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Minor > > Extend the s3a+ YARN tests to verify that log collection can use s3a:// URLs > as a destination -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17566) Über-jira: S3A Hadoop 3.3.2 features
[ https://issues.apache.org/jira/browse/HADOOP-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17566: Issue Type: Improvement (was: Bug) > Über-jira: S3A Hadoop 3.3.2 features > > > Key: HADOOP-17566 > URL: https://issues.apache.org/jira/browse/HADOOP-17566 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.3.2 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13293) add a special 0 byte input stream for empty blobs
[ https://issues.apache.org/jira/browse/HADOOP-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13293: Parent: HADOOP-18067 (was: HADOOP-17566) > add a special 0 byte input stream for empty blobs > - > > Key: HADOOP-13293 > URL: https://issues.apache.org/jira/browse/HADOOP-13293 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > S3a (and the other object stores) have a lot of IO going on, even for 0 byte > files. They don't need to: that's a special case which can be handled > locally. A special ZeroByteInputStream class could handle this for all the > object stores. > This isn't much of an optimization: code shouldn't normally need to go > through 0 byte files, but we see evidence it does sometimes happen. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13585) shell rm command to not rename to ~/.Trash in object stores
[
https://issues.apache.org/jira/browse/HADOOP-13585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13585:
Parent: HADOOP-18067 (was: HADOOP-17566)
> shell rm command to not rename to ~/.Trash in object stores
> ---
>
> Key: HADOOP-13585
> URL: https://issues.apache.org/jira/browse/HADOOP-13585
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: util
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Major
>
> When you do a {{hadoop fs -rm -s3a://bucket/large-file}} there's a long delay
> and then you are told that it's been moved to
> {{s3a://Users/stevel/.Trash/current/large-file}}. Where it still incurs
> costs. You need to then delete that file using {{-skipTrash}} because the
> {{fs -expunge}} command only works on the local fs: you can't point it at an
> object store unless that is the default FS.
> I'd like an option to tell the shell to tell it that it should bypass the
> renaming on an FS-by-FS basis. And the for {{fs expunge}} to take a
> filesystem as an optional argument.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12020) Support AWS S3 reduced redundancy storage class
[
https://issues.apache.org/jira/browse/HADOOP-12020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-12020:
Parent: HADOOP-18067 (was: HADOOP-17566)
> Support AWS S3 reduced redundancy storage class
> ---
>
> Key: HADOOP-12020
> URL: https://issues.apache.org/jira/browse/HADOOP-12020
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.0
> Environment: Hadoop on AWS
>Reporter: Yann Landrin-Schweitzer
>Priority: Major
>
> Amazon S3 uses, by default, the NORMAL_STORAGE class for s3 objects.
> This offers, according to Amazon's material, 99.% reliability.
> For many applications, however, the 99.99% reliability offered by the
> REDUCED_REDUNDANCY storage class is amply sufficient, and comes with a
> significant cost saving.
> HDFS, when using the legacy s3n protocol, or the new s3a scheme, should
> support overriding the default storage class of created s3 objects so that
> users can take advantage of this cost benefit.
> This would require minor changes of the s3n and s3a drivers, using
> a configuration property fs.s3n.storage.class to override the default storage
> when desirable.
> This override could be implemented in Jets3tNativeFileSystemStore with:
> S3Object object = new S3Object(key);
> ...
> if(storageClass!=null) object.setStorageClass(storageClass);
> It would take a more complex form in s3a, e.g. setting:
> InitiateMultipartUploadRequest initiateMPURequest =
> new InitiateMultipartUploadRequest(bucket, key, om);
> if(storageClass !=null ) {
> initiateMPURequest =
> initiateMPURequest.withStorageClass(storageClass);
> }
> and similar statements in various places.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16308) ITestS3AContractSeek teardown closes test FS before superclass can do its cleanup
[ https://issues.apache.org/jira/browse/HADOOP-16308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16308: Parent: HADOOP-18067 (was: HADOOP-17566) > ITestS3AContractSeek teardown closes test FS before superclass can do its > cleanup > - > > Key: HADOOP-16308 > URL: https://issues.apache.org/jira/browse/HADOOP-16308 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.0, 3.1.2, 3.2.1 >Reporter: Steve Loughran >Priority: Minor > > the cleanup for the ITestS3AContractSeek now adds a stack trace to the logs > warning that the getFileSystem() Fs has been closed. Looks like it came from > the parquet seek fix patch -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14776) clean up ITestS3AFileSystemContract
[
https://issues.apache.org/jira/browse/HADOOP-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14776:
Parent: HADOOP-18067 (was: HADOOP-17566)
> clean up ITestS3AFileSystemContract
> ---
>
> Key: HADOOP-14776
> URL: https://issues.apache.org/jira/browse/HADOOP-14776
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-14776.01.patch
>
>
> With the move of {{FileSystemContractTest}} test to JUnit4, the bits of
> {{ITestS3AFileSystemContract}} which override existing methods just to skip
> them can be cleaned up: The subclasses could throw assume() so their skippage
> gets noted.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16633) Tune hadoop-aws parallel test surefire/failsafe settings
[ https://issues.apache.org/jira/browse/HADOOP-16633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16633: Parent: HADOOP-18067 (was: HADOOP-17566) > Tune hadoop-aws parallel test surefire/failsafe settings > > > Key: HADOOP-16633 > URL: https://issues.apache.org/jira/browse/HADOOP-16633 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3, test >Affects Versions: 3.2.1 >Reporter: Steve Loughran >Priority: Major > > We can do more to improve our test runs by looking at the failsafe docs > [http://maven.apache.org/surefire/maven-failsafe-plugin/integration-test-mojo.html] > > * default value to be by core, eg. 2C > * use the runorder attribute which determines how parallel runs are chosen; > random for better nondeterminism > We'd need to experiment first, which can be done by setting failsafe.runOrder -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13846) S3A to implement rename(final Path src, final Path dst, final Rename... options)
[
https://issues.apache.org/jira/browse/HADOOP-13846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13846:
Parent: HADOOP-18067 (was: HADOOP-17566)
> S3A to implement rename(final Path src, final Path dst, final Rename...
> options)
>
>
> Key: HADOOP-13846
> URL: https://issues.apache.org/jira/browse/HADOOP-13846
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Major
>
> S3a now raises exceptions on invalid rename operations, but these get lost. I
> plan to use them in my s3guard committer HADOOP-13786.
> Rather than just make innerRename() private, S3A could implement
> {{FileSystem.rename(final Path src, final Path dst, final Rename...
> options)}} and so have an exception-raising rename which can be called
> without going more into the internals.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15915) Report problems w/ local S3A buffer directory meaningfully
[ https://issues.apache.org/jira/browse/HADOOP-15915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15915: Parent: HADOOP-18067 (was: HADOOP-17566) > Report problems w/ local S3A buffer directory meaningfully > -- > > Key: HADOOP-15915 > URL: https://issues.apache.org/jira/browse/HADOOP-15915 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.1 >Reporter: Steve Loughran >Priority: Major > > When there's a problem working with the temp directory used for block output > and the staging committers the actual path (and indeed config option) aren't > printed. > Improvements: tell the user which directory isn't writeable -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15347) S3ARetryPolicy to handle AWS 500 responses/error code TooBusyException with the throttle backoff policy
[ https://issues.apache.org/jira/browse/HADOOP-15347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15347: Parent: HADOOP-18067 (was: HADOOP-17566) > S3ARetryPolicy to handle AWS 500 responses/error code TooBusyException with > the throttle backoff policy > --- > > Key: HADOOP-15347 > URL: https://issues.apache.org/jira/browse/HADOOP-15347 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Priority: Minor > > FLINK-9061 implies that some 500 responses are caused by server-side overload > of some form. > That means they should really have the throttle retry policy applied, not the > connectivity one -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15575) Add a way for an FS instance to say "really, no trash interval at all"
[
https://issues.apache.org/jira/browse/HADOOP-15575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15575:
Parent: HADOOP-18067 (was: HADOOP-17566)
> Add a way for an FS instance to say "really, no trash interval at all"
> --
>
> Key: HADOOP-15575
> URL: https://issues.apache.org/jira/browse/HADOOP-15575
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Major
>
> Object stores like S3 often offer [object
> versioning|https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectVersioning.html]
> as a way to recover from accidental deletions. There is no need to use the
> Fs Shell trash mechanism.
> Proposed: a way for an FS instance to tell the shell to not use trash, even
> if the client has configured it. The current
> {{getServerDefaults().getTrashInterval()}} returns a value from the server,
> but it *only* overrides the ""fs.trash.interval" if it !=0. So if a server
> says "don't use trash" it gets ignored.
> If a special value (-1) is returned (or we use a new field?), FS instances
> can return to the shell saying "no need for trash". Then you can turn it on
> for a bucket by bucket basis; any store with the feature enabled can do the
> same thing.
> Alternative option: A special S3A-aware trash policy which does a
> getFileSystem.getConf.getBoolean("fs.s3a.trash.skip") & skips trash if set.
> This will all you to disable it on a bucket-by-bucket basis.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16538) S3AFilesystem trash handling should respect the current UGI
[ https://issues.apache.org/jira/browse/HADOOP-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16538: Parent: HADOOP-18067 (was: HADOOP-17566) > S3AFilesystem trash handling should respect the current UGI > --- > > Key: HADOOP-16538 > URL: https://issues.apache.org/jira/browse/HADOOP-16538 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Siddharth Seth >Priority: Major > > S3 move to trash currently relies upon System.getProperty(user.name). > Instead, it should be relying on the current UGI to figure out the username. > getHomeDirectory needs to be overridden to use UGI instead of > System.getProperty -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15303) make s3a read fault injection configurable including "off"
[ https://issues.apache.org/jira/browse/HADOOP-15303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15303: Parent: HADOOP-18067 (was: HADOOP-17566) > make s3a read fault injection configurable including "off" > -- > > Key: HADOOP-15303 > URL: https://issues.apache.org/jira/browse/HADOOP-15303 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Priority: Minor > > When trying to test distcp with large files and inconsistent destination (P > fail = 0.4), read() failures on the D/L can overload the retry logic in > S3AInput, even though all I want to see is how listings cope. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16574) ITestS3AAWSCredentialsProvider tests fail if a bucket has DTs enabled
[ https://issues.apache.org/jira/browse/HADOOP-16574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16574: Parent: HADOOP-18067 (was: HADOOP-17566) > ITestS3AAWSCredentialsProvider tests fail if a bucket has DTs enabled > - > > Key: HADOOP-16574 > URL: https://issues.apache.org/jira/browse/HADOOP-16574 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Minor > > If you enable DTs on a bucket, then those tests which force failures from bad > credential providers fail -the IOE they look for is wrapped in a > ServiceStateException > Proposed: catch those and rethrow the nested IOE -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15555) S3AFileStatus to add a serialVersionUID; review & test serialization
[ https://issues.apache.org/jira/browse/HADOOP-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-1: Parent: HADOOP-18067 (was: HADOOP-17566) > S3AFileStatus to add a serialVersionUID; review & test serialization > > > Key: HADOOP-1 > URL: https://issues.apache.org/jira/browse/HADOOP-1 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Priority: Major > > As FileStatus is now serializable, review S3AFilestatus > * add a version field > * review deserialization to see if we need any checks to stop > invalid/malicious status instances being created -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15807) ITestS3AContractRootDir failure on non-S3Guarded bucket
[ https://issues.apache.org/jira/browse/HADOOP-15807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15807: Parent: HADOOP-18067 (was: HADOOP-17566) > ITestS3AContractRootDir failure on non-S3Guarded bucket > --- > > Key: HADOOP-15807 > URL: https://issues.apache.org/jira/browse/HADOOP-15807 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Minor > > Got a root test failure against S3 London, possibly consistency related. > The abstract test case should use eventually() here -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15000) s3a new getdefaultblocksize be called in getFileStatus which has not been implemented in s3afilesystem yet
[
https://issues.apache.org/jira/browse/HADOOP-15000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15000:
Parent: HADOOP-18067 (was: HADOOP-17566)
> s3a new getdefaultblocksize be called in getFileStatus which has not been
> implemented in s3afilesystem yet
> --
>
> Key: HADOOP-15000
> URL: https://issues.apache.org/jira/browse/HADOOP-15000
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.9.0
>Reporter: Yonger
>Priority: Minor
>
> new implementation of getting block size has been called in getFileStatus
> method:
> {code:java}
> return new S3AFileStatus(meta.getContentLength(),
> dateToLong(meta.getLastModified()),
> path,
> getDefaultBlockSize(path),
> username);
> }
> {code}
> while we don't implement it in our s3afilesystem currently, also we need to
> implement this new method as the old one deprecated.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14603) S3A input stream to support ByteBufferReadable
[
https://issues.apache.org/jira/browse/HADOOP-14603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14603:
Parent: HADOOP-18067 (was: HADOOP-17566)
> S3A input stream to support ByteBufferReadable
> --
>
> Key: HADOOP-14603
> URL: https://issues.apache.org/jira/browse/HADOOP-14603
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Priority: Minor
>
> S3AInputStream could support {{ByteBufferReadable,
> HasEnhancedByteBufferAccess}} and the operations to read into byte buffers.
> This is only if we can see a clear performance benefit from doing this or the
> API is being more broadly used
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16432) Review S3A documentation to make sure it is consistent with the current codebase
[ https://issues.apache.org/jira/browse/HADOOP-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16432: Parent: HADOOP-18067 (was: HADOOP-17566) > Review S3A documentation to make sure it is consistent with the current > codebase > > > Key: HADOOP-16432 > URL: https://issues.apache.org/jira/browse/HADOOP-16432 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Major > > We need to review the hadoop-aws docs and make sure that it is consistent > with the current codebase > * The performance.md doc is out of date w.r.t DDB on-demand and the threads, > distcp (no -direct); maybe more. > * We should review the assumed role doc and combine the S3 and S3Guard > permissions into one list > * the list of AWS regions is out of date (china 1 & 2, hong-kong, osaka) > * benchmark rename performance of large & huge files > * see if there is any current work on time-to-consistency for lists and > updates > We also need to pull up the classpath problems "don't mix JARs unless you > like stack traces" as the #1 entry on the list, as its the recurrent problem > downstream, someone adding a new hadoop-aws JAR to a spark installation and > then complaining about CNFEs -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15069) support git-secrets commit hook to keep AWS secrets out of git
[ https://issues.apache.org/jira/browse/HADOOP-15069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15069: Parent: HADOOP-18067 (was: HADOOP-17566) > support git-secrets commit hook to keep AWS secrets out of git > -- > > Key: HADOOP-15069 > URL: https://issues.apache.org/jira/browse/HADOOP-15069 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-15069-001.patch, HADOOP-15069-002.patch > > > The latest Uber breach looks like it involved AWS keys in git repos. > Nobody wants that, which is why amazon provide > [git-secrets|https://github.com/awslabs/git-secrets]; a script you can use to > scan a repo and its history, *and* add as an automated check. > Anyone can set this up, but there are a few false positives in the scan, > mostly from longs and a few all-upper-case constants. These can all be added > to a .gitignore file. > Also: mention git-secrets in the aws testing docs; say "use it" -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15249) review S3A translateException translation matches IBM CORS spec
[ https://issues.apache.org/jira/browse/HADOOP-15249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15249: Parent: HADOOP-18067 (was: HADOOP-17566) > review S3A translateException translation matches IBM CORS spec > --- > > Key: HADOOP-15249 > URL: https://issues.apache.org/jira/browse/HADOOP-15249 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 > Environment: IBM's doc of their store [lists all exception and http > error codes|https://ibm-public-cos.github.io/crs-docs/api-reference] > We should review translateException to see that it is consistent with those > responses; as it seems a bit clearer about the values than the AWS SDK itself. >Reporter: Steve Loughran >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16155) S3AInputStream read(bytes[]) to not retry on read failure: pass action up
[ https://issues.apache.org/jira/browse/HADOOP-16155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16155: Parent: HADOOP-18067 (was: HADOOP-17566) > S3AInputStream read(bytes[]) to not retry on read failure: pass action up > - > > Key: HADOOP-16155 > URL: https://issues.apache.org/jira/browse/HADOOP-16155 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Priority: Minor > > The S3AInputStream reacts to read(byte[]) failure by reopening the stream, > just as for the single byte read(). We shouldn't need to do that. Instead > just close the stream, return 0 and let the caller decided what to do. > why so? > # its in the contract of InputStream.read(bytes[]), > # readFully() can handle the 0 in its loop > # other apps can decided what to do. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16154) ITestS3A select tests fail if user kinited in
[
https://issues.apache.org/jira/browse/HADOOP-16154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16154:
Parent: HADOOP-18067 (was: HADOOP-17566)
> ITestS3A select tests fail if user kinited in
> -
>
> Key: HADOOP-16154
> URL: https://issues.apache.org/jira/browse/HADOOP-16154
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Major
>
> {code}
> [ERROR] Errors:
> [ERROR] ITestS3Select.testInputSplit:962 » IO Can't get Master Kerberos
> principal for ...
> [ERROR] ITestS3SelectMRJob.testLandsatSelect:156 » IO Can't get Master
> Kerberos princi...
> [ERROR] ITestS3AMiniYarnCluster.testWithMiniCluster:117 » IO Can't get
> Master Kerberos...
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15224) builld up md5 checksum as blocks are built in S3ABlockOutputStream; validate upload
[
https://issues.apache.org/jira/browse/HADOOP-15224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15224:
Parent: HADOOP-18067 (was: HADOOP-17566)
> builld up md5 checksum as blocks are built in S3ABlockOutputStream; validate
> upload
> ---
>
> Key: HADOOP-15224
> URL: https://issues.apache.org/jira/browse/HADOOP-15224
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Priority: Minor
>
> [~rdblue] reports sometimes he sees corrupt data on S3. Given MD5 checks from
> upload to S3, its likelier to have happened in VM RAM, HDD or nearby.
> If the MD5 checksum for each block was built up as data was written to it,
> and checked against the etag RAM/HDD storage of the saved blocks could be
> removed as sources of corruption
> The obvious place would be
> {{org.apache.hadoop.fs.s3a.S3ADataBlocks.DataBlock}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13695) S3A to use a thread pool for async path operations
[
https://issues.apache.org/jira/browse/HADOOP-13695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13695:
Parent: HADOOP-18067 (was: HADOOP-17566)
> S3A to use a thread pool for async path operations
> --
>
> Key: HADOOP-13695
> URL: https://issues.apache.org/jira/browse/HADOOP-13695
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Major
>
> S3A path operations are often slow due to directory scanning, mock directory
> create/delete, etc. Many of these can be done asynchronously
> * because deletion is eventually consistent, deleting parent dirs after an
> operation has returned doesn't alter the behaviour, except in the special
> case of : operation failure.
> * scanning for paths/parents of a file in the create operation only needs to
> complete before the close() operation instantiates the object, no need to
> block create().
> * parallelized COPY calls would permit asynchronous rename.
> We could either use the thread pool used for block writes, or somehow isolate
> low cost path ops (GET, DELETE) from the more expensive calls (COPY, PUT) so
> that a thread doing basic IO doesn't block for the duration of the long op.
> Maybe also use {{Semaphore.tryAcquire()}} and only start async work if there
> actually is an idle thread, doing it synchronously if not. Maybe it depends
> on the operation. path query/cleanup before/after a write is something which
> could be scheduled as just more futures to schedule in the block write.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14302) Test MR split optimisation with recursive listing
[ https://issues.apache.org/jira/browse/HADOOP-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14302: Parent: HADOOP-18067 (was: HADOOP-17566) > Test MR split optimisation with recursive listing > - > > Key: HADOOP-14302 > URL: https://issues.apache.org/jira/browse/HADOOP-14302 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.0.0-alpha2 >Reporter: Steve Loughran >Priority: Major > > Once MAPREDUCE-5907 is in, add an S3A test which uses the metrics to measure > what's happening, verify it works. If it can be made a scale test, even better -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16522) Encrypt S3A buffered data on disk
[
https://issues.apache.org/jira/browse/HADOOP-16522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16522:
Parent: HADOOP-18067 (was: HADOOP-17566)
> Encrypt S3A buffered data on disk
> -
>
> Key: HADOOP-16522
> URL: https://issues.apache.org/jira/browse/HADOOP-16522
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Mike Yoder
>Priority: Major
>
> This came out of discussions with [~ste...@apache.org], [~irashid] and
> [~vanzin].
> Imran:
> {quote}
> Steve pointed out to me that the s3 libraries buffer data to disk. This is
> pretty much arbitrary user data.
>
> Spark has some settings to encrypt data that it writes to local disk (shuffle
> files etc.). Spark never has control of what arbitrary libraries are doing
> with data, so it doesn't guarantee that nothing ever ends up on disk -- but
> to the end user, they'd view those s3 libraries as part of the same system.
> So if a user is turning on spark's local-disk encryption, the users would be
> pretty surprised to find out that the data they're writing to S3 ends up on
> local-disk, unencrypted.
> {quote}
> Me:
> {quote}
> ... Regardless, this is still an s3a bug.
> {quote}
>
> Steve:
> {quote}
> I disagree
> we need to save intermediate data "somewhere" -people get a choice of disk or
> memory.
> encrypting data on disk was never considered as needed, on the basis that
> anyone malicious with read access under your home dir could lift the hadoop
> token file which YARN provides and so have full R/W access to all your data
> in the cluster filesystems until those tokens expire. If you don't have a
> good story there then the buffering of a few tens of MB of data during upload
> is a detail.
> There's also the extra complication that when uploading file blocks, we pass
> in the filename to the AWS SDK and let it do the uploads, rather than create
> the output stream; the SDK code has, in the past, been better at recovering
> failures there than output stream + mark and reset. that was a while back;
> things may change. But it is why I'd prefer any encrypted temp store as a new
> buffer option, rather than just silently change the "disk" buffer option to
> encrypt
> Be interesting to see where else in the code this needs to be addressed; I'd
> recommend looking at all uses if org.apache.hadoop.fs.LocalDirAllocator and
> making sure that Spark YARN launch+execute didn't use this indirectly
> JIRAs under HADOOP-15620 welcome; do look at the test policy in the
> hadoop-aws docs; we'd need a new subclass of AbstractSTestS3AHugeFiles for
> integration testing a different buffering option, plus whatever unit tests
> the encryption itself needed.
> {quote}
> Me:
> {quote}
> I get it. But ... there are a couple of subtleties here. One is that the
> tokens expire, while the data is still data. (This might or might not matter,
> depending on the threat...) Another is that customer policies in this area do
> not always align well with common sense. There are blanket policies like
> "data shall never be written to disk unencrypted" which we have come up
> against, which we'd like to be able to honestly answer in the affirmative.
> We have encrypted MR shuffle as one historical example, and encrypted impala
> memory spills as another.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17242) S3A (async) ObjectListingIterator to block in hasNext() for results
[ https://issues.apache.org/jira/browse/HADOOP-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17242: Parent: HADOOP-18067 (was: HADOOP-17566) > S3A (async) ObjectListingIterator to block in hasNext() for results > --- > > Key: HADOOP-17242 > URL: https://issues.apache.org/jira/browse/HADOOP-17242 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > HADOOP-17074 made listing async in S3A, but the iterator's hasNext Call > doesn't wait for the result. If invoked on an empty path it *may* return when > it should be failing. > Note: surfaced in code review, not seen in the wild and all our tests were > happy -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13611) FileSystem/s3a processDeleteOnExit to skip the exists() check
[
https://issues.apache.org/jira/browse/HADOOP-13611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13611:
Parent: HADOOP-18067 (was: HADOOP-17566)
> FileSystem/s3a processDeleteOnExit to skip the exists() check
> -
>
> Key: HADOOP-13611
> URL: https://issues.apache.org/jira/browse/HADOOP-13611
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/s3
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Priority: Trivial
>
> If you look at {{FileSystem.processDeleteOnExit()}}, it does an exists()
> check for each entry, before calling delete().
> That exists() check is superfluous; on s3 it add an extra 1-4 HTTP GETs
> This could be fixed with a subclass in s3a to avoid it, but as the call is
> superfluous in *all* filesystems, it could be removed in {{FileSystem}} and
> so picked up by all object stores.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15570) s3guard uploads command to list date and initiator of outstanding uploads
[
https://issues.apache.org/jira/browse/HADOOP-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15570:
Parent: HADOOP-18067 (was: HADOOP-17566)
> s3guard uploads command to list date and initiator of outstanding uploads
> -
>
> Key: HADOOP-15570
> URL: https://issues.apache.org/jira/browse/HADOOP-15570
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Minor
>
> The S3Guard uploads command doesn't provide that much info about outstanding
> uploads, such as who started them, and when
> {code}
> haooop s3guard uploads s3a/...
> tests3ascale/scale/hugefile
> nB3gctgP34ZSpzJ5_o2AVb7kKRiicXfbkBqIflA0y.IH7CimH1O0VUzohRCj3QfFstoQIdcge478ZidXu764yN0vuGI1j5kcOV3rDhsrc.RBZ5skZ93jVCN9g2c21QgB
> Total 1 uploads found.
> {code}
> Enough to to know its there, but its nice to know: how long, who and how much
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15782) Clarify committers.md around v2 failure handling
[
https://issues.apache.org/jira/browse/HADOOP-15782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15782:
Parent: HADOOP-18067 (was: HADOOP-17566)
> Clarify committers.md around v2 failure handling
>
>
> Key: HADOOP-15782
> URL: https://issues.apache.org/jira/browse/HADOOP-15782
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation
>Affects Versions: 3.1.0, 3.1.1
>Reporter: Gera Shegalov
>Priority: Major
>
> The doc file
> {{hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md}}
> refers to the default file output committer (v2) as not supporting job and
> task recovery throughout the doc:
> {quote}or just by rerunning everything (The "v2" algorithm and Spark).
> {quote}
> This is incorrect.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14661) S3A to support Requester Pays Buckets
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14661: Parent: HADOOP-18067 (was: HADOOP-17566) > S3A to support Requester Pays Buckets > - > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15651) NPE in S3AInputStream.read() in ITestS3AInconsistency.testOpenFailOnRead
[
https://issues.apache.org/jira/browse/HADOOP-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15651:
Parent: HADOOP-18067 (was: HADOOP-17566)
> NPE in S3AInputStream.read() in ITestS3AInconsistency.testOpenFailOnRead
>
>
> Key: HADOOP-15651
> URL: https://issues.apache.org/jira/browse/HADOOP-15651
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Priority: Major
> Attachments: org.apache.hadoop.fs.s3a.ITestS3AInconsistency-output.txt
>
>
> Test {{ITestS3AInconsistency.testOpenFailOnRead()}} raise an NPE in read();
> could only happen on that line if {{wrappedStream==null}}, which implies that
> previous attempts to re-open the closed stream failed and that this didn't
> trigger anything.
> Not seen this before, but given that the fault injection is random, it may be
> that so far test runs have been *unlucky* and missed this.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15628) S3A Filesystem does not check return from AmazonS3Client deleteObjects
[
https://issues.apache.org/jira/browse/HADOOP-15628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15628:
Parent: HADOOP-18067 (was: HADOOP-17566)
> S3A Filesystem does not check return from AmazonS3Client deleteObjects
> --
>
> Key: HADOOP-15628
> URL: https://issues.apache.org/jira/browse/HADOOP-15628
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.9.1, 2.8.4, 3.1.1, 3.0.3
> Environment: Hadoop 3.0.2 / Hadoop 2.8.3
> Hive 2.3.2 / Hive 2.3.3 / Hive 3.0.0
> Non-AWS S3 implementation
>Reporter: Steve Jacobs
>Priority: Minor
>
> Deletes in S3A that use the Multi-Delete functionality in the Amazon S3 api
> do not check to see if all objects have been succesfully delete. In the event
> of a failure, the api will still return a 200 OK (which isn't checked
> currently):
> [Delete Code from Hadoop
> 2.8|https://github.com/apache/hadoop/blob/a0da1ec01051108b77f86799dd5e97563b2a3962/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L574]
>
> {code:java}
> if (keysToDelete.size() == MAX_ENTRIES_TO_DELETE) {
> DeleteObjectsRequest deleteRequest =
> new DeleteObjectsRequest(bucket).withKeys(keysToDelete);
> s3.deleteObjects(deleteRequest);
> statistics.incrementWriteOps(1);
> keysToDelete.clear();
> }
> {code}
> This should be converted to use the DeleteObjectsResult class from the
> S3Client:
> [Amazon Code
> Example|https://docs.aws.amazon.com/AmazonS3/latest/dev/DeletingMultipleObjectsUsingJava.htm]
> {code:java}
> // Verify that the objects were deleted successfully.
> DeleteObjectsResult delObjRes =
> s3Client.deleteObjects(multiObjectDeleteRequest); int successfulDeletes =
> delObjRes.getDeletedObjects().size();
> System.out.println(successfulDeletes + " objects successfully deleted.");
> {code}
> Bucket policies can be misconfigured, and deletes will fail without warning
> by S3A clients.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16567) S3A Secret access to fall back to XML if credential provider raises IOE.
[
https://issues.apache.org/jira/browse/HADOOP-16567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16567:
Parent: HADOOP-18067 (was: HADOOP-17566)
> S3A Secret access to fall back to XML if credential provider raises IOE.
>
>
> Key: HADOOP-16567
> URL: https://issues.apache.org/jira/browse/HADOOP-16567
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.2
>Reporter: Steve Loughran
>Priority: Major
>
> This is hive related. Hive can put secrets into a JCEKS file which only hive
> may read.
> S3A file systems created on behalf of a user do not have access to this file.
> Yet it is listed as the credential provider in the
> hadoop.credential.providers option in core-site -and that is marked as final.
> When the S3 a initializre() method looks up passwords and encryption keys,
> the failure to open the file raises an IOE -and the FS cannot be instantiated.
> Proposed: {{S3AUtils.lookupPassword()}} to catch such exceptions, and fall
> back to using {{Configuration.get()}} and so retrieve any property in the
> XML. If there is one failing here, it is if the user did want to read from a
> credential provider, the failure to read the credential will be lost, and the
> filesystem will simply get the default value.
> There is a side issue, that permission exceptions can surface as found not
> found exceptions, which are then wrapped as generic IOEs in Configuration. It
> will be hard and brittle to attempt to only respond to permission
> restrictions. We could look at improving {{Configuration.getPassword()}} but
> that class is so widely used, I am not in a rush to break things.
> I think this means we have to add another option. Trying to be clever about
> when to fall back versus when to rethrow the exception is doomed.
> If this works for S3A, we will need to consider replicating it for ABFS.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16134) Add S3AWriteOpContext for write ops; pass in statistics and other settings
[
https://issues.apache.org/jira/browse/HADOOP-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16134:
Parent: HADOOP-18067 (was: HADOOP-17566)
> Add S3AWriteOpContext for write ops; pass in statistics and other settings
> --
>
> Key: HADOOP-16134
> URL: https://issues.apache.org/jira/browse/HADOOP-16134
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Major
>
> As discussed in HADOOP-16090; having an {{S3AWriteOpContext}} to match the
> read context lets us do some interesting things
> * if we record the state of the file (is this an overwrite)? we'll know not
> to bother with deleting any parent dir markers (Assumption: there are none(
> * If a check for a parent dir was made (which we don't currently do, but...)
> then iff that was an empty dir marker, we'd know to *only* delete that marker
> and not go further up the tree.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16571) s3a to improve diags on s3a bad request message
[ https://issues.apache.org/jira/browse/HADOOP-16571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16571: Parent: HADOOP-18067 (was: HADOOP-17566) > s3a to improve diags on s3a bad request message > --- > > Key: HADOOP-16571 > URL: https://issues.apache.org/jira/browse/HADOOP-16571 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran >Priority: Minor > > if you have the endpoint or signing algorithm wrong, your first sign of > trouble on an s3a FS operation is a 400 bad request during init. > Proposed: include those properties on failure. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16066) S3a DelegationToken bindings to to support a "correlation ID" for the UA header
[ https://issues.apache.org/jira/browse/HADOOP-16066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16066: Parent: HADOOP-18067 (was: HADOOP-17566) > S3a DelegationToken bindings to to support a "correlation ID" for the UA > header > --- > > Key: HADOOP-16066 > URL: https://issues.apache.org/jira/browse/HADOOP-16066 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran >Priority: Minor > > to help track down which user is using an assigned role from a DT, allow the > DT binding to provide some kind of DT/correlation ID. The binding can then > build that up when the token is issued, and then serve it up later > If this is picked up and added to the UA header, then S3 logs can let you go > backwards from requests to the specific DT issues/used, and then even the > principal. > For the standard bindings, we'd return: principal + UUID -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15650) Add custom InstanceProfileCredentialsProvider with more resilience to throttling
[
https://issues.apache.org/jira/browse/HADOOP-15650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15650:
Parent: HADOOP-18067 (was: HADOOP-17566)
> Add custom InstanceProfileCredentialsProvider with more resilience to
> throttling
>
>
> Key: HADOOP-15650
> URL: https://issues.apache.org/jira/browse/HADOOP-15650
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Minor
>
> Add our own InstanceProfileCredentialsProvider class which uses the AWS
> implementation to retrieve credentials from EC2's instance info, but more
> resilient to overloading.
> # pass in client config with retry logic (HADOOP-15603)
> # use Invoke.retry() to retry
> # log/measure failures
> # maybe use the Async feature of the AWS SDK class, so that credential
> renewer doesn't block IO.
> # be shared amongst all AWS auth chains which need these credentials.
> The singleton we current use for IAM auth doesn't do async, which is good as
> it ensures that we don't prematurely close it when
> {{AWSCredentialProviderList.close()}} closes its children.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17063) S3A deleteObjects hanging/retrying forever
[
https://issues.apache.org/jira/browse/HADOOP-17063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-17063:
Parent: HADOOP-18067 (was: HADOOP-17566)
> S3A deleteObjects hanging/retrying forever
> --
>
> Key: HADOOP-17063
> URL: https://issues.apache.org/jira/browse/HADOOP-17063
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.1
> Environment: hadoop 3.2.1
> spark 2.4.4
>
>Reporter: Dyno
>Priority: Minor
> Attachments: jstack_exec-34.log, jstack_exec-40.log,
> jstack_exec-74.log
>
>
> {code}
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:523)
> com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:82)
>
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.putObject(S3ABlockOutputStream.java:446)
>
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:365)
>
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> org.apache.parquet.hadoop.util.HadoopPositionOutputStream.close(HadoopPositionOutputStream.java:64)
> org.apache.parquet.hadoop.ParquetFileWriter.end(ParquetFileWriter.java:685)
> org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:122)
>
> org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:165)
>
> org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetOutputWriter.scala:42)
>
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:57)
>
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:74)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:247)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
>
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
>
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
> org.apache.spark.scheduler.Task.run(Task.scala:123)
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {code}
>
> we are using spark 2.4.4 with hadoop 3.2.1 on kubernetes/spark-operator,
> sometimes we see this hang with the stacktrace above. it looks like the
> putObject never return, we have to kill the executor to make the job move
> forward.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16164) S3aDelegationTokens to add accessor for tests to get at the token binding
[ https://issues.apache.org/jira/browse/HADOOP-16164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16164: Parent: HADOOP-18067 (was: HADOOP-17566) > S3aDelegationTokens to add accessor for tests to get at the token binding > - > > Key: HADOOP-16164 > URL: https://issues.apache.org/jira/browse/HADOOP-16164 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Major > > For testing, it turns out to be useful to get at the current token binding in > the S3ADelegationTokens instance of a filesystem. > provide an accessor, tagged as for testing only -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13704) S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use
[
https://issues.apache.org/jira/browse/HADOOP-13704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13704:
Parent: HADOOP-18067 (was: HADOOP-17566)
> S3A getContentSummary() to move to listFiles(recursive) to count children;
> instrument use
> -
>
> Key: HADOOP-13704
> URL: https://issues.apache.org/jira/browse/HADOOP-13704
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Minor
>
> Hive and a bit of Spark use {{getContentSummary()}} to get some summary stats
> of a filesystem. This is very expensive on S3A (and any other object store),
> especially as the base implementation does the recursive tree walk.
> Because of HADOOP-13208, we have a full enumeration of files under a path
> without directory costs...S3A can/should switch to this to speed up those
> places where the operation is called.
> Also
> * API call needs FS spec and contract tests
> * S3A could instrument invocation, so as to enable real-world popularity to
> be measured
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17746) compatibility table in directory_markers.md doesn't render right
[ https://issues.apache.org/jira/browse/HADOOP-17746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17746: Parent: HADOOP-18067 (was: HADOOP-17566) > compatibility table in directory_markers.md doesn't render right > > > Key: HADOOP-17746 > URL: https://issues.apache.org/jira/browse/HADOOP-17746 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, site >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: Screenshot 2021-06-03 at 20.44.19.png > > > The compatibility table in the directory markers doc doesn't render right; > looks like what is meant to be the top and bottom of the table are being > interpreted as horizontal lines, which confuses the maven site code > Renders ~OK on github > https://github.com/apache/hadoop/blob/branch-3.3.1/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/directory_markers.md -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12878) Impersonate hosts in s3a for better data locality handling
[
https://issues.apache.org/jira/browse/HADOOP-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-12878:
Parent: HADOOP-18067 (was: HADOOP-17566)
> Impersonate hosts in s3a for better data locality handling
> --
>
> Key: HADOOP-12878
> URL: https://issues.apache.org/jira/browse/HADOOP-12878
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Thomas Demoor
>Assignee: Thomas Demoor
>Priority: Major
>
> Currently, {{localhost}} is passed as locality for each block, causing all
> blocks involved in job to initially target the same node (RM), before being
> moved by the scheduler (to a rack-local node). This reduces parallelism for
> jobs (with short-lived mappers).
> We should mimic Azures implementation: a config setting
> {{fs.s3a.block.location.impersonatedhost}} where the user can enter the list
> of hostnames in the cluster to return to {{getFileBlockLocations}}.
> Possible optimization: for larger systems, it might be better to return N
> (5?) random hostnames to prevent passing a huge array (the downstream code
> assumes size = O(3)).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13276) S3a operations keep retrying if the password is wrong
[
https://issues.apache.org/jira/browse/HADOOP-13276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13276:
Parent: HADOOP-18067 (was: HADOOP-17566)
> S3a operations keep retrying if the password is wrong
> -
>
> Key: HADOOP-13276
> URL: https://issues.apache.org/jira/browse/HADOOP-13276
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.4
>Reporter: Steve Loughran
>Assignee: Thomas Poepping
>Priority: Minor
>
> If you do a {{hadoop fs}} command with the AWS account valid but the password
> wrong, it takes a while to timeout, because of retries happening underneath.
> Eventually it gives up, but failing fast would be better.
> # maybe: check the password length and fail if it is not the right length (is
> there a standard one? Or at least a range?)
> # consider a retry policy which fails faster on signature failures/403
> responses
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14334) S3 SSEC tests to downgrade when running against a mandatory encryption object store
[ https://issues.apache.org/jira/browse/HADOOP-14334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14334: Parent: HADOOP-18067 (was: HADOOP-17566) > S3 SSEC tests to downgrade when running against a mandatory encryption > object store > > > Key: HADOOP-14334 > URL: https://issues.apache.org/jira/browse/HADOOP-14334 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Minor > > If you run the S3A tests against a bucket with mandatory encryption, you need > to set the encryption in auth keys. This breaks the SSEC tests because the > encryption.key property being set breaks them. > That can be fixed (unset it), but they also need to handle failure from the > wrong encryption mech being used. Proposed: catch and downgrade to skipped. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16940) ITestCustomSigner uses absolute paths off the bucket root rather than fork-relative
[ https://issues.apache.org/jira/browse/HADOOP-16940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16940: Parent: HADOOP-18067 (was: HADOOP-17566) > ITestCustomSigner uses absolute paths off the bucket root rather than > fork-relative > --- > > Key: HADOOP-16940 > URL: https://issues.apache.org/jira/browse/HADOOP-16940 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Minor > > ITestCustomSigner.testCustomSignerAndInitializer is creating a a couple of > paths (/customsignerpath1, /customsignerpath2) rather than in a specific fork. > the paths are uniquely named enough to be low risk, but they should be moved > to being fork-relative -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16480) S3 Select Exceptions are not being converted to IOEs
[ https://issues.apache.org/jira/browse/HADOOP-16480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16480: Parent: HADOOP-18067 (was: HADOOP-17566) > S3 Select Exceptions are not being converted to IOEs > > > Key: HADOOP-16480 > URL: https://issues.apache.org/jira/browse/HADOOP-16480 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Minor > > Network outage seems to have raised a SelectObjectContentEventException > exception; it's not been translated to an IOE. > Issue: recoverable or not? A normal input stream would try to recover by > re-opening at the current position, but to restart a seek you'd have to > repeat the entire streaming. > For now, fail. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13728) S3A can support short user-friendly aliases for configuration of credential providers.
[ https://issues.apache.org/jira/browse/HADOOP-13728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13728: Parent: HADOOP-18067 (was: HADOOP-17566) > S3A can support short user-friendly aliases for configuration of credential > providers. > -- > > Key: HADOOP-13728 > URL: https://issues.apache.org/jira/browse/HADOOP-13728 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Priority: Minor > > This issue proposes to support configuration of the S3A credential provider > chain using short aliases to refer to the common credential providers in > addition to allowing full class names. Supporting short aliases would > provide a simpler operations experience for the most common cases. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13636) s3a rm on the CLI generates deprecation warning on io.bytes.per.checksum
[
https://issues.apache.org/jira/browse/HADOOP-13636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13636:
Parent: HADOOP-18067 (was: HADOOP-17566)
> s3a rm on the CLI generates deprecation warning on io.bytes.per.checksum
>
>
> Key: HADOOP-13636
> URL: https://issues.apache.org/jira/browse/HADOOP-13636
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Minor
>
> an rm -r triggers a deprecation warning of {{io.bytes.per.checksum}}. This
> is a property that is not explicitly used anywhere in S3: something else is
> asking for it —something that should switch to the new value
> {code}
> $ ./hadoop fs -rm -r s3a://XYZ/lib
> 16/09/21 16:24:04 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 16/09/21 16:24:05 INFO Configuration.deprecation: io.bytes.per.checksum is
> deprecated. Instead, use dfs.bytes-per-checksum
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
