[GitHub] [hadoop] anujmodi2021 commented on a diff in pull request #5987: HADOOP-18865 ABFS: Adding "100-continue" in userAgent String if enabled
anujmodi2021 commented on code in PR #5987:
URL: https://github.com/apache/hadoop/pull/5987#discussion_r1309675455
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsClient.java:
##
@@ -196,6 +197,41 @@ public void verifyUserAgentPrefix()
.doesNotContain(FS_AZURE_USER_AGENT_PREFIX);
}
+ /**
+ * This method represents a unit test for verifying the behavior of the
User-Agent header
+ * with respect to the "Expect: 100-continue" header setting in the Azure
Blob File System (ABFS) configuration.
+ *
+ * The test ensures that the User-Agent string includes or excludes specific
information based on whether the
+ * "Expect: 100-continue" header is enabled or disabled in the configuration.
+ *
+ */
+ @Test
+ public void verifyUserAgentExpectHeader()
+ throws IOException, IllegalAccessException {
+final Configuration configuration = new Configuration();
+configuration.addResource(TEST_CONFIGURATION_FILE_NAME);
+configuration.set(ConfigurationKeys.FS_AZURE_USER_AGENT_PREFIX_KEY,
FS_AZURE_USER_AGENT_PREFIX);
+
configuration.setBoolean(ConfigurationKeys.FS_AZURE_ACCOUNT_IS_EXPECT_HEADER_ENABLED,
true);
+AbfsConfiguration abfsConfiguration = new AbfsConfiguration(configuration,
+ACCOUNT_NAME);
+String userAgentStr = getUserAgentString(abfsConfiguration, false);
+
+verifybBasicInfo(userAgentStr);
Review Comment:
nit: Typo: verify_b_BasicInfo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18865) ABFS: Adding 100 continue in userAgent String and dynamically removing it if retry is without the header enabled.
[
https://issues.apache.org/jira/browse/HADOOP-18865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760241#comment-17760241
]
ASF GitHub Bot commented on HADOOP-18865:
-
anujmodi2021 commented on code in PR #5987:
URL: https://github.com/apache/hadoop/pull/5987#discussion_r1309675455
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsClient.java:
##
@@ -196,6 +197,41 @@ public void verifyUserAgentPrefix()
.doesNotContain(FS_AZURE_USER_AGENT_PREFIX);
}
+ /**
+ * This method represents a unit test for verifying the behavior of the
User-Agent header
+ * with respect to the "Expect: 100-continue" header setting in the Azure
Blob File System (ABFS) configuration.
+ *
+ * The test ensures that the User-Agent string includes or excludes specific
information based on whether the
+ * "Expect: 100-continue" header is enabled or disabled in the configuration.
+ *
+ */
+ @Test
+ public void verifyUserAgentExpectHeader()
+ throws IOException, IllegalAccessException {
+final Configuration configuration = new Configuration();
+configuration.addResource(TEST_CONFIGURATION_FILE_NAME);
+configuration.set(ConfigurationKeys.FS_AZURE_USER_AGENT_PREFIX_KEY,
FS_AZURE_USER_AGENT_PREFIX);
+
configuration.setBoolean(ConfigurationKeys.FS_AZURE_ACCOUNT_IS_EXPECT_HEADER_ENABLED,
true);
+AbfsConfiguration abfsConfiguration = new AbfsConfiguration(configuration,
+ACCOUNT_NAME);
+String userAgentStr = getUserAgentString(abfsConfiguration, false);
+
+verifybBasicInfo(userAgentStr);
Review Comment:
nit: Typo: verify_b_BasicInfo
> ABFS: Adding 100 continue in userAgent String and dynamically removing it if
> retry is without the header enabled.
> -
>
> Key: HADOOP-18865
> URL: https://issues.apache.org/jira/browse/HADOOP-18865
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Affects Versions: 3.3.6
>Reporter: Anmol Asrani
>Assignee: Anmol Asrani
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.3.6
>
>
> Adding 100 continue in userAgent String if enabled in AbfsConfiguration and
> dynamically removing it if retry is without the header enabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5957: YARN-6537. Consider running RM tests against the Router.
hadoop-yetus commented on PR #5957:
URL: https://github.com/apache/hadoop/pull/5957#issuecomment-1698499545
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 10 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 30s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 7s | | trunk passed |
| +1 :green_heart: | compile | 8m 4s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 7m 17s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 55s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 40s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +0 :ok: | spotbugs | 0m 36s | |
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
no spotbugs output file (spotbugsXml.xml) |
| -1 :x: | shadedclient | 36m 59s | | branch has errors when building
and testing our client artifacts. |
| -0 :warning: | patch | 37m 23s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 53s | | the patch passed |
| +1 :green_heart: | compile | 7m 22s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 7m 22s | | the patch passed |
| +1 :green_heart: | compile | 7m 11s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 7m 11s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 51s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5957/6/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 202 unchanged
- 0 fixed = 203 total (was 202) |
| +1 :green_heart: | mvnsite | 2m 51s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 35s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 26s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +0 :ok: | spotbugs | 0m 32s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests has
no data from spotbugs |
| -1 :x: | shadedclient | 37m 38s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 7s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 24m 18s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | unit | 3m 24s | | hadoop-yarn-server-tests in the
patch passed. |
| +1 :green_heart: | unit | 0m 37s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 54s | | The patch does not
generate ASF License warnings. |
| | | 225m 45s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5957/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5957 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 509caa4856d0 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git
[GitHub] [hadoop] hadoop-yetus commented on pull request #5978: YARN-11552. Timeline endpoint: /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store
hadoop-yetus commented on PR #5978:
URL: https://github.com/apache/hadoop/pull/5978#issuecomment-1698501086
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 16s | | trunk passed |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 56s | | trunk passed |
| -1 :x: | shadedclient | 33m 12s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 53s | | the patch passed |
| -1 :x: | shadedclient | 36m 20s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 48s | |
hadoop-yarn-server-timelineservice in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 128m 28s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5978/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5978 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d2e8e5b62633 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9ffe870fd74244842ec7a16982b1f2cdf0d9d79f |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5978/8/testReport/ |
| Max. process+thread count | 418 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5978/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[GitHub] [hadoop] hadoop-yetus commented on pull request #5990: HDFS-17166. RBF: Throwing NoNamenodesAvailableException for a long time, when failover
hadoop-yetus commented on PR #5990:
URL: https://github.com/apache/hadoop/pull/5990#issuecomment-1698491399
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 31s | | trunk passed |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 29s | | trunk passed |
| -1 :x: | shadedclient | 33m 58s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 22s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 23s | | the patch passed |
| -1 :x: | shadedclient | 32m 45s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 22m 45s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 149m 33s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5990/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5990 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 9ff633bec736 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e41f72c304cfd8de4ec13392aaed6271af0b1a39 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5990/5/testReport/ |
| Max. process+thread count | 2708 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5990/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5978: YARN-11552. Timeline endpoint: /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store
hadoop-yetus commented on PR #5978:
URL: https://github.com/apache/hadoop/pull/5978#issuecomment-1698494050
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 29s | | trunk passed |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 38s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 54s | | trunk passed |
| -1 :x: | shadedclient | 31m 7s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 49s | | the patch passed |
| -1 :x: | shadedclient | 31m 1s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 44s | |
hadoop-yarn-server-timelineservice in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 118m 30s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5978/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5978 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 02da51c9ee2e 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9ffe870fd74244842ec7a16982b1f2cdf0d9d79f |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5978/7/testReport/ |
| Max. process+thread count | 454 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5978/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[jira] [Commented] (HADOOP-18860) Upgrade mockito to 4.11.0
[
https://issues.apache.org/jira/browse/HADOOP-18860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760218#comment-17760218
]
ASF GitHub Bot commented on HADOOP-18860:
-
hadoop-yetus commented on PR #5999:
URL: https://github.com/apache/hadoop/pull/5999#issuecomment-1698430119
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 11s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 14 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 14m 26s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 40m 29s | | branch-3.3 passed |
| +1 :green_heart: | compile | 19m 20s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 2m 55s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 12m 42s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 7m 36s | | branch-3.3 passed |
| +0 :ok: | spotbugs | 0m 20s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 52m 15s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 35s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 10m 12s | | the patch passed |
| +1 :green_heart: | compile | 18m 47s | | the patch passed |
| -1 :x: | javac | 18m 47s |
[/results-compile-javac-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5999/1/artifact/out/results-compile-javac-root.txt)
| root generated 9 new + 1872 unchanged - 1 fixed = 1881 total (was 1873) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 56s | | the patch passed |
| +1 :green_heart: | mvnsite | 12m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 7m 28s | | the patch passed |
| +0 :ok: | spotbugs | 0m 19s | | hadoop-project has no data from
spotbugs |
| -1 :x: | shadedclient | 51m 55s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 20s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 18m 7s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 221m 48s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | unit | 17m 36s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | unit | 22m 59s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | unit | 94m 45s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 20m 51s | | hadoop-yarn-services-core in
the patch passed. |
| -1 :x: | unit | 226m 11s |
[/patch-unit-hadoop-yarn-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5999/1/artifact/out/patch-unit-hadoop-yarn-project.txt)
| hadoop-yarn-project in the patch passed. |
| +1 :green_heart: | asflicense | 1m 14s | | The patch does not
generate ASF License warnings. |
| | | 922m 13s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5999/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5999 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux f7ea89e21e56 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / bba896597027149a3fc3b14ea16c435672c3d50a |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5999/1/testReport/
[GitHub] [hadoop] hadoop-yetus commented on pull request #5999: HADOOP-18860. Upgrade mockito version to 4.11.0 (branch-3.3)
hadoop-yetus commented on PR #5999:
URL: https://github.com/apache/hadoop/pull/5999#issuecomment-1698430119
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 11s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 14 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 14m 26s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 40m 29s | | branch-3.3 passed |
| +1 :green_heart: | compile | 19m 20s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 2m 55s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 12m 42s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 7m 36s | | branch-3.3 passed |
| +0 :ok: | spotbugs | 0m 20s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 52m 15s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 35s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 10m 12s | | the patch passed |
| +1 :green_heart: | compile | 18m 47s | | the patch passed |
| -1 :x: | javac | 18m 47s |
[/results-compile-javac-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5999/1/artifact/out/results-compile-javac-root.txt)
| root generated 9 new + 1872 unchanged - 1 fixed = 1881 total (was 1873) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 56s | | the patch passed |
| +1 :green_heart: | mvnsite | 12m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 7m 28s | | the patch passed |
| +0 :ok: | spotbugs | 0m 19s | | hadoop-project has no data from
spotbugs |
| -1 :x: | shadedclient | 51m 55s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 20s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 18m 7s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 221m 48s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | unit | 17m 36s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | unit | 22m 59s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | unit | 94m 45s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 20m 51s | | hadoop-yarn-services-core in
the patch passed. |
| -1 :x: | unit | 226m 11s |
[/patch-unit-hadoop-yarn-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5999/1/artifact/out/patch-unit-hadoop-yarn-project.txt)
| hadoop-yarn-project in the patch passed. |
| +1 :green_heart: | asflicense | 1m 14s | | The patch does not
generate ASF License warnings. |
| | | 922m 13s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5999/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5999 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux f7ea89e21e56 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / bba896597027149a3fc3b14ea16c435672c3d50a |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5999/1/testReport/ |
| Max. process+thread count | 2265 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf
[GitHub] [hadoop] zhtttylz commented on a diff in pull request #5992: HDFS-17168. Support getTrashRoots API in WebHDFS
zhtttylz commented on code in PR #5992:
URL: https://github.com/apache/hadoop/pull/5992#discussion_r1309570373
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java:
##
@@ -877,6 +878,40 @@ public static Map
getErasureCodeCodecs(Map json) {
return map;
}
+ public static Collection getTrashRoots(Map json) {
+List objs = (List) json.get("Paths");
+if (objs != null) {
+ FileStatus[] trashRoots = new FileStatus[objs.size()];
+ for (int i = 0; i < objs.size(); i++) {
+Map m = (Map) objs.get(i);
+trashRoots[i] = toFileStatus(m);
+ }
+ return Arrays.asList(trashRoots);
+}
+return new ArrayList(0);
+ }
+
+ public static FileStatus toFileStatus(Map json) {
+Path path = new Path(getString(json, "path", ""));
+long length = getLong(json, "length", 0);
+boolean isdir = getBoolean(json, "isdir", false);
Review Comment:
Thanks for your suggestion, I'll make the required code changes promptly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5990: HDFS-17166. RBF: Throwing NoNamenodesAvailableException for a long time, when failover
hadoop-yetus commented on PR #5990:
URL: https://github.com/apache/hadoop/pull/5990#issuecomment-1698406461
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 8s | | trunk passed |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 0s | | trunk passed |
| -1 :x: | shadedclient | 20m 30s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 53s | | the patch passed |
| -1 :x: | shadedclient | 20m 5s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 9s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 103m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5990/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5990 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 77fc3e6b5d7d 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fd893312f62b5d1f3308fb0f34f213798bab5492 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5990/4/testReport/ |
| Max. process+thread count | 2638 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5990/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hadoop] KeeProMise commented on a diff in pull request #5990: HDFS-17166. RBF: Throwing NoNamenodesAvailableException for a long time, when failover
KeeProMise commented on code in PR #5990:
URL: https://github.com/apache/hadoop/pull/5990#discussion_r1309545848
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java:
##
@@ -478,4 +478,41 @@ private List
getRecentRegistrationForQuery(
public void setRouterId(String router) {
this.routerId = router;
}
+
+ /**
+ * Rotate cache, make the current namenode have the lowest priority,
+ * to ensure that the current namenode will not be accessed first next time.
+ *
+ * @param nsId name service id
+ * @param namenode namenode contexts
+ * @param listObserversFirst Observer read case, observer NN will be ranked
first
+ */
+ @Override
+ public synchronized void rotateCache(
+ String nsId, FederationNamenodeContext namenode, boolean
listObserversFirst) {
+cacheNS.compute(Pair.of(nsId, listObserversFirst), (ns, namenodeContexts)
-> {
+ if (namenodeContexts == null || namenodeContexts.size() <= 1) {
+return namenodeContexts;
+ }
+ FederationNamenodeContext firstNamenodeContext = namenodeContexts.get(0);
+ /*
+ * If the first nn in the cache is active, the active nn priority cannot
be lowered.
+ * This happens when other threads have already updated the cache.
+ */
+ if(firstNamenodeContext.getState().equals(ACTIVE)) {
+return namenodeContexts;
+ }
+ /*
+ * If the first nn in the cache at this time is not the nn
+ * that needs to be lowered in priority, there is no need to rotate.
+ * This happens when other threads have already rotated the cache.
+ */
+
if(firstNamenodeContext.getRpcAddress().equals(namenode.getRpcAddress())) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #5992: HDFS-17168. Support getTrashRoots API in WebHDFS
slfan1989 commented on PR #5992: URL: https://github.com/apache/hadoop/pull/5992#issuecomment-1698373008 @zhtttylz Thank you for your contribution! The code looks fine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5992: HDFS-17168. Support getTrashRoots API in WebHDFS
slfan1989 commented on code in PR #5992:
URL: https://github.com/apache/hadoop/pull/5992#discussion_r1309521869
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java:
##
@@ -877,6 +878,40 @@ public static Map
getErasureCodeCodecs(Map json) {
return map;
}
+ public static Collection getTrashRoots(Map json) {
+List objs = (List) json.get("Paths");
+if (objs != null) {
+ FileStatus[] trashRoots = new FileStatus[objs.size()];
+ for (int i = 0; i < objs.size(); i++) {
+Map m = (Map) objs.get(i);
+trashRoots[i] = toFileStatus(m);
+ }
+ return Arrays.asList(trashRoots);
+}
+return new ArrayList(0);
+ }
+
+ public static FileStatus toFileStatus(Map json) {
+Path path = new Path(getString(json, "path", ""));
+long length = getLong(json, "length", 0);
+boolean isdir = getBoolean(json, "isdir", false);
Review Comment:
Can "isdir" be a constant？
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java:
##
@@ -877,6 +878,40 @@ public static Map
getErasureCodeCodecs(Map json) {
return map;
}
+ public static Collection getTrashRoots(Map json) {
+List objs = (List) json.get("Paths");
+if (objs != null) {
+ FileStatus[] trashRoots = new FileStatus[objs.size()];
+ for (int i = 0; i < objs.size(); i++) {
+Map m = (Map) objs.get(i);
+trashRoots[i] = toFileStatus(m);
+ }
+ return Arrays.asList(trashRoots);
+}
+return new ArrayList(0);
+ }
+
+ public static FileStatus toFileStatus(Map json) {
+Path path = new Path(getString(json, "path", ""));
+long length = getLong(json, "length", 0);
+boolean isdir = getBoolean(json, "isdir", false);
Review Comment:
Can "isdir" be a constant？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] KeeProMise commented on pull request #5990: HDFS-17166. RBF: Throwing NoNamenodesAvailableException for a long time, when failover
KeeProMise commented on PR #5990: URL: https://github.com/apache/hadoop/pull/5990#issuecomment-1698369478 > @KeeProMise Thank you for your contribution, LGTM. The information described is very detaile Thank you for your review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5967: YARN-11435. [Router] FederationStateStoreFacade is not reinitialized with Router conf.
slfan1989 commented on code in PR #5967:
URL: https://github.com/apache/hadoop/pull/5967#discussion_r1309504955
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/rmadmin/TestFederationRMAdminInterceptor.java:
##
@@ -120,7 +120,7 @@ public void setUp() {
// Initialize facade & stateSore
stateStore = new MemoryFederationStateStore();
stateStore.init(this.getConf());
-facade = FederationStateStoreFacade.getInstance();
+facade = FederationStateStoreFacade.getInstance(this.getConf());
Review Comment:
I will read the code carefully to ensure the uniformity of usage. If there
is `configuration` in the usage class involved, we will use
`FederationStateStoreFacade.getInstance(this.getConf())`, if there is no
`configuration` in the usage class involved, we will use Will use
`FederationStateStoreFacade.getInstance()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5967: YARN-11435. [Router] FederationStateStoreFacade is not reinitialized with Router conf.
slfan1989 commented on code in PR #5967:
URL: https://github.com/apache/hadoop/pull/5967#discussion_r1309502149
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/FederationPoliciesTestUtil.java:
##
@@ -244,7 +245,7 @@ public static FederationStateStoreFacade initFacade(
List subClusterInfos, SubClusterPolicyConfiguration
policyConfiguration) throws YarnException {
FederationStateStoreFacade goodFacade = FederationStateStoreFacade
-.getInstance();
+.getInstance(new Configuration());
Review Comment:
Thanks for your suggestion! I will provide an empty getInstance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #5990: HDFS-17166. RBF: Throwing NoNamenodesAvailableException for a long time, when failover
simbadzina commented on code in PR #5990:
URL: https://github.com/apache/hadoop/pull/5990#discussion_r1309499172
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java:
##
@@ -478,4 +478,41 @@ private List
getRecentRegistrationForQuery(
public void setRouterId(String router) {
this.routerId = router;
}
+
+ /**
+ * Rotate cache, make the current namenode have the lowest priority,
+ * to ensure that the current namenode will not be accessed first next time.
+ *
+ * @param nsId name service id
+ * @param namenode namenode contexts
+ * @param listObserversFirst Observer read case, observer NN will be ranked
first
+ */
+ @Override
+ public synchronized void rotateCache(
+ String nsId, FederationNamenodeContext namenode, boolean
listObserversFirst) {
+cacheNS.compute(Pair.of(nsId, listObserversFirst), (ns, namenodeContexts)
-> {
+ if (namenodeContexts == null || namenodeContexts.size() <= 1) {
+return namenodeContexts;
+ }
+ FederationNamenodeContext firstNamenodeContext = namenodeContexts.get(0);
+ /*
+ * If the first nn in the cache is active, the active nn priority cannot
be lowered.
+ * This happens when other threads have already updated the cache.
+ */
+ if(firstNamenodeContext.getState().equals(ACTIVE)) {
+return namenodeContexts;
+ }
+ /*
+ * If the first nn in the cache at this time is not the nn
+ * that needs to be lowered in priority, there is no need to rotate.
+ * This happens when other threads have already rotated the cache.
+ */
+
if(firstNamenodeContext.getRpcAddress().equals(namenode.getRpcAddress())) {
Review Comment:
You may get checkstyle errors from missing whitespaces after the `if`s
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #5990: HDFS-17166. RBF: Throwing NoNamenodesAvailableException for a long time, when failover
slfan1989 commented on PR #5990: URL: https://github.com/apache/hadoop/pull/5990#issuecomment-1698350184 @KeeProMise Thank you for your contribution, LGTM. The information described is very detailed! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] KeeProMise commented on a diff in pull request #5990: HDFS-17166. RBF: Throwing NoNamenodesAvailableException for a long time, when failover
KeeProMise commented on code in PR #5990:
URL: https://github.com/apache/hadoop/pull/5990#discussion_r1309488817
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java:
##
@@ -478,4 +478,40 @@ private List
getRecentRegistrationForQuery(
public void setRouterId(String router) {
this.routerId = router;
}
+
+ /**
+ * Rotate cache, make the current namenode have the lowest priority,
+ * to ensure that the current namenode will not be accessed first next time.
+ *
+ * @param nsId name service id
+ * @param namenode namenode contexts
+ */
+ @Override
+ public synchronized void rotateCache(String nsId, FederationNamenodeContext
namenode) {
+cacheNS.compute(Pair.of(nsId, false), (ns, namenodeContexts) -> {
+ if (namenodeContexts == null || namenodeContexts.size() <= 1) {
+return namenodeContexts;
+ }
+ FederationNamenodeContext firstNamenodeContext = namenodeContexts.get(0);
+ /*
+ * If the first nn in the cache is active, the active nn priority cannot
be lowered.
+ * This happens when other threads have already updated the cache.
+ */
+ if(firstNamenodeContext.getState().equals(ACTIVE)){
+return namenodeContexts;
+ }
+ /*
+ * If the first nn in the cache at this time is not the nn
+ * that needs to be lowered in priority, there is no need to rotate.
+ * This happens when other threads have already rotated the cache.
+ */
+
if(firstNamenodeContext.getRpcAddress().equals(namenode.getRpcAddress())){
+List rotatedNnContexts = new
ArrayList<>(namenodeContexts);
+Collections.rotate(rotatedNnContexts, -1);
+return rotatedNnContexts;
+ }else {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] KeeProMise commented on a diff in pull request #5990: HDFS-17166. RBF: Throwing NoNamenodesAvailableException for a long time, when failover
KeeProMise commented on code in PR #5990:
URL: https://github.com/apache/hadoop/pull/5990#discussion_r1309488607
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java:
##
@@ -478,4 +478,40 @@ private List
getRecentRegistrationForQuery(
public void setRouterId(String router) {
this.routerId = router;
}
+
+ /**
+ * Rotate cache, make the current namenode have the lowest priority,
+ * to ensure that the current namenode will not be accessed first next time.
+ *
+ * @param nsId name service id
+ * @param namenode namenode contexts
+ */
+ @Override
+ public synchronized void rotateCache(String nsId, FederationNamenodeContext
namenode) {
+cacheNS.compute(Pair.of(nsId, false), (ns, namenodeContexts) -> {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #5990: HDFS-17166. RBF: Throwing NoNamenodesAvailableException for a long time, when failover
goiri commented on code in PR #5990:
URL: https://github.com/apache/hadoop/pull/5990#discussion_r1309344400
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java:
##
@@ -478,4 +478,40 @@ private List
getRecentRegistrationForQuery(
public void setRouterId(String router) {
this.routerId = router;
}
+
+ /**
+ * Rotate cache, make the current namenode have the lowest priority,
+ * to ensure that the current namenode will not be accessed first next time.
+ *
+ * @param nsId name service id
+ * @param namenode namenode contexts
+ */
+ @Override
+ public synchronized void rotateCache(String nsId, FederationNamenodeContext
namenode) {
+cacheNS.compute(Pair.of(nsId, false), (ns, namenodeContexts) -> {
+ if (namenodeContexts == null || namenodeContexts.size() <= 1) {
+return namenodeContexts;
+ }
+ FederationNamenodeContext firstNamenodeContext = namenodeContexts.get(0);
+ /*
+ * If the first nn in the cache is active, the active nn priority cannot
be lowered.
+ * This happens when other threads have already updated the cache.
+ */
+ if(firstNamenodeContext.getState().equals(ACTIVE)){
+return namenodeContexts;
+ }
+ /*
+ * If the first nn in the cache at this time is not the nn
+ * that needs to be lowered in priority, there is no need to rotate.
+ * This happens when other threads have already rotated the cache.
+ */
+
if(firstNamenodeContext.getRpcAddress().equals(namenode.getRpcAddress())){
+List rotatedNnContexts = new
ArrayList<>(namenodeContexts);
+Collections.rotate(rotatedNnContexts, -1);
+return rotatedNnContexts;
+ }else {
Review Comment:
No need for else. You can just return.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #5990: HDFS-17166. RBF: Throwing NoNamenodesAvailableException for a long time, when failover
goiri commented on code in PR #5990:
URL: https://github.com/apache/hadoop/pull/5990#discussion_r1309343537
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java:
##
@@ -478,4 +476,27 @@ private List
getRecentRegistrationForQuery(
public void setRouterId(String router) {
this.routerId = router;
}
+
+ /**
+ * Shuffle cache, to ensure that the current nn will not be accessed first
next time.
+ *
+ *
+ * @param nsId name service id
+ * @param namenode namenode contexts
+ */
+ @Override
+ public synchronized void shuffleCache(String nsId, FederationNamenodeContext
namenode) {
+cacheNS.compute(Pair.of(nsId, false), (ns, namenodeContexts) -> {
+ if (namenodeContexts != null
+ && namenodeContexts.size() > 0
+ && !namenodeContexts.get(0).getState().equals(ACTIVE)
+ &&
namenodeContexts.get(0).getRpcAddress().equals(namenode.getRpcAddress())) {
+List rotatedNnContexts = new
ArrayList<>(namenodeContexts);
+Collections.rotate(rotatedNnContexts, -1);
Review Comment:
rotateCache looks better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #5990: HDFS-17166. RBF: Throwing NoNamenodesAvailableException for a long time, when failover
simbadzina commented on code in PR #5990:
URL: https://github.com/apache/hadoop/pull/5990#discussion_r1309343672
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java:
##
@@ -478,4 +478,40 @@ private List
getRecentRegistrationForQuery(
public void setRouterId(String router) {
this.routerId = router;
}
+
+ /**
+ * Rotate cache, make the current namenode have the lowest priority,
+ * to ensure that the current namenode will not be accessed first next time.
+ *
+ * @param nsId name service id
+ * @param namenode namenode contexts
+ */
+ @Override
+ public synchronized void rotateCache(String nsId, FederationNamenodeContext
namenode) {
+cacheNS.compute(Pair.of(nsId, false), (ns, namenodeContexts) -> {
Review Comment:
I think we need to refresh the cache for the key `Pair.of(nsId, true)` at
times.
You can extend the signature of rotateCache to
```java
public synchronized void rotateCache(String nsId, FederationNamenodeContext
namenode, boolean listObserversFirst)
```
And then call it with
```java
this.namenodeResolver.rotateCache(nsId, namenode, shouldUseObserver);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5888: HDFS-17121. BPServiceActor to provide new thread to handle FBR
hadoop-yetus commented on PR #5888:
URL: https://github.com/apache/hadoop/pull/5888#issuecomment-1697952247
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 55s | | trunk passed |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 11s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 38s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 20s | | trunk passed |
| -1 :x: | shadedclient | 36m 46s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 11s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 11s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5888/18/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | blanks | 0m 0s |
[/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5888/18/artifact/out/blanks-tabs.txt)
| The patch 260 line(s) with tabs. |
| +1 :green_heart: | checkstyle | 1m 0s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 33s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 27s | | the patch passed |
| -1 :x: | shadedclient | 37m 17s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 223m 43s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5888/18/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | asflicense | 0m 53s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5888/18/artifact/out/results-asflicense.txt)
| The patch generated 1 ASF License warnings. |
| | | 369m 46s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5888/18/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5888 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac
javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 093f0911da75 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 94d48778f690d22f44098edf073acbfd17a1cba4 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
[GitHub] [hadoop] hadoop-yetus commented on pull request #5996: HDFS-17171. CONGESTION_RATIO should be configurable
hadoop-yetus commented on PR #5996:
URL: https://github.com/apache/hadoop/pull/5996#issuecomment-1697962229
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 46m 37s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5996/3/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 13s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 35s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 13s | | trunk passed |
| -1 :x: | shadedclient | 33m 42s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 7s | | the patch passed |
| +1 :green_heart: | compile | 1m 11s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 4s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 1s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 11s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 9s | | the patch passed |
| -1 :x: | shadedclient | 33m 30s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 223m 16s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 55s | | The patch does not
generate ASF License warnings. |
| | | 361m 54s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5996/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5996 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 9c20276c8c87 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6452276e16d05e147e04f5dc63607af4154a7fb7 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5996/3/testReport/ |
| Max. process+thread count | 3337 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5996/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
[GitHub] [hadoop] hadoop-yetus commented on pull request #5947: HDFS-17158. Show the rate of metrics in EC recovery task.
hadoop-yetus commented on PR #5947:
URL: https://github.com/apache/hadoop/pull/5947#issuecomment-1697965692
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 24s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 26s | | trunk passed |
| -1 :x: | shadedclient | 40m 32s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 0s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 24s | | the patch passed |
| -1 :x: | shadedclient | 39m 36s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 240m 53s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5947/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 393m 56s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5947/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5947 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 74ac913e37a8 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d0a03ff45b4c73f9b0c21d8d6e221dee51e5fb28 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5947/5/testReport/ |
| Max. process+thread count | 2434 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5947/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This
[jira] [Commented] (HADOOP-18867) Upgrade ZooKeeper to 3.6.4
[
https://issues.apache.org/jira/browse/HADOOP-18867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760025#comment-17760025
]
ASF GitHub Bot commented on HADOOP-18867:
-
hadoop-yetus commented on PR #5988:
URL: https://github.com/apache/hadoop/pull/5988#issuecomment-1697778489
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 54s | | trunk passed |
| +1 :green_heart: | compile | 0m 19s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 19s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | mvnsite | 0m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 21s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| -1 :x: | shadedclient | 53m 30s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 12s | | the patch passed |
| +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 12s | | the patch passed |
| +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 12s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 12s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| -1 :x: | shadedclient | 21m 2s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 15s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 79m 45s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5988/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5988 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | Linux 78ebff6cafd1 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 91ae110ce18a1fb7b0c604b768f2400fe6cbc7ea |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5988/2/testReport/ |
| Max. process+thread count | 419 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5988/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Upgrade ZooKeeper to 3.6.4
> --
>
> Key: HADOOP-18867
> URL: https://issues.apache.org/jira/browse/HADOOP-18867
> Project: Hadoop Common
> Issue Type: Improvement
>
[GitHub] [hadoop] hadoop-yetus commented on pull request #5988: HADOOP-18867. Upgrade ZooKeeper to 3.6.4.
hadoop-yetus commented on PR #5988:
URL: https://github.com/apache/hadoop/pull/5988#issuecomment-1697778489
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 54s | | trunk passed |
| +1 :green_heart: | compile | 0m 19s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 19s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | mvnsite | 0m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 21s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| -1 :x: | shadedclient | 53m 30s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 12s | | the patch passed |
| +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 12s | | the patch passed |
| +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 12s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 12s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| -1 :x: | shadedclient | 21m 2s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 15s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 79m 45s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5988/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5988 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | Linux 78ebff6cafd1 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 91ae110ce18a1fb7b0c604b768f2400fe6cbc7ea |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5988/2/testReport/ |
| Max. process+thread count | 419 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5988/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands,
[jira] [Commented] (HADOOP-18871) S3ARetryPolicy to use sdk exception retryable() if it is valid
[
https://issues.apache.org/jira/browse/HADOOP-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760019#comment-17760019
]
Steve Loughran commented on HADOOP-18871:
-
+review S3AUtils.isMessageTranslatableToEOF() to see if that is still valid
> S3ARetryPolicy to use sdk exception retryable() if it is valid
> --
>
> Key: HADOOP-18871
> URL: https://issues.apache.org/jira/browse/HADOOP-18871
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Priority: Minor
>
> S3ARetryPolicy to use sdk exception retryable() if it is appropriate
> An initial {{RetryFromAWSClientIOException}} policy has been written, but not
> turned on as it there's too much risk of suddenly not retrying where we did
> today.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18868) Provide fallbacks for callqueue overflow trigger failover enable
[
https://issues.apache.org/jira/browse/HADOOP-18868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760021#comment-17760021
]
ASF GitHub Bot commented on HADOOP-18868:
-
hadoop-yetus commented on PR #5998:
URL: https://github.com/apache/hadoop/pull/5998#issuecomment-1697773096
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 3s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 36s | | trunk passed |
| +1 :green_heart: | compile | 10m 5s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 9m 6s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 2m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 14s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 58s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 7s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 38s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 20s | | the patch passed |
| +1 :green_heart: | compile | 9m 12s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 9m 11s | | the patch passed |
| +1 :green_heart: | compile | 9m 1s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 9m 1s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 25s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5998/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 3 new + 282 unchanged - 1 fixed = 285 total (was
283) |
| +1 :green_heart: | mvnsite | 2m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 53s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 6s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 57s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 20s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 23s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 193m 45s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 4s | | The patch does not
generate ASF License warnings. |
| | | 358m 35s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5998/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5998 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
markdownlint |
| uname | Linux ccd817dff8b5 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8b56e51f945ed37a77d4d9e1da96c5e5cf10ac88 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
[GitHub] [hadoop] hadoop-yetus commented on pull request #5998: HADOOP-18868. Provide fallbacks for callqueue overflow trigger failover enable
hadoop-yetus commented on PR #5998:
URL: https://github.com/apache/hadoop/pull/5998#issuecomment-1697773096
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 3s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 36s | | trunk passed |
| +1 :green_heart: | compile | 10m 5s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 9m 6s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 2m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 14s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 58s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 7s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 38s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 20s | | the patch passed |
| +1 :green_heart: | compile | 9m 12s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 9m 11s | | the patch passed |
| +1 :green_heart: | compile | 9m 1s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 9m 1s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 25s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5998/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 3 new + 282 unchanged - 1 fixed = 285 total (was
283) |
| +1 :green_heart: | mvnsite | 2m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 53s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 6s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 57s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 20s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 23s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 193m 45s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 4s | | The patch does not
generate ASF License warnings. |
| | | 358m 35s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5998/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5998 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
markdownlint |
| uname | Linux ccd817dff8b5 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8b56e51f945ed37a77d4d9e1da96c5e5cf10ac88 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5998/1/testReport/ |
| Max. process+thread count | 3894 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
[jira] [Created] (HADOOP-18871) S3ARetryPolicy to use sdk exception retryable() if it is valid
Steve Loughran created HADOOP-18871:
---
Summary: S3ARetryPolicy to use sdk exception retryable() if it is
valid
Key: HADOOP-18871
URL: https://issues.apache.org/jira/browse/HADOOP-18871
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.4.0
Reporter: Steve Loughran
S3ARetryPolicy to use sdk exception retryable() if it is appropriate
An initial {{RetryFromAWSClientIOException}} policy has been written, but not
turned on as it there's too much risk of suddenly not retrying where we did
today.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18862) [JDK17] MiniYarnClusters don't launch in hadoop-aws integration tests
[
https://issues.apache.org/jira/browse/HADOOP-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-18862:
Parent Issue: HADOOP-17177 (was: HADOOP-16795)
> [JDK17] MiniYarnClusters don't launch in hadoop-aws integration tests
> -
>
> Key: HADOOP-18862
> URL: https://issues.apache.org/jira/browse/HADOOP-18862
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Priority: Major
>
> I've tried running hadoop-aws tests under java17; everything which tries to
> launch a MiniYarnCluster fails because google guice is trying to stuff in
> java.land module that is now forbidden
> {code}
> Caused by: java.lang.ExceptionInInitializerError: Exception
> com.google.inject.internal.cglib.core.$CodeGenerationException:
> java.lang.reflect.InaccessibleObjectException-->Unable to make protected
> final java.lang.Class
> java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
> throws java.lang.ClassFormatError accessible: module java.base does not
> "opens java.lang" to unnamed module @7ee7980d [in thread "Thread-109"]
> {code}
> short term fix is to add the params to the surefire and failsafe jvm launcher
> to allow access
> {code}
> --add-opens java.base/java.lang=ALL-UNNAMED
> {code}
> I don't know if updating guice will make it go away completely. if it doesn't
> then the history server itself needs to be launched with this
> rather than just add an option for hadoop-aws, we ought to consider a general
> cross-module variable for junit.jvm.options which is set everywhere; the base
> impl is "" and a java profile could add the new stuff
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18747) AWS SDK V2 - sigv2 support
[ https://issues.apache.org/jira/browse/HADOOP-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760008#comment-17760008 ] Steve Loughran commented on HADOOP-18747: - pending/without this, HADOOP-18857 proposes failing better > AWS SDK V2 - sigv2 support > -- > > Key: HADOOP-18747 > URL: https://issues.apache.org/jira/browse/HADOOP-18747 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Priority: Major > > AWS SDK V2 does not support sigV2 signing. However, the S3 client supports > configurable signers so a custom sigV2 signer can be implemented and > configured. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18749) AWS SDK V2 - ITestS3AHugeFilesNoMultipart failure
[ https://issues.apache.org/jira/browse/HADOOP-18749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760006#comment-17760006 ] Steve Loughran commented on HADOOP-18749: - this is from HADOOP-18863; fix that and this test passes > AWS SDK V2 - ITestS3AHugeFilesNoMultipart failure > - > > Key: HADOOP-18749 > URL: https://issues.apache.org/jira/browse/HADOOP-18749 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Steve Loughran >Priority: Major > Fix For: 3.4.0 > > > ITestS3AHugeFilesNoMultipart fails with > java.lang.AssertionError: Expected a > org.apache.hadoop.fs.s3a.api.UnsupportedRequestException to be thrown, but > got the result: : true > Happens because the transfer manager currently does not do any MPU when used > with the Java async client, so the UnsupportedRequestException never gets > thrown. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18749) AWS SDK V2 - ITestS3AHugeFilesNoMultipart failure
[ https://issues.apache.org/jira/browse/HADOOP-18749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18749. - Fix Version/s: 3.4.0 Resolution: Duplicate > AWS SDK V2 - ITestS3AHugeFilesNoMultipart failure > - > > Key: HADOOP-18749 > URL: https://issues.apache.org/jira/browse/HADOOP-18749 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Steve Loughran >Priority: Major > Fix For: 3.4.0 > > > ITestS3AHugeFilesNoMultipart fails with > java.lang.AssertionError: Expected a > org.apache.hadoop.fs.s3a.api.UnsupportedRequestException to be thrown, but > got the result: : true > Happens because the transfer manager currently does not do any MPU when used > with the Java async client, so the UnsupportedRequestException never gets > thrown. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18818) Merge aws v2 upgrade feature branch into trunk
[
https://issues.apache.org/jira/browse/HADOOP-18818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760005#comment-17760005
]
ASF GitHub Bot commented on HADOOP-18818:
-
steveloughran commented on code in PR #5995:
URL: https://github.com/apache/hadoop/pull/5995#discussion_r1309008794
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/AWSCredentialProviderList.java:
##
@@ -330,4 +332,27 @@ public void close() {
public int size() {
return providers.size();
}
+
+
+ /**
+ * Translate an exception if it or its inner exception is an
+ * {@link CredentialInitializationException}.
+ * If this condition is not met, null is returned.
+ * @param path path of operation.
+ * @param throwable exception
+ * @return a translated exception or null.
+ */
+ public static IOException maybeTranslateCredentialException(String path,
Review Comment:
the audit and CredentialInitializationExceptions are both
SdkClientException. they may both be wrapped by async handler, and while I
didn't try to replicate it, if it happened in the same request I'd expect the
same failure. [and not expected in sync s3 client].
> Merge aws v2 upgrade feature branch into trunk
> --
>
> Key: HADOOP-18818
> URL: https://issues.apache.org/jira/browse/HADOOP-18818
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> do the merge, with everything we need as a blocker for that marked as a
> blocker of this task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18749) AWS SDK V2 - ITestS3AHugeFilesNoMultipart failure
[ https://issues.apache.org/jira/browse/HADOOP-18749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-18749: --- Assignee: Steve Loughran > AWS SDK V2 - ITestS3AHugeFilesNoMultipart failure > - > > Key: HADOOP-18749 > URL: https://issues.apache.org/jira/browse/HADOOP-18749 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Steve Loughran >Priority: Major > > ITestS3AHugeFilesNoMultipart fails with > java.lang.AssertionError: Expected a > org.apache.hadoop.fs.s3a.api.UnsupportedRequestException to be thrown, but > got the result: : true > Happens because the transfer manager currently does not do any MPU when used > with the Java async client, so the UnsupportedRequestException never gets > thrown. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5995: HADOOP-18818. Merge aws v2 upgrade feature branch into trunk
steveloughran commented on code in PR #5995:
URL: https://github.com/apache/hadoop/pull/5995#discussion_r1309008794
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/AWSCredentialProviderList.java:
##
@@ -330,4 +332,27 @@ public void close() {
public int size() {
return providers.size();
}
+
+
+ /**
+ * Translate an exception if it or its inner exception is an
+ * {@link CredentialInitializationException}.
+ * If this condition is not met, null is returned.
+ * @param path path of operation.
+ * @param throwable exception
+ * @return a translated exception or null.
+ */
+ public static IOException maybeTranslateCredentialException(String path,
Review Comment:
the audit and CredentialInitializationExceptions are both
SdkClientException. they may both be wrapped by async handler, and while I
didn't try to replicate it, if it happened in the same request I'd expect the
same failure. [and not expected in sync s3 client].
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18870) CURATOR-599 change broke functionality introduced in HADOOP-18139 and HADOOP-18709
[ https://issues.apache.org/jira/browse/HADOOP-18870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Erdelyi updated HADOOP-18870: Description: [[Curator PR#391 |https://github.com/apache/curator/pull/391/files#diff-687a4ed1252bfb4f56b3aeeb28bee4413b7df9bec4b969b72215587158ac875dR59] introduced a default method in the ZooKeeperFactory interface, hence the override of the 4-parameter NewZookeeper method in the HadoopZookeeperFactory class is not taking effect due to this. Proposing routing the 4-parameter method to a 5-parameter method, which instantiates the ZKConfiguration as the 5th parameter. This is a non-breaking change, as the ZKConfiguration is currently instantiated within the method. was: [Curator PR#391 |https://github.com/apache/curator/pull/391]introduced a default method in the ZooKeeperFactory interface, hence the override of the 4-parameter NewZookeeper method in the HadoopZookeeperFactory class is not taking effect. Proposing routing the 4-parameter method to a 5-parameter method, which instantiates the ZKConfiguration as the 5th parameter. This is a non-breaking change, as the ZKConfiguration is currently instantiated within the method. > CURATOR-599 change broke functionality introduced in HADOOP-18139 and > HADOOP-18709 > -- > > Key: HADOOP-18870 > URL: https://issues.apache.org/jira/browse/HADOOP-18870 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.4.0, 3.3.5 >Reporter: Ferenc Erdelyi >Assignee: Ferenc Erdelyi >Priority: Major > > [[Curator PR#391 > |https://github.com/apache/curator/pull/391/files#diff-687a4ed1252bfb4f56b3aeeb28bee4413b7df9bec4b969b72215587158ac875dR59] > introduced a default method in the ZooKeeperFactory interface, hence the > override of the 4-parameter NewZookeeper method in the HadoopZookeeperFactory > class is not taking effect due to this. > Proposing routing the 4-parameter method to a 5-parameter method, which > instantiates the ZKConfiguration as the 5th parameter. This is a non-breaking > change, as the ZKConfiguration is currently instantiated within the method. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18818) Merge aws v2 upgrade feature branch into trunk
[
https://issues.apache.org/jira/browse/HADOOP-18818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760002#comment-17760002
]
ASF GitHub Bot commented on HADOOP-18818:
-
steveloughran commented on code in PR #5995:
URL: https://github.com/apache/hadoop/pull/5995#discussion_r1309006133
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java:
##
@@ -139,6 +144,9 @@ public S3ARetryPolicy(Configuration conf) {
retryIdempotentCalls = new FailNonIOEs(
new IdempotencyRetryFilter(baseExponentialRetry));
+//retryAwsClientExceptions = new
RetryFromAWSClientIOException(retryIdempotentCalls);
+retryAwsClientExceptions = retryIdempotentCalls;
Review Comment:
i will create a jira and reference it
> Merge aws v2 upgrade feature branch into trunk
> --
>
> Key: HADOOP-18818
> URL: https://issues.apache.org/jira/browse/HADOOP-18818
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> do the merge, with everything we need as a blocker for that marked as a
> blocker of this task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18870) CURATOR-599 change broke functionality introduced in HADOOP-18139 and HADOOP-18709
[ https://issues.apache.org/jira/browse/HADOOP-18870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Erdelyi updated HADOOP-18870: Description: [Curator PR#391 |https://github.com/apache/curator/pull/391/files#diff-687a4ed1252bfb4f56b3aeeb28bee4413b7df9bec4b969b72215587158ac875dR59] introduced a default method in the ZooKeeperFactory interface, hence the override of the 4-parameter NewZookeeper method in the HadoopZookeeperFactory class is not taking effect due to this. Proposing routing the 4-parameter method to a 5-parameter method, which instantiates the ZKConfiguration as the 5th parameter. This is a non-breaking change, as the ZKConfiguration is currently instantiated within the method. was: [[Curator PR#391 |https://github.com/apache/curator/pull/391/files#diff-687a4ed1252bfb4f56b3aeeb28bee4413b7df9bec4b969b72215587158ac875dR59] introduced a default method in the ZooKeeperFactory interface, hence the override of the 4-parameter NewZookeeper method in the HadoopZookeeperFactory class is not taking effect due to this. Proposing routing the 4-parameter method to a 5-parameter method, which instantiates the ZKConfiguration as the 5th parameter. This is a non-breaking change, as the ZKConfiguration is currently instantiated within the method. > CURATOR-599 change broke functionality introduced in HADOOP-18139 and > HADOOP-18709 > -- > > Key: HADOOP-18870 > URL: https://issues.apache.org/jira/browse/HADOOP-18870 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.4.0, 3.3.5 >Reporter: Ferenc Erdelyi >Assignee: Ferenc Erdelyi >Priority: Major > > [Curator PR#391 > |https://github.com/apache/curator/pull/391/files#diff-687a4ed1252bfb4f56b3aeeb28bee4413b7df9bec4b969b72215587158ac875dR59] > introduced a default method in the ZooKeeperFactory interface, hence the > override of the 4-parameter NewZookeeper method in the HadoopZookeeperFactory > class is not taking effect due to this. > Proposing routing the 4-parameter method to a 5-parameter method, which > instantiates the ZKConfiguration as the 5th parameter. This is a non-breaking > change, as the ZKConfiguration is currently instantiated within the method. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5995: HADOOP-18818. Merge aws v2 upgrade feature branch into trunk
steveloughran commented on code in PR #5995:
URL: https://github.com/apache/hadoop/pull/5995#discussion_r1309006133
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java:
##
@@ -139,6 +144,9 @@ public S3ARetryPolicy(Configuration conf) {
retryIdempotentCalls = new FailNonIOEs(
new IdempotencyRetryFilter(baseExponentialRetry));
+//retryAwsClientExceptions = new
RetryFromAWSClientIOException(retryIdempotentCalls);
+retryAwsClientExceptions = retryIdempotentCalls;
Review Comment:
i will create a jira and reference it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18818) Merge aws v2 upgrade feature branch into trunk
[
https://issues.apache.org/jira/browse/HADOOP-18818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760001#comment-17760001
]
ASF GitHub Bot commented on HADOOP-18818:
-
steveloughran commented on code in PR #5995:
URL: https://github.com/apache/hadoop/pull/5995#discussion_r1309005801
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java:
##
@@ -224,9 +232,11 @@ protected Map, RetryPolicy>
createExceptionMap() {
// server didn't respond.
policyMap.put(AWSNoResponseException.class, retryIdempotentCalls);
+// translated AWS exceptions are retried if retryable() is true.
+policyMap.put(AWSClientIOException.class, retryAwsClientExceptions);
+policyMap.put(AWSServiceIOException.class, retryAwsClientExceptions);
Review Comment:
that's why I left this out. the thing is "retryable()" is meant to indicate
when things can't be retried, and we've been getting complaints about useless
retrying. but I've left it out of this change.
> Merge aws v2 upgrade feature branch into trunk
> --
>
> Key: HADOOP-18818
> URL: https://issues.apache.org/jira/browse/HADOOP-18818
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> do the merge, with everything we need as a blocker for that marked as a
> blocker of this task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5995: HADOOP-18818. Merge aws v2 upgrade feature branch into trunk
steveloughran commented on code in PR #5995:
URL: https://github.com/apache/hadoop/pull/5995#discussion_r1309005801
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java:
##
@@ -224,9 +232,11 @@ protected Map, RetryPolicy>
createExceptionMap() {
// server didn't respond.
policyMap.put(AWSNoResponseException.class, retryIdempotentCalls);
+// translated AWS exceptions are retried if retryable() is true.
+policyMap.put(AWSClientIOException.class, retryAwsClientExceptions);
+policyMap.put(AWSServiceIOException.class, retryAwsClientExceptions);
Review Comment:
that's why I left this out. the thing is "retryable()" is meant to indicate
when things can't be retried, and we've been getting complaints about useless
retrying. but I've left it out of this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18870) CURATOR-599 change broke functionality introduced in HADOOP-18139 and HADOOP-18709
Ferenc Erdelyi created HADOOP-18870: --- Summary: CURATOR-599 change broke functionality introduced in HADOOP-18139 and HADOOP-18709 Key: HADOOP-18870 URL: https://issues.apache.org/jira/browse/HADOOP-18870 Project: Hadoop Common Issue Type: Bug Components: common Affects Versions: 3.3.5, 3.4.0 Reporter: Ferenc Erdelyi [Curator PR#391 |https://github.com/apache/curator/pull/391]introduced a default method in the ZooKeeperFactory interface, hence the override of the 4-parameter NewZookeeper method in the HadoopZookeeperFactory class is not taking effect. Proposing routing the 4-parameter method to a 5-parameter method, which instantiates the ZKConfiguration as the 5th parameter. This is a non-breaking change, as the ZKConfiguration is currently instantiated within the method. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18870) CURATOR-599 change broke functionality introduced in HADOOP-18139 and HADOOP-18709
[ https://issues.apache.org/jira/browse/HADOOP-18870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Erdelyi reassigned HADOOP-18870: --- Assignee: Ferenc Erdelyi > CURATOR-599 change broke functionality introduced in HADOOP-18139 and > HADOOP-18709 > -- > > Key: HADOOP-18870 > URL: https://issues.apache.org/jira/browse/HADOOP-18870 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.4.0, 3.3.5 >Reporter: Ferenc Erdelyi >Assignee: Ferenc Erdelyi >Priority: Major > > [Curator PR#391 |https://github.com/apache/curator/pull/391]introduced a > default method in the ZooKeeperFactory interface, hence the override of the > 4-parameter NewZookeeper method in the HadoopZookeeperFactory > class is not taking effect. Proposing routing the 4-parameter method to a > 5-parameter method, which instantiates the ZKConfiguration as the 5th > parameter. This is a non-breaking change, as the ZKConfiguration is currently > instantiated within the method. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18818) Merge aws v2 upgrade feature branch into trunk
[
https://issues.apache.org/jira/browse/HADOOP-18818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759976#comment-17759976
]
ASF GitHub Bot commented on HADOOP-18818:
-
ahmarsuhail commented on code in PR #5995:
URL: https://github.com/apache/hadoop/pull/5995#discussion_r1308825152
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java:
##
@@ -139,6 +144,9 @@ public S3ARetryPolicy(Configuration conf) {
retryIdempotentCalls = new FailNonIOEs(
new IdempotencyRetryFilter(baseExponentialRetry));
+//retryAwsClientExceptions = new
RetryFromAWSClientIOException(retryIdempotentCalls);
+retryAwsClientExceptions = retryIdempotentCalls;
Review Comment:
nit: add a TODO here so it's clear why this commented out. confused me till
I read the comment on the commit
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java:
##
@@ -224,9 +232,11 @@ protected Map, RetryPolicy>
createExceptionMap() {
// server didn't respond.
policyMap.put(AWSNoResponseException.class, retryIdempotentCalls);
+// translated AWS exceptions are retried if retryable() is true.
+policyMap.put(AWSClientIOException.class, retryAwsClientExceptions);
+policyMap.put(AWSServiceIOException.class, retryAwsClientExceptions);
Review Comment:
when we do switch to the new retry policy, doesn't this mean
AWSServiceIOException will not be retried? I think previously these were
retried, but not sure
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/AWSCredentialProviderList.java:
##
@@ -330,4 +332,27 @@ public void close() {
public int size() {
return providers.size();
}
+
+
+ /**
+ * Translate an exception if it or its inner exception is an
+ * {@link CredentialInitializationException}.
+ * If this condition is not met, null is returned.
+ * @param path path of operation.
+ * @param throwable exception
+ * @return a translated exception or null.
+ */
+ public static IOException maybeTranslateCredentialException(String path,
Review Comment:
why do we need to do this? I get the AuditFailures are getting wrapped in a
SDKClient exception, what's wrapping CredentialInitializationException?
> Merge aws v2 upgrade feature branch into trunk
> --
>
> Key: HADOOP-18818
> URL: https://issues.apache.org/jira/browse/HADOOP-18818
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> do the merge, with everything we need as a blocker for that marked as a
> blocker of this task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ahmarsuhail commented on a diff in pull request #5995: HADOOP-18818. Merge aws v2 upgrade feature branch into trunk
ahmarsuhail commented on code in PR #5995:
URL: https://github.com/apache/hadoop/pull/5995#discussion_r1308825152
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java:
##
@@ -139,6 +144,9 @@ public S3ARetryPolicy(Configuration conf) {
retryIdempotentCalls = new FailNonIOEs(
new IdempotencyRetryFilter(baseExponentialRetry));
+//retryAwsClientExceptions = new
RetryFromAWSClientIOException(retryIdempotentCalls);
+retryAwsClientExceptions = retryIdempotentCalls;
Review Comment:
nit: add a TODO here so it's clear why this commented out. confused me till
I read the comment on the commit
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java:
##
@@ -224,9 +232,11 @@ protected Map, RetryPolicy>
createExceptionMap() {
// server didn't respond.
policyMap.put(AWSNoResponseException.class, retryIdempotentCalls);
+// translated AWS exceptions are retried if retryable() is true.
+policyMap.put(AWSClientIOException.class, retryAwsClientExceptions);
+policyMap.put(AWSServiceIOException.class, retryAwsClientExceptions);
Review Comment:
when we do switch to the new retry policy, doesn't this mean
AWSServiceIOException will not be retried? I think previously these were
retried, but not sure
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/AWSCredentialProviderList.java:
##
@@ -330,4 +332,27 @@ public void close() {
public int size() {
return providers.size();
}
+
+
+ /**
+ * Translate an exception if it or its inner exception is an
+ * {@link CredentialInitializationException}.
+ * If this condition is not met, null is returned.
+ * @param path path of operation.
+ * @param throwable exception
+ * @return a translated exception or null.
+ */
+ public static IOException maybeTranslateCredentialException(String path,
Review Comment:
why do we need to do this? I get the AuditFailures are getting wrapped in a
SDKClient exception, what's wrapping CredentialInitializationException?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5881: Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout.
hadoop-yetus commented on PR #5881:
URL: https://github.com/apache/hadoop/pull/5881#issuecomment-1697436592
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 27s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 41s | | trunk passed |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 53s | | trunk passed |
| -1 :x: | shadedclient | 21m 18s | | branch has errors when building
and testing our client artifacts. |
| -0 :warning: | patch | 21m 36s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 23s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 16s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/11/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 8 new + 6 unchanged - 0
fixed = 14 total (was 6) |
| +1 :green_heart: | mvnsite | 0m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 48s | | the patch passed |
| -1 :x: | shadedclient | 21m 32s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 57s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 90m 5s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/11/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5881 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 0c10f6ec2340 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7b9ec51c20316dff74aa61d875b3a47e6d80fc80 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/11/testReport/ |
| Max. process+thread count | 454 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/11/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0
[jira] [Commented] (HADOOP-17377) ABFS: MsiTokenProvider doesn't retry HTTP 429 from the Instance Metadata Service
[
https://issues.apache.org/jira/browse/HADOOP-17377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759960#comment-17759960
]
ASF GitHub Bot commented on HADOOP-17377:
-
hadoop-yetus commented on PR #5273:
URL: https://github.com/apache/hadoop/pull/5273#issuecomment-1697434060
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 35m 9s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/15/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 0m 28s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 53s | | trunk passed |
| -1 :x: | shadedclient | 21m 27s | | branch has errors when building
and testing our client artifacts. |
| -0 :warning: | patch | 21m 42s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 15s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/15/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 2 new + 0 unchanged - 0
fixed = 2 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| -1 :x: | javadoc | 0m 21s |
[/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/15/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt)
| hadoop-azure in the patch failed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 20s |
[/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/15/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-azure in the patch failed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| +1 :green_heart: | spotbugs | 0m 50s | | the patch passed |
| -1 :x: | shadedclient | 21m 21s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 53s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 90m 51s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/15/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5273 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient
[GitHub] [hadoop] hadoop-yetus commented on pull request #5273: HADOOP-17377: AABFS: MsiTokenProvider doesn't retry HTTP 429/410 from the Instance Metadata Service
hadoop-yetus commented on PR #5273:
URL: https://github.com/apache/hadoop/pull/5273#issuecomment-1697434060
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 35m 9s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/15/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 0m 28s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 53s | | trunk passed |
| -1 :x: | shadedclient | 21m 27s | | branch has errors when building
and testing our client artifacts. |
| -0 :warning: | patch | 21m 42s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 15s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/15/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 2 new + 0 unchanged - 0
fixed = 2 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| -1 :x: | javadoc | 0m 21s |
[/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/15/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt)
| hadoop-azure in the patch failed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 20s |
[/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/15/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-azure in the patch failed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| +1 :green_heart: | spotbugs | 0m 50s | | the patch passed |
| -1 :x: | shadedclient | 21m 21s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 53s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 90m 51s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/15/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5273 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 04bde9dbd212 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh
[GitHub] [hadoop] hfutatzhanghb commented on pull request #5996: HDFS-17171. CONGESTION_RATIO should be configurable
hfutatzhanghb commented on PR #5996: URL: https://github.com/apache/hadoop/pull/5996#issuecomment-1697384102 @Hexiaoqiao @ayushtkn @tomscut Sir, could you please help me review this minor modification when you have free time? Thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18818) Merge aws v2 upgrade feature branch into trunk
[ https://issues.apache.org/jira/browse/HADOOP-18818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759952#comment-17759952 ] ASF GitHub Bot commented on HADOOP-18818: - steveloughran commented on PR #5995: URL: https://github.com/apache/hadoop/pull/5995#issuecomment-1697377893 I really don't get the fact that the enforcer is rejecting the code, even though its ok locally. will have to comment out those lines for now. > Merge aws v2 upgrade feature branch into trunk > -- > > Key: HADOOP-18818 > URL: https://issues.apache.org/jira/browse/HADOOP-18818 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > do the merge, with everything we need as a blocker for that marked as a > blocker of this task. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5995: HADOOP-18818. Merge aws v2 upgrade feature branch into trunk
steveloughran commented on PR #5995: URL: https://github.com/apache/hadoop/pull/5995#issuecomment-1697377893 I really don't get the fact that the enforcer is rejecting the code, even though its ok locally. will have to comment out those lines for now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18818) Merge aws v2 upgrade feature branch into trunk
[ https://issues.apache.org/jira/browse/HADOOP-18818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759947#comment-17759947 ] ASF GitHub Bot commented on HADOOP-18818: - steveloughran commented on PR #5995: URL: https://github.com/apache/hadoop/pull/5995#issuecomment-1697359782 tested s3 london with: -Dparallel-tests -DtestsThreadCount=8 -Dscale -Dprefetch tests to 28 min; starting to get slow again with the prefetch parameterization. long term: get prefetching into the state where we can switch to it everywhere shorter term: look at those slow tests (huge uploads, terasorts) and see how we can cut back on them. > Merge aws v2 upgrade feature branch into trunk > -- > > Key: HADOOP-18818 > URL: https://issues.apache.org/jira/browse/HADOOP-18818 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > do the merge, with everything we need as a blocker for that marked as a > blocker of this task. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5995: HADOOP-18818. Merge aws v2 upgrade feature branch into trunk
steveloughran commented on PR #5995: URL: https://github.com/apache/hadoop/pull/5995#issuecomment-1697359782 tested s3 london with: -Dparallel-tests -DtestsThreadCount=8 -Dscale -Dprefetch tests to 28 min; starting to get slow again with the prefetch parameterization. long term: get prefetching into the state where we can switch to it everywhere shorter term: look at those slow tests (huge uploads, terasorts) and see how we can cut back on them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5996: HDFS-17171. CONGESTION_RATIO should be configurable
hadoop-yetus commented on PR #5996:
URL: https://github.com/apache/hadoop/pull/5996#issuecomment-1697325166
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 42s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 49s | | trunk passed |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 19s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 38s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 26s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 8s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 30s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 221m 16s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5996/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 53s | | The patch does not
generate ASF License warnings. |
| | | 379m 27s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5996/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5996 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e03541761b5e 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 38ab1b95ddc5c458755f321267e97a23d09e15bc |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5996/2/testReport/ |
| Max. process+thread count | 3352 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5996/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0
[GitHub] [hadoop] hadoop-yetus commented on pull request #5881: Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout.
hadoop-yetus commented on PR #5881:
URL: https://github.com/apache/hadoop/pull/5881#issuecomment-1697311358
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 5s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 22s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 5s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 0s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 39m 21s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/10/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 8 new + 6 unchanged - 0
fixed = 14 total (was 6) |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| -1 :x: | javadoc | 0m 25s |
[/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/10/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt)
| hadoop-azure in the patch failed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 24s |
[/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/10/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-azure in the patch failed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 11s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 9s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 143m 15s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5881 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 83f0ef370da0 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3ef2efd7c0a36411f6d147eab213994d0a80a501 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
[jira] [Commented] (HADOOP-18867) Upgrade ZooKeeper to 3.6.4
[
https://issues.apache.org/jira/browse/HADOOP-18867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759940#comment-17759940
]
ASF GitHub Bot commented on HADOOP-18867:
-
iwasakims commented on PR #5988:
URL: https://github.com/apache/hadoop/pull/5988#issuecomment-1697309071
Hmm, I got javadoc error only on aarch64. I'm looking into the cause.
```
$ uname -a
Linux ip-172-31-141-109 5.19.0-1025-aws #26~22.04.1-Ubuntu SMP Mon Apr 24
01:58:03 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
$ mvn clean install -DskipTests -DskipShade -Pdocs
...
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-javadoc-plugin:3.0.1:javadoc-no-fork (default)
on project hadoop-common: An error has occurred in Javadoc report generation:
[ERROR] Exit code: 1 - javadoc: error - class file for
org.apache.yetus.audience.InterfaceAudience not found
[ERROR]
[ERROR] Command line was:
/usr/lib/jvm/java-8-openjdk-arm64/jre/../bin/javadoc -J-Xmx768m @options
@packages
[ERROR]
[ERROR] Refer to the generated Javadoc files in
'/home/ubuntu/srcs/hadoop/hadoop-common-project/hadoop-common/target/site/home/ubuntu/srcs/hadoop/hadoop-common-project/hadoop-common/target/api'
dir.
```
> Upgrade ZooKeeper to 3.6.4
> --
>
> Key: HADOOP-18867
> URL: https://issues.apache.org/jira/browse/HADOOP-18867
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.3.6
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Minor
> Labels: pull-request-available
>
> While ZooKeeper 3.6 is already EOL, we can upgrade to the final release of
> the ZooKeeper 3.6 as short-term fix until bumping to ZooKeeper 3.7 or later.
> Dependency convergence error must be addressed on {{-Dhbase.profile=2.0}}.
> {noformat}
> $ mvn clean install -Dzookeeper.version=3.6.4 -Dhbase.profile=2.0 -DskipTests
> clean install
> Dependency convergence error for
> org.apache.yetus:audience-annotations:jar:0.13.0:compile paths to dependency
> are:
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-common:jar:3.4.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-common:test-jar:tests:3.4.0-SNAPSHOT:test
> +-org.apache.zookeeper:zookeeper:jar:3.6.4:compile
> +-org.apache.zookeeper:zookeeper-jute:jar:3.6.4:compile
> +-org.apache.yetus:audience-annotations:jar:0.13.0:compile
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-common:jar:3.4.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-common:test-jar:tests:3.4.0-SNAPSHOT:test
> +-org.apache.zookeeper:zookeeper:jar:3.6.4:compile
> +-org.apache.yetus:audience-annotations:jar:0.13.0:compile
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-common:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-common:jar:2.2.4:compile
> +-org.apache.yetus:audience-annotations:jar:0.5.0:compile
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on pull request #5988: HADOOP-18867. Upgrade ZooKeeper to 3.6.4.
iwasakims commented on PR #5988: URL: https://github.com/apache/hadoop/pull/5988#issuecomment-1697309071 Hmm, I got javadoc error only on aarch64. I'm looking into the cause. ``` $ uname -a Linux ip-172-31-141-109 5.19.0-1025-aws #26~22.04.1-Ubuntu SMP Mon Apr 24 01:58:03 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux $ mvn clean install -DskipTests -DskipShade -Pdocs ... [ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:3.0.1:javadoc-no-fork (default) on project hadoop-common: An error has occurred in Javadoc report generation: [ERROR] Exit code: 1 - javadoc: error - class file for org.apache.yetus.audience.InterfaceAudience not found [ERROR] [ERROR] Command line was: /usr/lib/jvm/java-8-openjdk-arm64/jre/../bin/javadoc -J-Xmx768m @options @packages [ERROR] [ERROR] Refer to the generated Javadoc files in '/home/ubuntu/srcs/hadoop/hadoop-common-project/hadoop-common/target/site/home/ubuntu/srcs/hadoop/hadoop-common-project/hadoop-common/target/api' dir. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anmolanmol1234 opened a new pull request, #5999: HADOOP-18860. Upgrade mockito version to 4.11.0
anmolanmol1234 opened a new pull request, #5999: URL: https://github.com/apache/hadoop/pull/5999 [HADOOP-18860](https://issues.apache.org/jira/browse/HADOOP-18860): Upgrade mockito version to 4.11.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18860) Upgrade mockito to 4.11.0
[ https://issues.apache.org/jira/browse/HADOOP-18860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759938#comment-17759938 ] ASF GitHub Bot commented on HADOOP-18860: - anmolanmol1234 opened a new pull request, #5999: URL: https://github.com/apache/hadoop/pull/5999 [HADOOP-18860](https://issues.apache.org/jira/browse/HADOOP-18860): Upgrade mockito version to 4.11.0 > Upgrade mockito to 4.11.0 > - > > Key: HADOOP-18860 > URL: https://issues.apache.org/jira/browse/HADOOP-18860 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.3.6 >Reporter: Anmol Asrani >Assignee: Anmol Asrani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Upgrading mockito in hadoop-project -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18869) ABFS: Fixing Behavior of a File System APIs on root path
Anuj Modi created HADOOP-18869:
--
Summary: ABFS: Fixing Behavior of a File System APIs on root path
Key: HADOOP-18869
URL: https://issues.apache.org/jira/browse/HADOOP-18869
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/azure
Affects Versions: 3.3.6
Reporter: Anuj Modi
Assignee: Anuj Modi
Following HDFS Apis are failing when called on a root path.
{*}{*}{*}{*}{*}{*}
|FS Call|Status|Error thrown to caller|
|create()|Failing|Operation failed: "The request URI is invalid.", 400, PUT,
https://anujtesthns.dfs.core.windows.net/abfs-testcontainer-02076119-21ed-4ada-bcd0-14afaae54013/?resource=file=90,
InvalidUri, "The request URI is invalid.
RequestId:1d23f8c2-d01f-0059-61b6-c60c2400
Time:2023-08-04T09:29:55.4813818Z"|
|createNonRecursive()|Failing|Runtime Exception:
java.lang.IllegalArgumentException: null path (This is occuring because
getParentPath is null and getFileStatus is called on null)|
|setXAttr()|Failing|Operation failed: "The request URI is invalid.", 400, HEAD,
https://anujtesthns.dfs.core.windows.net/abfs-testcontainer-491399b3-c3d0-4568-9d4a-a26e0aa8f000/?upn=false=90|
|getXAttr()|Failing|Operation failed: "The request URI is invalid.", 400, HEAD,
https://anujtesthns.dfs.core.windows.net/abfs-testcontainer-491399b3-c3d0-4568-9d4a-a26e0aa8f000/?upn=false=91|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5978: YARN-11552. Timeline endpoint: /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store
hadoop-yetus commented on PR #5978:
URL: https://github.com/apache/hadoop/pull/5978#issuecomment-1697282539
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 27s | | trunk passed |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 23s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 16s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5978/6/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice:
The patch generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 52s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 48s | |
hadoop-yarn-server-timelineservice in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 135m 15s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5978/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5978 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 195c50293fe9 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 15fb90cde11185e409abd44e2a0c7d031c9f4796 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5978/6/testReport/ |
| Max. process+thread count | 723 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
|
| Console output |
[GitHub] [hadoop] hadoop-yetus commented on pull request #5978: YARN-11552. Timeline endpoint: /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store
hadoop-yetus commented on PR #5978:
URL: https://github.com/apache/hadoop/pull/5978#issuecomment-1697276800
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 17s | | trunk passed |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 23s | | the patch passed |
| +1 :green_heart: | compile | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 23s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 15s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5978/5/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice:
The patch generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 50s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 47s | |
hadoop-yarn-server-timelineservice in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 131m 6s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5978/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5978 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 94d1e19cc075 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 15fb90cde11185e409abd44e2a0c7d031c9f4796 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5978/5/testReport/ |
| Max. process+thread count | 555 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
|
| Console output |
[jira] [Commented] (HADOOP-18818) Merge aws v2 upgrade feature branch into trunk
[
https://issues.apache.org/jira/browse/HADOOP-18818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759923#comment-17759923
]
ASF GitHub Bot commented on HADOOP-18818:
-
hadoop-yetus commented on PR #5995:
URL: https://github.com/apache/hadoop/pull/5995#issuecomment-1697261975
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 13m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 5s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 83 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 20s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 57s | | trunk passed |
| +1 :green_heart: | compile | 17m 24s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 16m 13s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 19m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 8m 48s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 34s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +0 :ok: | spotbugs | 0m 24s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 62m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 2s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 17s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5995/1/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | mvninstall | 29m 22s |
[/patch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5995/1/artifact/out/patch-mvninstall-root.txt)
| root in the patch failed. |
| -1 :x: | compile | 16m 10s |
[/patch-compile-root-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5995/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javac | 16m 10s |
[/patch-compile-root-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5995/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | compile | 15m 49s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5995/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| root in the patch failed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| -1 :x: | javac | 15m 49s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5995/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| root in the patch failed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 23s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5995/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 7 new + 72 unchanged - 13 fixed = 79 total (was
85) |
| -1 :x: | mvnsite | 5m 7s |
[/patch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5995/1/artifact/out/patch-mvnsite-root.txt)
| root in the patch failed. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| -1 :x: | javadoc | 8m 14s |
[GitHub] [hadoop] hadoop-yetus commented on pull request #5995: HADOOP-18818. Merge aws v2 upgrade feature branch into trunk
hadoop-yetus commented on PR #5995:
URL: https://github.com/apache/hadoop/pull/5995#issuecomment-1697261975
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 13m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 5s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 83 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 20s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 57s | | trunk passed |
| +1 :green_heart: | compile | 17m 24s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 16m 13s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 19m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 8m 48s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 34s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +0 :ok: | spotbugs | 0m 24s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 62m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 2s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 17s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5995/1/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | mvninstall | 29m 22s |
[/patch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5995/1/artifact/out/patch-mvninstall-root.txt)
| root in the patch failed. |
| -1 :x: | compile | 16m 10s |
[/patch-compile-root-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5995/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javac | 16m 10s |
[/patch-compile-root-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5995/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | compile | 15m 49s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5995/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| root in the patch failed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| -1 :x: | javac | 15m 49s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5995/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| root in the patch failed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 23s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5995/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 7 new + 72 unchanged - 13 fixed = 79 total (was
85) |
| -1 :x: | mvnsite | 5m 7s |
[/patch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5995/1/artifact/out/patch-mvnsite-root.txt)
| root in the patch failed. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| -1 :x: | javadoc | 8m 14s |
[/patch-javadoc-root-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5995/1/artifact/out/patch-javadoc-root-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt)
| root in the patch
[jira] [Commented] (HADOOP-18797) S3A committer fix lost data on concurrent jobs
[
https://issues.apache.org/jira/browse/HADOOP-18797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759922#comment-17759922
]
Syed Shameerur Rahman commented on HADOOP-18797:
yes, i noticed, MagicCommitPaths#isMagicPath and
MagicCommitPaths#magicElementIndex needs to changed as well.
> S3A committer fix lost data on concurrent jobs
> --
>
> Key: HADOOP-18797
> URL: https://issues.apache.org/jira/browse/HADOOP-18797
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.3.6
>Reporter: Emanuel Velzi
>Priority: Major
>
> There is a failure in the commit process when multiple jobs are writing to a
> s3 directory *concurrently* using {*}magic committers{*}.
> This issue is closely related HADOOP-17318.
> When multiple Spark jobs write to the same S3A directory, they upload files
> simultaneously using "__magic" as the base directory for staging. Inside this
> directory, there are multiple "/job-some-uuid" directories, each representing
> a concurrently running job.
> To fix some preoblems related to concunrrency a property was introduced in
> the previous fix: "spark.hadoop.fs.s3a.committer.abort.pending.uploads". When
> set to false, it ensures that during the cleanup stage, finalizing jobs do
> not abort pending uploads from other jobs. So we see in logs this line:
> {code:java}
> DEBUG [main] o.a.h.fs.s3a.commit.AbstractS3ACommitter (819): Not cleanup up
> pending uploads to s3a ...{code}
> (from
> [AbstractS3ACommitter.java#L952|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java#L952])
> However, in the next step, the {*}"__magic" directory is recursively
> deleted{*}:
> {code:java}
> INFO [main] o.a.h.fs.s3a.commit.magic.MagicS3GuardCommitter (98): Deleting
> magic directory s3a://my-bucket/my-table/__magic: duration 0:00.560s {code}
> (from [AbstractS3ACommitter.java#L1112
> |https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java#L1112]and
>
> [MagicS3GuardCommitter.java#L137)|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java#L137)]
> This deletion operation *affects the second job* that is still running
> because it loses pending uploads (i.e., ".pendingset" and ".pending" files).
> The consequences can range from an exception in the best case to a silent
> loss of data in the worst case. The latter occurs when Job_1 deletes files
> just before Job_2 executes "listPendingUploadsToCommit" to list ".pendingset"
> files in the job attempt directory previous to complete the uploads with POST
> requests.
> To resolve this issue, it's important {*}to ensure that only the prefix
> associated with the job currently finalizing is cleaned{*}.
> Here's a possible solution:
> {code:java}
> /**
> * Delete the magic directory.
> */
> public void cleanupStagingDirs() {
> final Path out = getOutputPath();
> //Path path = magicSubdir(getOutputPath());
> Path path = new Path(magicSubdir(out), formatJobDir(getUUID()));
> try(DurationInfo ignored = new DurationInfo(LOG, true,
> "Deleting magic directory %s", path)) {
> Invoker.ignoreIOExceptions(LOG, "cleanup magic directory",
> path.toString(),
> () -> deleteWithWarning(getDestFS(), path, true));
> }
> } {code}
>
> The side effect of this issue is that the "__magic" directory is never
> cleaned up. However, I believe this is a minor concern, even considering that
> other folders such as "_SUCCESS" also persist after jobs end.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18818) Merge aws v2 upgrade feature branch into trunk
[ https://issues.apache.org/jira/browse/HADOOP-18818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18818: Labels: pull-request-available (was: ) > Merge aws v2 upgrade feature branch into trunk > -- > > Key: HADOOP-18818 > URL: https://issues.apache.org/jira/browse/HADOOP-18818 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > do the merge, with everything we need as a blocker for that marked as a > blocker of this task. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18818) Merge aws v2 upgrade feature branch into trunk
[ https://issues.apache.org/jira/browse/HADOOP-18818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759920#comment-17759920 ] ASF GitHub Bot commented on HADOOP-18818: - steveloughran commented on PR #5995: URL: https://github.com/apache/hadoop/pull/5995#issuecomment-1697245491 well, this is taking a while! decided that we should rename getAwsV2ClientFortesting and just cut the ForTesting bit out, as there may be other needs and the whole "internals" concept makes that clear you are going into the depths > Merge aws v2 upgrade feature branch into trunk > -- > > Key: HADOOP-18818 > URL: https://issues.apache.org/jira/browse/HADOOP-18818 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > do the merge, with everything we need as a blocker for that marked as a > blocker of this task. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5995: HADOOP-18818. Merge aws v2 upgrade feature branch into trunk
steveloughran commented on PR #5995: URL: https://github.com/apache/hadoop/pull/5995#issuecomment-1697245491 well, this is taking a while! decided that we should rename getAwsV2ClientFortesting and just cut the ForTesting bit out, as there may be other needs and the whole "internals" concept makes that clear you are going into the depths -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18860) Upgrade mockito to 4.11.0
[ https://issues.apache.org/jira/browse/HADOOP-18860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759919#comment-17759919 ] Steve Loughran commented on HADOOP-18860: - merged to trunk, will backport. apologies to everyone who is going to have problems cherrypicking mockito changes back to older branches, but that goes with every mockito update > Upgrade mockito to 4.11.0 > - > > Key: HADOOP-18860 > URL: https://issues.apache.org/jira/browse/HADOOP-18860 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.3.6 >Reporter: Anmol Asrani >Assignee: Anmol Asrani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Upgrading mockito in hadoop-project -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18860) Upgrade mockito to 4.11.0
[ https://issues.apache.org/jira/browse/HADOOP-18860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18860: Component/s: test > Upgrade mockito to 4.11.0 > - > > Key: HADOOP-18860 > URL: https://issues.apache.org/jira/browse/HADOOP-18860 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.3.6 >Reporter: Anmol Asrani >Assignee: Anmol Asrani >Priority: Major > Labels: pull-request-available > > Upgrading mockito in hadoop-project -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18860) Upgrade mockito to 4.11.0
[ https://issues.apache.org/jira/browse/HADOOP-18860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18860: Fix Version/s: 3.4.0 > Upgrade mockito to 4.11.0 > - > > Key: HADOOP-18860 > URL: https://issues.apache.org/jira/browse/HADOOP-18860 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.3.6 >Reporter: Anmol Asrani >Assignee: Anmol Asrani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Upgrading mockito in hadoop-project -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18860) Upgrade mockito to 4.11.0
[ https://issues.apache.org/jira/browse/HADOOP-18860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759918#comment-17759918 ] ASF GitHub Bot commented on HADOOP-18860: - steveloughran commented on PR #5977: URL: https://github.com/apache/hadoop/pull/5977#issuecomment-1697241950 ok, merged @anmolanmol1234 1. can you do a pr of this cherrypicked to branch-3.3, as we will need it there before we can get your real patch in. 2. and submit the abfs patch on top of trunk now > Upgrade mockito to 4.11.0 > - > > Key: HADOOP-18860 > URL: https://issues.apache.org/jira/browse/HADOOP-18860 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.3.6 >Reporter: Anmol Asrani >Assignee: Anmol Asrani >Priority: Major > Labels: pull-request-available > > Upgrading mockito in hadoop-project -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5977: HADOOP-18860. Upgrade mockito version to 4.11.0
steveloughran commented on PR #5977: URL: https://github.com/apache/hadoop/pull/5977#issuecomment-1697241950 ok, merged @anmolanmol1234 1. can you do a pr of this cherrypicked to branch-3.3, as we will need it there before we can get your real patch in. 2. and submit the abfs patch on top of trunk now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18860) Upgrade mockito to 4.11.0
[ https://issues.apache.org/jira/browse/HADOOP-18860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759915#comment-17759915 ] ASF GitHub Bot commented on HADOOP-18860: - steveloughran merged PR #5977: URL: https://github.com/apache/hadoop/pull/5977 > Upgrade mockito to 4.11.0 > - > > Key: HADOOP-18860 > URL: https://issues.apache.org/jira/browse/HADOOP-18860 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.3.6 >Reporter: Anmol Asrani >Assignee: Anmol Asrani >Priority: Major > Labels: pull-request-available > > Upgrading mockito in hadoop-project -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5881: Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout.
hadoop-yetus commented on PR #5881:
URL: https://github.com/apache/hadoop/pull/5881#issuecomment-1697240195
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 6s | | trunk passed |
| +1 :green_heart: | compile | 0m 26s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 26s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 53s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 11s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 26s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 21s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 12s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/9/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 8 new + 6 unchanged - 0
fixed = 14 total (was 6) |
| +1 :green_heart: | mvnsite | 0m 21s | | the patch passed |
| -1 :x: | javadoc | 0m 20s |
[/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/9/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt)
| hadoop-azure in the patch failed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 19s |
[/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/9/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-azure in the patch failed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| +1 :green_heart: | spotbugs | 0m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 57s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 96m 13s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5881 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 737b3a22ae28 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3ef2efd7c0a36411f6d147eab213994d0a80a501 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
[GitHub] [hadoop] steveloughran merged pull request #5977: HADOOP-18860. Upgrade mockito version to 4.11.0
steveloughran merged PR #5977: URL: https://github.com/apache/hadoop/pull/5977 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5881: Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout.
hadoop-yetus commented on PR #5881:
URL: https://github.com/apache/hadoop/pull/5881#issuecomment-1697221833
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 15s | | trunk passed |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 13s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 30s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 21s | | the patch passed |
| +1 :green_heart: | compile | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 13s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/8/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 8 new + 6 unchanged - 0
fixed = 14 total (was 6) |
| +1 :green_heart: | mvnsite | 0m 21s | | the patch passed |
| -1 :x: | javadoc | 0m 20s |
[/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/8/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt)
| hadoop-azure in the patch failed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 19s |
[/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/8/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-azure in the patch failed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| +1 :green_heart: | spotbugs | 0m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 54s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 97m 44s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5881 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 977930e873b7 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bb013d512e378e37b8c62cbf24d1c9cc3cbb167c |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
[jira] [Commented] (HADOOP-18797) S3A committer fix lost data on concurrent jobs
[
https://issues.apache.org/jira/browse/HADOOP-18797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759904#comment-17759904
]
Steve Loughran commented on HADOOP-18797:
-
oh, except one complication. currently the "__magic" path is the one which
indicates "all writes of files don't get manifest here but have destination
files created relative to the parent using the dirs under __base to indicate
where the base dir is"
this'd need to be expanded to support more prefixes, e.g __magic_job_${jobid}
one concern i had when doing the committer was "what if anyone used __magic in
their code?". nobody has ever complained, but its why we added an off switch. I
think if someone tried to do a recursive dir copy *or distcp from hdfs to s3*
bad things would happen, which is why there's that switch to turn it off if
ever someone complained. to date: nobody
> S3A committer fix lost data on concurrent jobs
> --
>
> Key: HADOOP-18797
> URL: https://issues.apache.org/jira/browse/HADOOP-18797
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.3.6
>Reporter: Emanuel Velzi
>Priority: Major
>
> There is a failure in the commit process when multiple jobs are writing to a
> s3 directory *concurrently* using {*}magic committers{*}.
> This issue is closely related HADOOP-17318.
> When multiple Spark jobs write to the same S3A directory, they upload files
> simultaneously using "__magic" as the base directory for staging. Inside this
> directory, there are multiple "/job-some-uuid" directories, each representing
> a concurrently running job.
> To fix some preoblems related to concunrrency a property was introduced in
> the previous fix: "spark.hadoop.fs.s3a.committer.abort.pending.uploads". When
> set to false, it ensures that during the cleanup stage, finalizing jobs do
> not abort pending uploads from other jobs. So we see in logs this line:
> {code:java}
> DEBUG [main] o.a.h.fs.s3a.commit.AbstractS3ACommitter (819): Not cleanup up
> pending uploads to s3a ...{code}
> (from
> [AbstractS3ACommitter.java#L952|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java#L952])
> However, in the next step, the {*}"__magic" directory is recursively
> deleted{*}:
> {code:java}
> INFO [main] o.a.h.fs.s3a.commit.magic.MagicS3GuardCommitter (98): Deleting
> magic directory s3a://my-bucket/my-table/__magic: duration 0:00.560s {code}
> (from [AbstractS3ACommitter.java#L1112
> |https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java#L1112]and
>
> [MagicS3GuardCommitter.java#L137)|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java#L137)]
> This deletion operation *affects the second job* that is still running
> because it loses pending uploads (i.e., ".pendingset" and ".pending" files).
> The consequences can range from an exception in the best case to a silent
> loss of data in the worst case. The latter occurs when Job_1 deletes files
> just before Job_2 executes "listPendingUploadsToCommit" to list ".pendingset"
> files in the job attempt directory previous to complete the uploads with POST
> requests.
> To resolve this issue, it's important {*}to ensure that only the prefix
> associated with the job currently finalizing is cleaned{*}.
> Here's a possible solution:
> {code:java}
> /**
> * Delete the magic directory.
> */
> public void cleanupStagingDirs() {
> final Path out = getOutputPath();
> //Path path = magicSubdir(getOutputPath());
> Path path = new Path(magicSubdir(out), formatJobDir(getUUID()));
> try(DurationInfo ignored = new DurationInfo(LOG, true,
> "Deleting magic directory %s", path)) {
> Invoker.ignoreIOExceptions(LOG, "cleanup magic directory",
> path.toString(),
> () -> deleteWithWarning(getDestFS(), path, true));
> }
> } {code}
>
> The side effect of this issue is that the "__magic" directory is never
> cleaned up. However, I believe this is a minor concern, even considering that
> other folders such as "_SUCCESS" also persist after jobs end.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18679) Add API for bulk/paged object deletion
[ https://issues.apache.org/jira/browse/HADOOP-18679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759902#comment-17759902 ] ASF GitHub Bot commented on HADOOP-18679: - steveloughran commented on PR #5993: URL: https://github.com/apache/hadoop/pull/5993#issuecomment-1697184517 writing up spec made me decide we should have a .opt to indicate when a bulk delete is a "background" operation, which may be executed at a rate to interfere less with live queries, e.g: smaller pages, rate limited buildup of pages, different throttle retry policy. > Add API for bulk/paged object deletion > -- > > Key: HADOOP-18679 > URL: https://issues.apache.org/jira/browse/HADOOP-18679 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Priority: Major > Labels: pull-request-available > > iceberg and hbase could benefit from being able to give a list of individual > files to delete -files which may be scattered round the bucket for better > read peformance. > Add some new optional interface for an object store which allows a caller to > submit a list of paths to files to delete, where > the expectation is > * if a path is a file: delete > * if a path is a dir, outcome undefined > For s3 that'd let us build these into DeleteRequest objects, and submit, > without any probes first. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5993: HADOOP-18679. Add API for bulk/paged object deletion
steveloughran commented on PR #5993: URL: https://github.com/apache/hadoop/pull/5993#issuecomment-1697184517 writing up spec made me decide we should have a .opt to indicate when a bulk delete is a "background" operation, which may be executed at a rate to interfere less with live queries, e.g: smaller pages, rate limited buildup of pages, different throttle retry policy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5881: Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout.
hadoop-yetus commented on PR #5881:
URL: https://github.com/apache/hadoop/pull/5881#issuecomment-1697171214
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 35m 45s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/7/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 0m 29s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 39s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 23m 54s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 23s | | the patch passed |
| +1 :green_heart: | compile | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 16s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/7/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 8 new + 6 unchanged - 0
fixed = 14 total (was 6) |
| +1 :green_heart: | mvnsite | 0m 24s | | the patch passed |
| -1 :x: | javadoc | 0m 21s |
[/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/7/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt)
| hadoop-azure in the patch failed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 19s |
[/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/7/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-azure in the patch failed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| +1 :green_heart: | spotbugs | 0m 45s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 52s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 97m 26s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5881 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 9a2d7070d691 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk /
[jira] [Commented] (HADOOP-18868) Provide fallbacks for callqueue overflow trigger failover enable
[ https://issues.apache.org/jira/browse/HADOOP-18868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759901#comment-17759901 ] ASF GitHub Bot commented on HADOOP-18868: - haiyang1987 opened a new pull request, #5998: URL: https://github.com/apache/hadoop/pull/5998 ### Description of PR In [HADOOP-16268](https://issues.apache.org/jira/browse/HADOOP-16268) implemented enable callqueue overflow trigger failover for stateless servers when RPC call queue is filled. - here will to provide default properties for callqueue.overflow.trigger.failover such that if properties with port is not configured, we can fallback to default property (port-less). - will support reconfigure callqueue.overflow.trigger.failover enable for Namenode > Provide fallbacks for callqueue overflow trigger failover enable > > > Key: HADOOP-18868 > URL: https://issues.apache.org/jira/browse/HADOOP-18868 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > > In HADOOP-16268 implemented enable callqueue overflow trigger failover for > stateless servers when RPC call queue is filled, > here will to provide default properties for > callqueue.overflow.trigger.failover such that if properties with port is not > configured, we can fallback to default property (port-less). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18868) Provide fallbacks for callqueue overflow trigger failover enable
[ https://issues.apache.org/jira/browse/HADOOP-18868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18868: Labels: pull-request-available (was: ) > Provide fallbacks for callqueue overflow trigger failover enable > > > Key: HADOOP-18868 > URL: https://issues.apache.org/jira/browse/HADOOP-18868 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > > In HADOOP-16268 implemented enable callqueue overflow trigger failover for > stateless servers when RPC call queue is filled, > here will to provide default properties for > callqueue.overflow.trigger.failover such that if properties with port is not > configured, we can fallback to default property (port-less). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] haiyang1987 opened a new pull request, #5998: HADOOP-18868. Provide fallbacks for callqueue overflow trigger failover enable
haiyang1987 opened a new pull request, #5998: URL: https://github.com/apache/hadoop/pull/5998 ### Description of PR In [HADOOP-16268](https://issues.apache.org/jira/browse/HADOOP-16268) implemented enable callqueue overflow trigger failover for stateless servers when RPC call queue is filled. - here will to provide default properties for callqueue.overflow.trigger.failover such that if properties with port is not configured, we can fallback to default property (port-less). - will support reconfigure callqueue.overflow.trigger.failover enable for Namenode -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5994: HDFS-17170. add metris for datanode in function processQueueMessages and reportTo
hadoop-yetus commented on PR #5994:
URL: https://github.com/apache/hadoop/pull/5994#issuecomment-1697129868
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 38s | | trunk passed |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 12s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 1s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 0s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5994/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 130 unchanged
- 0 fixed = 131 total (was 130) |
| +1 :green_heart: | mvnsite | 1m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 33s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 254m 18s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 396m 57s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5994/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5994 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 133725e6ad63 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c35928b8b5a0ba63d728f61afc7945ee1b3db9af |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5994/2/testReport/ |
| Max. process+thread count | 2984 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5994/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from
[GitHub] [hadoop] hadoop-yetus commented on pull request #5996: HDFS-17171. CONGESTION_RATIO should be configurable
hadoop-yetus commented on PR #5996:
URL: https://github.com/apache/hadoop/pull/5996#issuecomment-1697118501
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 28s | | trunk passed |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 35s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 20s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 7s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 1m 5s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 5s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 1s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5996/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 267 unchanged
- 0 fixed = 268 total (was 267) |
| +1 :green_heart: | mvnsite | 1m 11s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 32s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 223m 2s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5996/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not
generate ASF License warnings. |
| | | 364m 5s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5996/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5996 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3f1d6552da3c 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fde604d3f7cc3a9375a84058d815ccea6ec99847 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5996/1/testReport/ |
| Max. process+thread count | 3174 (vs. ulimit of 5500) |
[GitHub] [hadoop] hadoop-yetus commented on pull request #5967: YARN-11435. [Router] FederationStateStoreFacade is not reinitialized with Router conf.
hadoop-yetus commented on PR #5967:
URL: https://github.com/apache/hadoop/pull/5967#issuecomment-1697085255
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 25 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 10s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 12s | | trunk passed |
| +1 :green_heart: | compile | 7m 52s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 7m 12s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 2m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 24s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 6s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 7m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 36s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 44s | | the patch passed |
| +1 :green_heart: | compile | 7m 2s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 7m 2s | | the patch passed |
| +1 :green_heart: | compile | 7m 5s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 7m 5s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 16s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 7s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 8m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 50s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 48s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 1m 33s | | hadoop-yarn-server-web-proxy in
the patch passed. |
| +1 :green_heart: | unit | 24m 56s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| -1 :x: | unit | 28m 50s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5967/5/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt)
| hadoop-yarn-client in the patch passed. |
| +1 :green_heart: | unit | 0m 53s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | unit | 1m 7s | |
hadoop-yarn-server-globalpolicygenerator in the patch passed. |
| +1 :green_heart: | asflicense | 1m 6s | | The patch does not
generate ASF License warnings. |
| | | 267m 59s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.yarn.client.TestRMFailover |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5967/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5967 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e9edacd085e3 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4a79ca30114976003ce5a63c7054ea2c73731fbf |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
[GitHub] [hadoop] anujmodi2021 commented on a diff in pull request #5881: Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout.
anujmodi2021 commented on code in PR #5881:
URL: https://github.com/apache/hadoop/pull/5881#discussion_r1308427326
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/StaticRetryPolicy.java:
##
@@ -0,0 +1,68 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+
+/**
+ * Retry policy used by AbfsClient for Network Errors.
+ * */
+public class StaticRetryPolicy extends AbfsRetryPolicy {
+
+ private static final int STATIC_RETRY_INTERVAL_DEFAULT = 2000; // 2s
+
+ /**
+ * Represents the constant retry interval to be used with Static Retry Policy
+ */
+ private int retryInterval;
Review Comment:
We need to initialize the value of static retry interval in constructor
based on the file system configurations provided...
I will try to find another way of having this as a final variable. Please
suggest me if you have something in mind...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anujmodi2021 commented on a diff in pull request #5881: Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout.
anujmodi2021 commented on code in PR #5881:
URL: https://github.com/apache/hadoop/pull/5881#discussion_r1308427326
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/StaticRetryPolicy.java:
##
@@ -0,0 +1,68 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+
+/**
+ * Retry policy used by AbfsClient for Network Errors.
+ * */
+public class StaticRetryPolicy extends AbfsRetryPolicy {
+
+ private static final int STATIC_RETRY_INTERVAL_DEFAULT = 2000; // 2s
+
+ /**
+ * Represents the constant retry interval to be used with Static Retry Policy
+ */
+ private int retryInterval;
Review Comment:
We need to initialize the value of static retry interval in constructor
based on the file system configurations provided...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anujmodi2021 commented on a diff in pull request #5881: Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout.
anujmodi2021 commented on code in PR #5881:
URL: https://github.com/apache/hadoop/pull/5881#discussion_r1308420397
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java:
##
@@ -224,15 +226,18 @@ private void completeExecute(TracingContext
tracingContext)
requestHeaders.add(httpHeader);
}
+// By Default Exponential Retry Policy Will be used
retryCount = 0;
+retryPolicy = client.getExponentialRetryPolicy();
LOG.debug("First execution of REST operation - {}", operationType);
while (!executeHttpOperation(retryCount, tracingContext)) {
try {
++retryCount;
tracingContext.setRetryCount(retryCount);
-LOG.debug("Retrying REST operation {}. RetryCount = {}",
-operationType, retryCount);
-Thread.sleep(client.getRetryPolicy().getRetryInterval(retryCount));
+long retryInterval = retryPolicy.getRetryInterval(retryCount);
+LOG.debug("Rest operation {} failed with failureReason: {}. Retrying
with retryCount = {}, retryPolicy: {} and sleepInterval: {}",
Review Comment:
We already have URL logged inside executeHttpOperation() function. In case
of retriable failures, URL will be logged just before this log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5881: Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout.
hadoop-yetus commented on PR #5881:
URL: https://github.com/apache/hadoop/pull/5881#issuecomment-1697001326
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 35m 49s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/6/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 50s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 39s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 22m 57s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 23s | | the patch passed |
| +1 :green_heart: | compile | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 15s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/6/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 7 new + 6 unchanged - 0
fixed = 13 total (was 6) |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| -1 :x: | javadoc | 0m 21s |
[/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/6/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt)
| hadoop-azure in the patch failed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 20s |
[/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/6/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-azure in the patch failed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| -1 :x: | spotbugs | 0m 46s |
[/new-spotbugs-hadoop-tools_hadoop-azure.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5881/6/artifact/out/new-spotbugs-hadoop-tools_hadoop-azure.html)
| hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total
(was 0) |
| +1 :green_heart: | shadedclient | 20m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 56s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 94m 34s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-tools/hadoop-azure |
| | Dead store to sol in
org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getWriteMaxConcurrentRequestCount()
At
AbfsConfiguration.java:org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getWriteMaxConcurrentRequestCount()
At AbfsConfiguration.java:[line 1071] |
| Subsystem | Report/Notes |
|--:|:-|
|
[GitHub] [hadoop] hadoop-yetus commented on pull request #5997: Plain connect0hook
hadoop-yetus commented on PR #5997:
URL: https://github.com/apache/hadoop/pull/5997#issuecomment-1696953263
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 52s | | trunk passed |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 50s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| -1 :x: | javac | 0m 23s |
[/results-compile-javac-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5997/1/artifact/out/results-compile-javac-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt)
| hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 generated 18 new + 0
unchanged - 0 fixed = 18 total (was 0) |
| +1 :green_heart: | compile | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| -1 :x: | javac | 0m 20s |
[/results-compile-javac-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5997/1/artifact/out/results-compile-javac-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05
with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 generated 18 new + 0
unchanged - 0 fixed = 18 total (was 0) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 16s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5997/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 25 new + 0 unchanged - 0
fixed = 25 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| -1 :x: | javadoc | 0m 18s |
[/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5997/1/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt)
| hadoop-azure in the patch failed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04. |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| -1 :x: | spotbugs | 0m 49s |
[/new-spotbugs-hadoop-tools_hadoop-azure.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5997/1/artifact/out/new-spotbugs-hadoop-tools_hadoop-azure.html)
| hadoop-tools/hadoop-azure generated 3 new + 0 unchanged - 0 fixed = 3 total
(was 0) |
| +1 :green_heart: | shadedclient | 23m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 54s | | hadoop-azure in the patch
passed. |
| -1 :x: | asflicense | 0m 30s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5997/1/artifact/out/results-asflicense.txt)
| The patch generated 3 ASF License warnings. |
| | | 90m 50s | | |
| Reason | Tests |
[GitHub] [hadoop] anujmodi2021 commented on a diff in pull request #5881: Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout.
anujmodi2021 commented on code in PR #5881:
URL: https://github.com/apache/hadoop/pull/5881#discussion_r1308306476
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestStaticRetryPolicy.java:
##
@@ -0,0 +1,124 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.AZURE_MAX_IO_RETRIES;
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.AZURE_STATIC_RETRY_FOR_CONNECTION_TIMEOUT_ENABLED;
+import static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.AZURE_STATIC_RETRY_INTERVAL;
+import static
org.apache.hadoop.fs.azurebfs.services.RetryReasonConstants.CONNECTION_RESET_ABBREVIATION;
+import static
org.apache.hadoop.fs.azurebfs.services.RetryReasonConstants.CONNECTION_TIMEOUT_ABBREVIATION;
+
+/**
+ * Class to test the behavior of Static Retry policy as well the inheritance
+ * between {@link AbfsRetryPolicy}, {@link ExponentialRetryPolicy}, {@link
StaticRetryPolicy}
+ */
+public class ITestStaticRetryPolicy extends AbstractAbfsIntegrationTest {
+
+ public ITestStaticRetryPolicy() throws Exception {
+super();
+ }
+
+ /**
+ * Tests for retry policy related configurations.
+ * Asserting that the correct retry policy is used for a given set of
+ * configurations including default ones
+ * @throws Exception
+ */
+ @Test
+ public void testStaticRetryPolicyInitializationDefault() throws Exception {
+Configuration config = new Configuration(this.getRawConfiguration());
+assertInitialization(config, StaticRetryPolicy.class);
+ }
+
+ @Test
+ public void testStaticRetryPolicyInitialization1() throws Exception {
+Configuration config = new Configuration(this.getRawConfiguration());
+config.set(AZURE_STATIC_RETRY_FOR_CONNECTION_TIMEOUT_ENABLED, "true");
+assertInitialization(config, StaticRetryPolicy.class);
+ }
+
+ @Test
+ public void testStaticRetryPolicyInitialization2() throws Exception {
+Configuration config = new Configuration(this.getRawConfiguration());
+config.set(AZURE_STATIC_RETRY_FOR_CONNECTION_TIMEOUT_ENABLED, "false");
+assertInitialization(config, ExponentialRetryPolicy.class);
+ }
+
+ private void assertInitialization(Configuration config, Class
retryPolicyClass) throws Exception{
+final AzureBlobFileSystem fs = (AzureBlobFileSystem) FileSystem
+.newInstance(getFileSystem().getUri(), config);
+AbfsClient client = fs.getAbfsStore().getClient();
+
+// Assert that static retry policy will be used only for CT Failures
+AbfsRetryPolicy retryPolicy =
client.getRetryPolicy(CONNECTION_TIMEOUT_ABBREVIATION);
+Assertions.assertThat(retryPolicy).isInstanceOf(retryPolicyClass);
+
+// For all other possible values of failureReason, Exponential retry is
used
+retryPolicy = client.getRetryPolicy("");
+
Assertions.assertThat(retryPolicy).isInstanceOf(ExponentialRetryPolicy.class);
+retryPolicy = client.getRetryPolicy(null);
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anujmodi2021 commented on a diff in pull request #5881: Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout.
anujmodi2021 commented on code in PR #5881:
URL: https://github.com/apache/hadoop/pull/5881#discussion_r1308305668
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/RetryPolicyConstants.java:
##
@@ -0,0 +1,28 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+public final class RetryPolicyConstants {
+
+ private RetryPolicyConstants() {
+
+ }
+ public static final String EXPONENTIAL_RETRY_POLICY_ABBREVIATION= "E";
Review Comment:
Added
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/StaticRetryPolicy.java:
##
@@ -0,0 +1,68 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+
+/**
+ * Retry policy used by AbfsClient for Network Errors.
+ * */
+public class StaticRetryPolicy extends AbfsRetryPolicy {
+
+ private static final int STATIC_RETRY_INTERVAL_DEFAULT = 2000; // 2s
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anujmodi2021 commented on a diff in pull request #5881: Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout.
anujmodi2021 commented on code in PR #5881:
URL: https://github.com/apache/hadoop/pull/5881#discussion_r1308304654
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRetryPolicy.java:
##
@@ -0,0 +1,73 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.HTTP_CONTINUE;
+
+/**
+ * Abstract Class for Retry policy to be used by {@link AbfsClient}
+ * Implementation to be used is based on retry cause.
+ */
+public abstract class AbfsRetryPolicy {
+
+ /**
+ * The maximum number of retry attempts.
+ */
+ private final int maxRetryCount;
+
+ public AbfsRetryPolicy(final int maxRetryCount) {
+this.maxRetryCount = maxRetryCount;
+ }
+
+ /**
+ * Returns if a request should be retried based on the retry count, current
response,
+ * and the current strategy. The valid http status code lies in the range of
1xx-5xx.
+ * But an invalid status code might be set due to network or timeout kind of
issues.
+ * Such invalid status code also qualify for retry.
+ *
+ * @param retryCount The current retry attempt count.
+ * @param statusCode The status code of the response, or -1 for socket error.
+ * @return true if the request should be retried; false otherwise.
+ */
+ public boolean shouldRetry(final int retryCount, final int statusCode) {
+return retryCount < this.maxRetryCount
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] anujmodi2021 commented on a diff in pull request #5881: Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout.
anujmodi2021 commented on code in PR #5881:
URL: https://github.com/apache/hadoop/pull/5881#discussion_r1308304654
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRetryPolicy.java:
##
@@ -0,0 +1,73 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.HTTP_CONTINUE;
+
+/**
+ * Abstract Class for Retry policy to be used by {@link AbfsClient}
+ * Implementation to be used is based on retry cause.
+ */
+public abstract class AbfsRetryPolicy {
+
+ /**
+ * The maximum number of retry attempts.
+ */
+ private final int maxRetryCount;
+
+ public AbfsRetryPolicy(final int maxRetryCount) {
+this.maxRetryCount = maxRetryCount;
+ }
+
+ /**
+ * Returns if a request should be retried based on the retry count, current
response,
+ * and the current strategy. The valid http status code lies in the range of
1xx-5xx.
+ * But an invalid status code might be set due to network or timeout kind of
issues.
+ * Such invalid status code also qualify for retry.
+ *
+ * @param retryCount The current retry attempt count.
+ * @param statusCode The status code of the response, or -1 for socket error.
+ * @return true if the request should be retried; false otherwise.
+ */
+ public boolean shouldRetry(final int retryCount, final int statusCode) {
+return retryCount < this.maxRetryCount
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
