[jira] [Moved] (HADOOP-17866) YarnClient Caching Addresses
[ https://issues.apache.org/jira/browse/HADOOP-17866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph moved YARN-10857 to HADOOP-17866: --- Component/s: (was: yarn) (was: client) Key: HADOOP-17866 (was: YARN-10857) Project: Hadoop Common (was: Hadoop YARN) > YarnClient Caching Addresses > > > Key: HADOOP-17866 > URL: https://issues.apache.org/jira/browse/HADOOP-17866 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Steve Suh >Assignee: Prabhu Joseph >Priority: Minor > > We have noticed that when the YarnClient is initialized and used, it is not > very resilient when dns or /etc/hosts is modified in the following scenario: > Take for instance the following (and reproducable) sequence of events that > can occur on a service that instantiates and uses YarnClient. > - Yarn has rm HA enabled (*yarn.resourcemanager.ha.enabled* is *true*) and > there are two rms (rm1 and rm2). > - *yarn.client.failover-proxy-provider* is set to > *org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider* > 1)rm2 is currently the active rm > 2)/etc/hosts (or dns) is missing host information for rm2 > 3)A service is started and it initializes the YarnClient at startup. > 4)At some point in time after YarnClient is done initializing, /etc/hosts > is updated and contains host information for rm2 > 5)Yarn is queried, for instance calling *yarnclient.getApplications()* > 6)All YarnClient attempts to communicate with rm2 fail with > UnknownHostExceptions, even though /etc/hosts now contains host information > for it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17865) [KMS] Some error messages are not logged in the KMS server
[ https://issues.apache.org/jira/browse/HADOOP-17865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reassigned HADOOP-17865: Assignee: Wei-Chiu Chuang > [KMS] Some error messages are not logged in the KMS server > -- > > Key: HADOOP-17865 > URL: https://issues.apache.org/jira/browse/HADOOP-17865 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > While troubleshooting a recent KMS issue, it was found a certain exceptions > are not logged in the server side. When client got an error, for example, > "token xxx not found in the cache", we wanted to find out more details from > the server side (stack trace, exact exception...etc) but it was not found. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17865) [KMS] Some error messages are not logged in the KMS server
Wei-Chiu Chuang created HADOOP-17865: Summary: [KMS] Some error messages are not logged in the KMS server Key: HADOOP-17865 URL: https://issues.apache.org/jira/browse/HADOOP-17865 Project: Hadoop Common Issue Type: Bug Components: kms Reporter: Wei-Chiu Chuang While troubleshooting a recent KMS issue, it was found a certain exceptions are not logged in the server side. When client got an error, for example, "token xxx not found in the cache", we wanted to find out more details from the server side (stack trace, exact exception...etc) but it was not found. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #3320: HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heteroge
Hexiaoqiao commented on pull request #3320: URL: https://github.com/apache/hadoop/pull/3320#issuecomment-906072715 @Neilxzn Please fix checkstyle and check failed unit tests first. Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Hexiaoqiao commented on pull request #3200: URL: https://github.com/apache/hadoop/pull/3200#issuecomment-906071805 @jojochuang Would you mind to have another check? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on a change in pull request #3322: HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS.
jojochuang commented on a change in pull request #3322:
URL: https://github.com/apache/hadoop/pull/3322#discussion_r696260085
##
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
##
@@ -908,4 +911,51 @@ private static SnapshotStatus toSnapshotStatus(
SnapshotStatus.getParentPath(fullPath)));
return snapshotStatus;
}
+
+ static BlockLocation[] toBlockLocationArray(Map json)
+ throws IOException {
+final Map rootmap =
+(Map) json.get(BlockLocation.class.getSimpleName() + "s");
+final List array =
+JsonUtilClient.getList(rootmap, BlockLocation.class.getSimpleName());
+Preconditions.checkNotNull(array);
+final BlockLocation[] locations = new BlockLocation[array.size()];
+int i = 0;
+for (Object object : array) {
+ final Map m = (Map) object;
+ locations[i++] = JsonUtilClient.toBlockLocation(m);
+}
+return locations;
+ }
+
+ /** Convert a Json map to BlockLocation. **/
+ static BlockLocation toBlockLocation(Map m) throws IOException {
Review comment:
this is only used by toBlockLocationArray() and the test of that covers
it too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
Hexiaoqiao commented on pull request #3200: URL: https://github.com/apache/hadoop/pull/3200#issuecomment-906060946 Thanks @amahussein for your works. I trigger Yetus and it seems works well. Failed unit test `TestDirectoryScanner ` looks not related to changes. AND verify this feature with pseudo cluster, it also seems work well. Just notice that this PR include different cherry-pick commits together. I am not sure if it is traced gracefully when check in. @brahmareddybattula do you have any idea here? If not I would like give my +1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Neilxzn commented on pull request #3320: HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneo
Neilxzn commented on pull request #3320: URL: https://github.com/apache/hadoop/pull/3320#issuecomment-906027161 Failed junit tests seem unrelated. And I try these tests in IDEA locally, these tests run pass. - hadoop.hdfs.server.namenode.ha.TestEditLogTailer - hadoop.hdfs.TestHDFSFileSystemContract - hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on a change in pull request #3322: HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS.
jojochuang commented on a change in pull request #3322:
URL: https://github.com/apache/hadoop/pull/3322#discussion_r696240622
##
File path:
hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java
##
@@ -2002,4 +2003,38 @@ public void testContentType() throws Exception {
() -> HttpFSUtils.jsonParse(conn));
conn.disconnect();
}
+
+ @Test
+ @TestDir
+ @TestJetty
+ @TestHdfs
+ public void testGetFileBlockLocations() throws Exception {
+createHttpFSServer(false, false);
+// Create a test directory
+String pathStr = "/tmp/tmp-snap-diff-test";
+createDirWithHttp(pathStr, "700", null);
+
+Path path = new Path(pathStr);
+DistributedFileSystem dfs = (DistributedFileSystem) FileSystem
+.get(path.toUri(), TestHdfsHelper.getHdfsConf());
+// Enable snapshot
+dfs.allowSnapshot(path);
+Assert.assertTrue(dfs.getFileStatus(path).isSnapshotEnabled());
+// Create a file and take a snapshot
+String file1 = pathStr + "/file1";
+createWithHttp(file1, null);
+HttpURLConnection conn = sendRequestToHttpFSServer(file1,
+"GETFILEBLOCKLOCATIONS", "length=10");
+Assert.assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+BlockLocation[] locations1 =
+dfs.getFileBlockLocations(new Path(file1), 0, 1);
+Assert.assertNotNull(locations1);
Review comment:
it makes no sense to check nullity of locations1 using hdfs. The code
doesn't change the file, so why check it?
##
File path:
hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java
##
@@ -2002,4 +2003,38 @@ public void testContentType() throws Exception {
() -> HttpFSUtils.jsonParse(conn));
conn.disconnect();
}
+
+ @Test
+ @TestDir
+ @TestJetty
+ @TestHdfs
+ public void testGetFileBlockLocations() throws Exception {
+createHttpFSServer(false, false);
+// Create a test directory
+String pathStr = "/tmp/tmp-snap-diff-test";
+createDirWithHttp(pathStr, "700", null);
+
+Path path = new Path(pathStr);
+DistributedFileSystem dfs = (DistributedFileSystem) FileSystem
+.get(path.toUri(), TestHdfsHelper.getHdfsConf());
+// Enable snapshot
+dfs.allowSnapshot(path);
+Assert.assertTrue(dfs.getFileStatus(path).isSnapshotEnabled());
+// Create a file and take a snapshot
+String file1 = pathStr + "/file1";
+createWithHttp(file1, null);
+HttpURLConnection conn = sendRequestToHttpFSServer(file1,
+"GETFILEBLOCKLOCATIONS", "length=10");
+Assert.assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+BlockLocation[] locations1 =
+dfs.getFileBlockLocations(new Path(file1), 0, 1);
+Assert.assertNotNull(locations1);
+
+HttpURLConnection conn1 = sendRequestToHttpFSServer(file1,
+"GET_BLOCK_LOCATIONS", "length=10");
+Assert.assertEquals(HttpURLConnection.HTTP_OK, conn1.getResponseCode());
+BlockLocation[] locations2 =
+dfs.getFileBlockLocations(new Path(file1), 0, 1);
+Assert.assertNotNull(locations2);
Review comment:
here, too.
##
File path:
hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java
##
@@ -2002,4 +2003,38 @@ public void testContentType() throws Exception {
() -> HttpFSUtils.jsonParse(conn));
conn.disconnect();
}
+
+ @Test
+ @TestDir
+ @TestJetty
+ @TestHdfs
+ public void testGetFileBlockLocations() throws Exception {
+createHttpFSServer(false, false);
+// Create a test directory
+String pathStr = "/tmp/tmp-snap-diff-test";
+createDirWithHttp(pathStr, "700", null);
+
+Path path = new Path(pathStr);
+DistributedFileSystem dfs = (DistributedFileSystem) FileSystem
+.get(path.toUri(), TestHdfsHelper.getHdfsConf());
+// Enable snapshot
Review comment:
no i don't understand this. it doesn't look like snapshot is used at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on a change in pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
jianghuazhu commented on a change in pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#discussion_r696235839
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java
##
@@ -329,6 +340,25 @@ private void setNumberThreads(String numberThreadsString) {
}
}
+private void setThreadPoolQueueSize(String numThreadPoolQueueSize) {
+ if (numThreadPoolQueueSize != null) {
+int parsedValue = Integer.parseInt(numThreadPoolQueueSize);
+if (parsedValue < 1) {
+ LOG.warn("The value of the thread pool queue size cannot be " +
+ "less than 1, and the default value is used here. " +
+ "The default size is 1024.");
+ threadPoolQueueSize = 1024;
+} else {
+ threadPoolQueueSize = parsedValue;
+}
+ }
+}
+
+@VisibleForTesting
+public int getThreadPoolQueueSize() {
Review comment:
Thanks @aajisaka for the suggestion, I will continue to work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] fengnanli commented on pull request #3284: HDFS-16157. Support configuring DNS record to get list of journal nodes.
fengnanli commented on pull request #3284: URL: https://github.com/apache/hadoop/pull/3284#issuecomment-905970607 Thanks for the work @LeonGao91 and review from @Hexiaoqiao -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] fengnanli merged pull request #3284: HDFS-16157. Support configuring DNS record to get list of journal nodes.
fengnanli merged pull request #3284: URL: https://github.com/apache/hadoop/pull/3284 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3320: HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Hetero
hadoop-yetus commented on pull request #3320:
URL: https://github.com/apache/hadoop/pull/3320#issuecomment-905964389
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 6s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 34s | | trunk passed |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 6s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 11s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 11s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 54s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3320/3/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 153 unchanged
- 2 fixed = 155 total (was 155) |
| +1 :green_heart: | mvnsite | 1m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 353m 51s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3320/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 447m 30s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.TestHDFSFileSystemContract |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3320/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3320 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux f0c899902df0 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 90fcf70f258934eb56c25cea56533b3aaa481e5d |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3320/3/testReport/ |
| Max. process+thread count | 2005 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[GitHub] [hadoop] hadoop-yetus commented on pull request #3314: YARN-10891. Extend QueueInfo with max-parallel-apps in CS.
hadoop-yetus commented on pull request #3314:
URL: https://github.com/apache/hadoop/pull/3314#issuecomment-905915020
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 3s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 41s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 18s | | trunk passed |
| +1 :green_heart: | compile | 12m 32s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 11m 1s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 0s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 35s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 30s | |
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file
(spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 19m 19s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 25s | | the patch passed |
| +1 :green_heart: | compile | 11m 40s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | cc | 11m 40s | | the patch passed |
| +1 :green_heart: | javac | 11m 40s | | the patch passed |
| +1 :green_heart: | compile | 10m 53s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | cc | 10m 53s |
[/results-compile-cc-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3314/9/artifact/out/results-compile-cc-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
|
hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 3 new
+ 179 unchanged - 3 fixed = 182 total (was 182) |
| +1 :green_heart: | javac | 10m 53s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 59s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3314/9/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 4 new + 628 unchanged
- 0 fixed = 632 total (was 628) |
| +1 :green_heart: | mvnsite | 4m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 35s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 27s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 25s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs |
| +1 :green_heart: | shadedclient | 19m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 9s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 7s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 106m 18s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 27m 19s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | unit | 0m 27s | | hadoop-yarn-site in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not
generate ASF License warnings. |
| | | 323m 26s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
[jira] [Commented] (HADOOP-17857) Check real user ACLs in addition to proxied user ACLs
[
https://issues.apache.org/jira/browse/HADOOP-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17404740#comment-17404740
]
Hadoop QA commented on HADOOP-17857:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
11s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 30m
44s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
4s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
22s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 14s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 21m
30s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m
25s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
26s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
26s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
16s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m
16s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 7s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-HADOOP-Build/228/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt{color}
| {color:orange} hadoop-common-project/hadoop-common: The patch generated 1
new + 39 unchanged - 1 fixed = 40 total (was 40) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 16s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green}{color} |
[jira] [Work logged] (HADOOP-17682) ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
[
https://issues.apache.org/jira/browse/HADOOP-17682?focusedWorklogId=641945=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641945
]
ASF GitHub Bot logged work on HADOOP-17682:
---
Author: ASF GitHub Bot
Created on: 25/Aug/21 19:59
Start Date: 25/Aug/21 19:59
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#2975:
URL: https://github.com/apache/hadoop/pull/2975#discussion_r696063619
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##
@@ -669,44 +672,64 @@ public void createDirectory(final Path path, final
FsPermission permission,
public AbfsInputStream openFileForRead(final Path path,
final FileSystem.Statistics statistics, TracingContext tracingContext)
- throws AzureBlobFileSystemException {
-return openFileForRead(path, Optional.empty(), statistics, tracingContext);
+ throws IOException {
+return openFileForRead(path, Optional.empty(), statistics,
+tracingContext);
}
- public AbfsInputStream openFileForRead(final Path path,
- final Optional options,
+ public AbfsInputStream openFileForRead(Path path,
+ final Optional parameters,
final FileSystem.Statistics statistics, TracingContext tracingContext)
- throws AzureBlobFileSystemException {
-try (AbfsPerfInfo perfInfo = startTracking("openFileForRead",
"getPathStatus")) {
+ throws IOException {
+try (AbfsPerfInfo perfInfo = startTracking("openFileForRead",
+"getPathStatus")) {
LOG.debug("openFileForRead filesystem: {} path: {}",
- client.getFileSystem(),
- path);
+ client.getFileSystem(), path);
+ FileStatus fileStatus = parameters.map(OpenFileParameters::getStatus)
+ .orElse(null);
String relativePath = getRelativePath(path);
-
- final AbfsRestOperation op = client
- .getPathStatus(relativePath, false, tracingContext);
- perfInfo.registerResult(op.getResult());
-
- final String resourceType =
op.getResult().getResponseHeader(HttpHeaderConfigurations.X_MS_RESOURCE_TYPE);
- final long contentLength =
Long.parseLong(op.getResult().getResponseHeader(HttpHeaderConfigurations.CONTENT_LENGTH));
- final String eTag =
op.getResult().getResponseHeader(HttpHeaderConfigurations.ETAG);
+ String resourceType, eTag;
+ long contentLength;
+ if (fileStatus instanceof VersionedFileStatus) {
+path = path.makeQualified(this.uri, path);
+Preconditions.checkArgument(fileStatus.getPath().equals(path),
+String.format(
+"Filestatus path [%s] does not match with given path [%s]",
+fileStatus.getPath(), path));
+resourceType = fileStatus.isFile() ? FILE : DIRECTORY;
+contentLength = fileStatus.getLen();
+eTag = ((VersionedFileStatus) fileStatus).getVersion();
+ } else {
+if (fileStatus != null) {
+ LOG.warn(
Review comment:
no, just use length & type, don't worry about etag. Or at least just
log @ debug
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##
@@ -192,6 +201,106 @@ public TestAbfsInputStream() throws Exception {
ReadBufferManager.getBufferManager().setThresholdAgeMilliseconds(REDUCED_READ_BUFFER_AGE_THRESHOLD);
}
+ private void writeBufferToNewFile(Path testFile, byte[] buffer) throws
IOException {
+AzureBlobFileSystem fs = getFileSystem();
+fs.create(testFile);
+FSDataOutputStream out = fs.append(testFile);
+out.write(buffer);
+out.close();
+ }
+
+ private void verifyOpenWithProvidedStatus(Path path, FileStatus fileStatus,
+ byte[] buf, AbfsRestOperationType source)
+ throws IOException, ExecutionException, InterruptedException {
+byte[] readBuf = new byte[buf.length];
+AzureBlobFileSystem fs = getFileSystem();
+FutureDataInputStreamBuilder builder = fs.openFile(path);
+builder.withFileStatus(fileStatus);
+FSDataInputStream in = builder.build().get();
+assertEquals(String.format(
Review comment:
use AssertJ
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##
@@ -669,44 +672,64 @@ public void createDirectory(final Path path, final
FsPermission permission,
public AbfsInputStream openFileForRead(final Path path,
final FileSystem.Statistics statistics, TracingContext tracingContext)
- throws AzureBlobFileSystemException {
-return openFileForRead(path, Optional.empty(), statistics, tracingContext);
+ throws IOException {
+return openFileForRead(path, Optional.empty(), statistics,
+tracingContext);
}
- public
[GitHub] [hadoop] steveloughran commented on a change in pull request #2975: HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
steveloughran commented on a change in pull request #2975:
URL: https://github.com/apache/hadoop/pull/2975#discussion_r696063619
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##
@@ -669,44 +672,64 @@ public void createDirectory(final Path path, final
FsPermission permission,
public AbfsInputStream openFileForRead(final Path path,
final FileSystem.Statistics statistics, TracingContext tracingContext)
- throws AzureBlobFileSystemException {
-return openFileForRead(path, Optional.empty(), statistics, tracingContext);
+ throws IOException {
+return openFileForRead(path, Optional.empty(), statistics,
+tracingContext);
}
- public AbfsInputStream openFileForRead(final Path path,
- final Optional options,
+ public AbfsInputStream openFileForRead(Path path,
+ final Optional parameters,
final FileSystem.Statistics statistics, TracingContext tracingContext)
- throws AzureBlobFileSystemException {
-try (AbfsPerfInfo perfInfo = startTracking("openFileForRead",
"getPathStatus")) {
+ throws IOException {
+try (AbfsPerfInfo perfInfo = startTracking("openFileForRead",
+"getPathStatus")) {
LOG.debug("openFileForRead filesystem: {} path: {}",
- client.getFileSystem(),
- path);
+ client.getFileSystem(), path);
+ FileStatus fileStatus = parameters.map(OpenFileParameters::getStatus)
+ .orElse(null);
String relativePath = getRelativePath(path);
-
- final AbfsRestOperation op = client
- .getPathStatus(relativePath, false, tracingContext);
- perfInfo.registerResult(op.getResult());
-
- final String resourceType =
op.getResult().getResponseHeader(HttpHeaderConfigurations.X_MS_RESOURCE_TYPE);
- final long contentLength =
Long.parseLong(op.getResult().getResponseHeader(HttpHeaderConfigurations.CONTENT_LENGTH));
- final String eTag =
op.getResult().getResponseHeader(HttpHeaderConfigurations.ETAG);
+ String resourceType, eTag;
+ long contentLength;
+ if (fileStatus instanceof VersionedFileStatus) {
+path = path.makeQualified(this.uri, path);
+Preconditions.checkArgument(fileStatus.getPath().equals(path),
+String.format(
+"Filestatus path [%s] does not match with given path [%s]",
+fileStatus.getPath(), path));
+resourceType = fileStatus.isFile() ? FILE : DIRECTORY;
+contentLength = fileStatus.getLen();
+eTag = ((VersionedFileStatus) fileStatus).getVersion();
+ } else {
+if (fileStatus != null) {
+ LOG.warn(
Review comment:
no, just use length & type, don't worry about etag. Or at least just
log @ debug
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##
@@ -192,6 +201,106 @@ public TestAbfsInputStream() throws Exception {
ReadBufferManager.getBufferManager().setThresholdAgeMilliseconds(REDUCED_READ_BUFFER_AGE_THRESHOLD);
}
+ private void writeBufferToNewFile(Path testFile, byte[] buffer) throws
IOException {
+AzureBlobFileSystem fs = getFileSystem();
+fs.create(testFile);
+FSDataOutputStream out = fs.append(testFile);
+out.write(buffer);
+out.close();
+ }
+
+ private void verifyOpenWithProvidedStatus(Path path, FileStatus fileStatus,
+ byte[] buf, AbfsRestOperationType source)
+ throws IOException, ExecutionException, InterruptedException {
+byte[] readBuf = new byte[buf.length];
+AzureBlobFileSystem fs = getFileSystem();
+FutureDataInputStreamBuilder builder = fs.openFile(path);
+builder.withFileStatus(fileStatus);
+FSDataInputStream in = builder.build().get();
+assertEquals(String.format(
Review comment:
use AssertJ
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##
@@ -669,44 +672,64 @@ public void createDirectory(final Path path, final
FsPermission permission,
public AbfsInputStream openFileForRead(final Path path,
final FileSystem.Statistics statistics, TracingContext tracingContext)
- throws AzureBlobFileSystemException {
-return openFileForRead(path, Optional.empty(), statistics, tracingContext);
+ throws IOException {
+return openFileForRead(path, Optional.empty(), statistics,
+tracingContext);
}
- public AbfsInputStream openFileForRead(final Path path,
- final Optional options,
+ public AbfsInputStream openFileForRead(Path path,
+ final Optional parameters,
final FileSystem.Statistics statistics, TracingContext tracingContext)
- throws AzureBlobFileSystemException {
-try (AbfsPerfInfo perfInfo = startTracking("openFileForRead",
"getPathStatus")) {
+ throws IOException {
+try (AbfsPerfInfo perfInfo
[jira] [Work logged] (HADOOP-17682) ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
[ https://issues.apache.org/jira/browse/HADOOP-17682?focusedWorklogId=641936=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641936 ] ASF GitHub Bot logged work on HADOOP-17682: --- Author: ASF GitHub Bot Created on: 25/Aug/21 19:30 Start Date: 25/Aug/21 19:30 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2975: URL: https://github.com/apache/hadoop/pull/2975#issuecomment-49397 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 641936) Time Spent: 7h (was: 6h 50m) > ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters > -- > > Key: HADOOP-17682 > URL: https://issues.apache.org/jira/browse/HADOOP-17682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 7h > Remaining Estimate: 0h > > ABFS open methods require certain information (contentLength, eTag, etc) to > to create an InputStream for the file at the given path. This information is > retrieved via a GetFileStatus request to backend. > However, client applications may often have access to the FileStatus prior to > invoking the open API. Providing this FileStatus to the driver through the > OpenFileParameters argument of openFileWithOptions() can help avoid the call > to Store for FileStatus. > This PR adds handling for the FileStatus instance (if any) provided via the > OpenFileParameters argument. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17682) ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
[ https://issues.apache.org/jira/browse/HADOOP-17682?focusedWorklogId=641935=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641935 ] ASF GitHub Bot logged work on HADOOP-17682: --- Author: ASF GitHub Bot Created on: 25/Aug/21 19:30 Start Date: 25/Aug/21 19:30 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2975: URL: https://github.com/apache/hadoop/pull/2975#issuecomment-888794912 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 641935) Time Spent: 6h 50m (was: 6h 40m) > ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters > -- > > Key: HADOOP-17682 > URL: https://issues.apache.org/jira/browse/HADOOP-17682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 6h 50m > Remaining Estimate: 0h > > ABFS open methods require certain information (contentLength, eTag, etc) to > to create an InputStream for the file at the given path. This information is > retrieved via a GetFileStatus request to backend. > However, client applications may often have access to the FileStatus prior to > invoking the open API. Providing this FileStatus to the driver through the > OpenFileParameters argument of openFileWithOptions() can help avoid the call > to Store for FileStatus. > This PR adds handling for the FileStatus instance (if any) provided via the > OpenFileParameters argument. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17682) ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
[ https://issues.apache.org/jira/browse/HADOOP-17682?focusedWorklogId=641934=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641934 ] ASF GitHub Bot logged work on HADOOP-17682: --- Author: ASF GitHub Bot Created on: 25/Aug/21 19:30 Start Date: 25/Aug/21 19:30 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2975: URL: https://github.com/apache/hadoop/pull/2975#issuecomment-879797819 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 641934) Time Spent: 6h 40m (was: 6.5h) > ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters > -- > > Key: HADOOP-17682 > URL: https://issues.apache.org/jira/browse/HADOOP-17682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 6h 40m > Remaining Estimate: 0h > > ABFS open methods require certain information (contentLength, eTag, etc) to > to create an InputStream for the file at the given path. This information is > retrieved via a GetFileStatus request to backend. > However, client applications may often have access to the FileStatus prior to > invoking the open API. Providing this FileStatus to the driver through the > OpenFileParameters argument of openFileWithOptions() can help avoid the call > to Store for FileStatus. > This PR adds handling for the FileStatus instance (if any) provided via the > OpenFileParameters argument. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2975: HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
hadoop-yetus removed a comment on pull request #2975: URL: https://github.com/apache/hadoop/pull/2975#issuecomment-49397 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2975: HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
hadoop-yetus removed a comment on pull request #2975: URL: https://github.com/apache/hadoop/pull/2975#issuecomment-888794912 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2975: HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
hadoop-yetus removed a comment on pull request #2975: URL: https://github.com/apache/hadoop/pull/2975#issuecomment-879797819 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17682) ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
[ https://issues.apache.org/jira/browse/HADOOP-17682?focusedWorklogId=641932=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641932 ] ASF GitHub Bot logged work on HADOOP-17682: --- Author: ASF GitHub Bot Created on: 25/Aug/21 19:29 Start Date: 25/Aug/21 19:29 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2975: URL: https://github.com/apache/hadoop/pull/2975#issuecomment-833554277 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 641932) Time Spent: 6h 20m (was: 6h 10m) > ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters > -- > > Key: HADOOP-17682 > URL: https://issues.apache.org/jira/browse/HADOOP-17682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 6h 20m > Remaining Estimate: 0h > > ABFS open methods require certain information (contentLength, eTag, etc) to > to create an InputStream for the file at the given path. This information is > retrieved via a GetFileStatus request to backend. > However, client applications may often have access to the FileStatus prior to > invoking the open API. Providing this FileStatus to the driver through the > OpenFileParameters argument of openFileWithOptions() can help avoid the call > to Store for FileStatus. > This PR adds handling for the FileStatus instance (if any) provided via the > OpenFileParameters argument. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17682) ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
[
https://issues.apache.org/jira/browse/HADOOP-17682?focusedWorklogId=641933=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641933
]
ASF GitHub Bot logged work on HADOOP-17682:
---
Author: ASF GitHub Bot
Created on: 25/Aug/21 19:29
Start Date: 25/Aug/21 19:29
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #2975:
URL: https://github.com/apache/hadoop/pull/2975#issuecomment-879494586
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 59s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | spotbugs | 1m 10s |
[/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2975/12/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html)
| hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 13m 59s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 20s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2975/12/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 1 new + 4 unchanged - 0
fixed = 5 total (was 4) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 53s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 86m 12s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2975/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2975 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 9f99dad8eb07 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2e23baff44f6422075a3897f9bb2d23a91e4e338 |
| Default Java | Private
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2975: HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
hadoop-yetus removed a comment on pull request #2975:
URL: https://github.com/apache/hadoop/pull/2975#issuecomment-879494586
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 59s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | spotbugs | 1m 10s |
[/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2975/12/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html)
| hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 13m 59s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 20s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2975/12/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 1 new + 4 unchanged - 0
fixed = 5 total (was 4) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 53s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 86m 12s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2975/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2975 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 9f99dad8eb07 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2e23baff44f6422075a3897f9bb2d23a91e4e338 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2975/12/testReport/ |
| Max. process+thread count | 714 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U:
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2975: HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
hadoop-yetus removed a comment on pull request #2975: URL: https://github.com/apache/hadoop/pull/2975#issuecomment-833554277 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17682) ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
[ https://issues.apache.org/jira/browse/HADOOP-17682?focusedWorklogId=641930=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641930 ] ASF GitHub Bot logged work on HADOOP-17682: --- Author: ASF GitHub Bot Created on: 25/Aug/21 19:28 Start Date: 25/Aug/21 19:28 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2975: URL: https://github.com/apache/hadoop/pull/2975#issuecomment-831949886 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 641930) Time Spent: 6h (was: 5h 50m) > ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters > -- > > Key: HADOOP-17682 > URL: https://issues.apache.org/jira/browse/HADOOP-17682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 6h > Remaining Estimate: 0h > > ABFS open methods require certain information (contentLength, eTag, etc) to > to create an InputStream for the file at the given path. This information is > retrieved via a GetFileStatus request to backend. > However, client applications may often have access to the FileStatus prior to > invoking the open API. Providing this FileStatus to the driver through the > OpenFileParameters argument of openFileWithOptions() can help avoid the call > to Store for FileStatus. > This PR adds handling for the FileStatus instance (if any) provided via the > OpenFileParameters argument. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17682) ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
[ https://issues.apache.org/jira/browse/HADOOP-17682?focusedWorklogId=641931=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641931 ] ASF GitHub Bot logged work on HADOOP-17682: --- Author: ASF GitHub Bot Created on: 25/Aug/21 19:28 Start Date: 25/Aug/21 19:28 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2975: URL: https://github.com/apache/hadoop/pull/2975#issuecomment-833013347 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 641931) Time Spent: 6h 10m (was: 6h) > ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters > -- > > Key: HADOOP-17682 > URL: https://issues.apache.org/jira/browse/HADOOP-17682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 6h 10m > Remaining Estimate: 0h > > ABFS open methods require certain information (contentLength, eTag, etc) to > to create an InputStream for the file at the given path. This information is > retrieved via a GetFileStatus request to backend. > However, client applications may often have access to the FileStatus prior to > invoking the open API. Providing this FileStatus to the driver through the > OpenFileParameters argument of openFileWithOptions() can help avoid the call > to Store for FileStatus. > This PR adds handling for the FileStatus instance (if any) provided via the > OpenFileParameters argument. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17682) ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
[
https://issues.apache.org/jira/browse/HADOOP-17682?focusedWorklogId=641929=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641929
]
ASF GitHub Bot logged work on HADOOP-17682:
---
Author: ASF GitHub Bot
Created on: 25/Aug/21 19:28
Start Date: 25/Aug/21 19:28
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #2975:
URL: https://github.com/apache/hadoop/pull/2975#issuecomment-831945482
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 7s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 15s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 17s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2975/2/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 1 new + 2 unchanged - 0
fixed = 3 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 56s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 3s | | hadoop-azure in the patch
passed. |
| -1 :x: | asflicense | 0m 33s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2975/2/artifact/out/results-asflicense.txt)
| The patch generated 1 ASF License warnings. |
| | | 75m 19s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2975/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2975 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 3c035b084ac3 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17
17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6f123194952f2473a7f7335305c7af633c2d0d5e |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2975/2/testReport/ |
| Max.
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2975: HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
hadoop-yetus removed a comment on pull request #2975: URL: https://github.com/apache/hadoop/pull/2975#issuecomment-833013347 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2975: HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
hadoop-yetus removed a comment on pull request #2975: URL: https://github.com/apache/hadoop/pull/2975#issuecomment-831949886 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2975: HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
hadoop-yetus removed a comment on pull request #2975:
URL: https://github.com/apache/hadoop/pull/2975#issuecomment-831945482
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 7s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 15s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 17s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2975/2/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 1 new + 2 unchanged - 0
fixed = 3 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 56s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 3s | | hadoop-azure in the patch
passed. |
| -1 :x: | asflicense | 0m 33s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2975/2/artifact/out/results-asflicense.txt)
| The patch generated 1 ASF License warnings. |
| | | 75m 19s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2975/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2975 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 3c035b084ac3 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17
17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6f123194952f2473a7f7335305c7af633c2d0d5e |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2975/2/testReport/ |
| Max. process+thread count | 720 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2975/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an
[jira] [Commented] (HADOOP-17857) Check real user ACLs in addition to proxied user ACLs
[
https://issues.apache.org/jira/browse/HADOOP-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17404659#comment-17404659
]
Eric Payne commented on HADOOP-17857:
-
[~zhoukang] , [~snemeth], I believe that this JIRA is the first step to
addressing the requirements outlined in YARN-9975. Can you please review the
changes and let me know what you think?
> Check real user ACLs in addition to proxied user ACLs
> -
>
> Key: HADOOP-17857
> URL: https://issues.apache.org/jira/browse/HADOOP-17857
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.2.2, 2.10.1, 3.3.1
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Major
> Attachments: HADOOP-17857.001.patch, HADOOP-17857.002.patch
>
>
> In a secure cluster, it is possible to configure the services to allow a
> super-user to proxy to a regular user and perform actions on behalf of the
> proxied user (see [Proxy user - Superusers Acting On Behalf Of Other
> Users|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html]).
> This is useful for automating server access for multiple different users in a
> multi-tenant cluster. For example, this can be used by a super user
> submitting jobs to a YARN queue, accessing HDFS files, scheduling Oozie
> workflows, etc, which will then execute the service as the proxied user.
> Usually when these services check ACLs to determine if the user has access to
> the requested resources, the service only needs to check the ACLs for the
> proxied user. However, it is sometimes desirable to allow the proxied user to
> have access to the resources when only the real user has open ACLs.
> For instance, let's say the user {{adm}} is the only user with submit ACLs to
> the {{dataload}} queue, and the {{adm}} user wants to submit apps to the
> {{dataload}} queue on behalf of users {{headless1}} and {{headless2}}. In
> addition, we want to be able to bill {{headless1}} and {{headless2}}
> separately for the YARN resources used in the {{dataload}} queue. In order to
> do this, the apps need to run in the {{dataload}} queue as the respective
> headless users. We could open up the ACLs to the {{dataload}} queue to allow
> {{headless1}} and {{headless2}} to submit apps. But this would allow those
> users to submit any app to that queue, and not be limited to just the data
> loading apps, and we don't trust the {{headless1}} and {{headless2}} owners
> to honor that restriction.
> This JIRA proposes that we define a way to set up ACLs to restrict a
> resource's access to a super-user, but when the access happens, run it as
> the proxied user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17857) Check real user ACLs in addition to proxied user ACLs
[
https://issues.apache.org/jira/browse/HADOOP-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17404656#comment-17404656
]
Eric Payne commented on HADOOP-17857:
-
Thanks [~ahussein] for the review and comments! I have attached v2 of the patch.
> Check real user ACLs in addition to proxied user ACLs
> -
>
> Key: HADOOP-17857
> URL: https://issues.apache.org/jira/browse/HADOOP-17857
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.2.2, 2.10.1, 3.3.1
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Major
> Attachments: HADOOP-17857.001.patch, HADOOP-17857.002.patch
>
>
> In a secure cluster, it is possible to configure the services to allow a
> super-user to proxy to a regular user and perform actions on behalf of the
> proxied user (see [Proxy user - Superusers Acting On Behalf Of Other
> Users|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html]).
> This is useful for automating server access for multiple different users in a
> multi-tenant cluster. For example, this can be used by a super user
> submitting jobs to a YARN queue, accessing HDFS files, scheduling Oozie
> workflows, etc, which will then execute the service as the proxied user.
> Usually when these services check ACLs to determine if the user has access to
> the requested resources, the service only needs to check the ACLs for the
> proxied user. However, it is sometimes desirable to allow the proxied user to
> have access to the resources when only the real user has open ACLs.
> For instance, let's say the user {{adm}} is the only user with submit ACLs to
> the {{dataload}} queue, and the {{adm}} user wants to submit apps to the
> {{dataload}} queue on behalf of users {{headless1}} and {{headless2}}. In
> addition, we want to be able to bill {{headless1}} and {{headless2}}
> separately for the YARN resources used in the {{dataload}} queue. In order to
> do this, the apps need to run in the {{dataload}} queue as the respective
> headless users. We could open up the ACLs to the {{dataload}} queue to allow
> {{headless1}} and {{headless2}} to submit apps. But this would allow those
> users to submit any app to that queue, and not be limited to just the data
> loading apps, and we don't trust the {{headless1}} and {{headless2}} owners
> to honor that restriction.
> This JIRA proposes that we define a way to set up ACLs to restrict a
> resource's access to a super-user, but when the access happens, run it as
> the proxied user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17857) Check real user ACLs in addition to proxied user ACLs
[
https://issues.apache.org/jira/browse/HADOOP-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated HADOOP-17857:
Attachment: HADOOP-17857.002.patch
> Check real user ACLs in addition to proxied user ACLs
> -
>
> Key: HADOOP-17857
> URL: https://issues.apache.org/jira/browse/HADOOP-17857
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.2.2, 2.10.1, 3.3.1
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Major
> Attachments: HADOOP-17857.001.patch, HADOOP-17857.002.patch
>
>
> In a secure cluster, it is possible to configure the services to allow a
> super-user to proxy to a regular user and perform actions on behalf of the
> proxied user (see [Proxy user - Superusers Acting On Behalf Of Other
> Users|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html]).
> This is useful for automating server access for multiple different users in a
> multi-tenant cluster. For example, this can be used by a super user
> submitting jobs to a YARN queue, accessing HDFS files, scheduling Oozie
> workflows, etc, which will then execute the service as the proxied user.
> Usually when these services check ACLs to determine if the user has access to
> the requested resources, the service only needs to check the ACLs for the
> proxied user. However, it is sometimes desirable to allow the proxied user to
> have access to the resources when only the real user has open ACLs.
> For instance, let's say the user {{adm}} is the only user with submit ACLs to
> the {{dataload}} queue, and the {{adm}} user wants to submit apps to the
> {{dataload}} queue on behalf of users {{headless1}} and {{headless2}}. In
> addition, we want to be able to bill {{headless1}} and {{headless2}}
> separately for the YARN resources used in the {{dataload}} queue. In order to
> do this, the apps need to run in the {{dataload}} queue as the respective
> headless users. We could open up the ACLs to the {{dataload}} queue to allow
> {{headless1}} and {{headless2}} to submit apps. But this would allow those
> users to submit any app to that queue, and not be limited to just the data
> loading apps, and we don't trust the {{headless1}} and {{headless2}} owners
> to honor that restriction.
> This JIRA proposes that we define a way to set up ACLs to restrict a
> resource's access to a super-user, but when the access happens, run it as
> the proxied user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17833) Improve Magic Committer Performance
[
https://issues.apache.org/jira/browse/HADOOP-17833?focusedWorklogId=641861=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641861
]
ASF GitHub Bot logged work on HADOOP-17833:
---
Author: ASF GitHub Bot
Created on: 25/Aug/21 17:42
Start Date: 25/Aug/21 17:42
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3289:
URL: https://github.com/apache/hadoop/pull/3289#issuecomment-905742205
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 25 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 35s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 0s | | trunk passed |
| +1 :green_heart: | compile | 24m 9s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 21m 33s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 24s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 41s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 6s | | the patch passed |
| +1 :green_heart: | compile | 31m 41s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 31m 41s |
[/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/8/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 1922 unchanged - 0
fixed = 1923 total (was 1922) |
| +1 :green_heart: | compile | 28m 37s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 28m 37s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/8/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 1794
unchanged - 0 fixed = 1795 total (was 1794) |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/8/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 5m 13s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/8/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 28 unchanged - 1 fixed = 29 total (was
29) |
| +1 :green_heart: | mvnsite | 3m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 2s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 10s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 31s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 20m 56s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 6s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 6s | | The patch does not
[GitHub] [hadoop] hadoop-yetus commented on pull request #3289: HADOOP-17833. Improve Magic Committer performance
hadoop-yetus commented on pull request #3289:
URL: https://github.com/apache/hadoop/pull/3289#issuecomment-905742205
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 25 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 35s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 0s | | trunk passed |
| +1 :green_heart: | compile | 24m 9s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 21m 33s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 24s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 41s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 6s | | the patch passed |
| +1 :green_heart: | compile | 31m 41s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 31m 41s |
[/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/8/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 1922 unchanged - 0
fixed = 1923 total (was 1922) |
| +1 :green_heart: | compile | 28m 37s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 28m 37s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/8/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new + 1794
unchanged - 0 fixed = 1795 total (was 1794) |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/8/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 5m 13s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/8/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 28 unchanged - 1 fixed = 29 total (was
29) |
| +1 :green_heart: | mvnsite | 3m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 2s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 10s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 31s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 20m 56s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 6s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 6s | | The patch does not
generate ASF License warnings. |
| | | 250m 2s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3289 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit
[GitHub] [hadoop] Neilxzn commented on pull request #3320: HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneo
Neilxzn commented on pull request #3320: URL: https://github.com/apache/hadoop/pull/3320#issuecomment-905714180 > Thanks @Neilxzn for your works. Great catch here! LGTM. +1 once leaved nits comments fixed. I think it is a bug just when fallback storage policy happens. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Neilxzn removed a comment on pull request #3320: HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Het
Neilxzn removed a comment on pull request #3320: URL: https://github.com/apache/hadoop/pull/3320#issuecomment-905710927 Just wonder if this is common bug for every chooseTarget invokes when assign requiredStorageTypes.size() to numOfReplicas here? or just for set storagePolicy? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Neilxzn commented on pull request #3320: HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneo
Neilxzn commented on pull request #3320: URL: https://github.com/apache/hadoop/pull/3320#issuecomment-905710927 Just wonder if this is common bug for every chooseTarget invokes when assign requiredStorageTypes.size() to numOfReplicas here? or just for set storagePolicy? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Neilxzn commented on a change in pull request #3320: HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with
Neilxzn commented on a change in pull request #3320:
URL: https://github.com/apache/hadoop/pull/3320#discussion_r695940727
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
##
@@ -1337,6 +1337,50 @@ public void testChooseSsdOverDisk() throws Exception {
Assert.assertEquals(StorageType.DISK, targets[1].getStorageType());
}
+ @Test
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3314: YARN-10891. Extend QueueInfo with max-parallel-apps in CS.
hadoop-yetus commented on pull request #3314:
URL: https://github.com/apache/hadoop/pull/3314#issuecomment-905697347
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 4s | | trunk passed |
| +1 :green_heart: | compile | 10m 49s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 8m 55s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 48s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 14s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 11s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 27s | |
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file
(spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 16m 13s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 44s | | the patch passed |
| +1 :green_heart: | compile | 9m 33s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | cc | 9m 33s | | the patch passed |
| +1 :green_heart: | javac | 9m 33s | | the patch passed |
| +1 :green_heart: | compile | 8m 27s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | cc | 8m 27s |
[/results-compile-cc-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3314/8/artifact/out/results-compile-cc-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
|
hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 3 new
+ 179 unchanged - 3 fixed = 182 total (was 182) |
| +1 :green_heart: | javac | 8m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 37s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3314/8/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 4 new + 628 unchanged
- 0 fixed = 632 total (was 628) |
| +1 :green_heart: | mvnsite | 3m 42s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 10s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 0s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 25s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs |
| +1 :green_heart: | shadedclient | 16m 28s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 58s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 4m 41s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 100m 3s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 28m 1s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | unit | 0m 25s | | hadoop-yarn-site in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 289m 46s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
[GitHub] [hadoop] hadoop-yetus commented on pull request #3333: YARN-10576. Update Capacity Scheduler documentation about JSON-based …
hadoop-yetus commented on pull request #:
URL: https://github.com/apache/hadoop/pull/#issuecomment-905681089
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 5s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 50m 6s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 69m 44s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/ |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint |
| uname | Linux 3a4550ff8c7c 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 75e929ec824a747ca807667f86d7935ccb7797ed |
| Max. process+thread count | 518 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #3328: HDFS-16183. RBF: Delete unnecessary metric of proxyOpComplete in routerRpcClient
goiri commented on a change in pull request #3328:
URL: https://github.com/apache/hadoop/pull/3328#discussion_r695904216
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java
##
@@ -517,7 +517,6 @@ private Object invokeMethod(
// Communication retries are handled by the retry policy
if (this.rpcMonitor != null) {
this.rpcMonitor.proxyOpFailureCommunicate();
-this.rpcMonitor.proxyOpComplete(false);
Review comment:
True, I didn't realize that the proxyOpComplete just skips this.
We would need to log the failures here though.
As it is the success flag doesn't make sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] brumi1024 opened a new pull request #3333: YARN-10576. Update Capacity Scheduler documentation about JSON-based …
brumi1024 opened a new pull request #: URL: https://github.com/apache/hadoop/pull/ …placement mapping ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
hadoop-yetus commented on pull request #3200:
URL: https://github.com/apache/hadoop/pull/3200#issuecomment-905580662
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 10s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 9 new or modified test files. |
_ branch-3.2 Compile Tests _ |
| +0 :ok: | mvndep | 3m 18s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 29s | | branch-3.2 passed |
| +1 :green_heart: | compile | 15m 28s | | branch-3.2 passed |
| +1 :green_heart: | checkstyle | 2m 42s | | branch-3.2 passed |
| +1 :green_heart: | mvnsite | 2m 32s | | branch-3.2 passed |
| +1 :green_heart: | javadoc | 2m 8s | | branch-3.2 passed |
| +1 :green_heart: | spotbugs | 5m 8s | | branch-3.2 passed |
| +1 :green_heart: | shadedclient | 16m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 56s | | the patch passed |
| +1 :green_heart: | compile | 14m 55s | | the patch passed |
| -1 :x: | javac | 14m 55s |
[/results-compile-javac-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/4/artifact/out/results-compile-javac-root.txt)
| root generated 1 new + 1587 unchanged - 1 fixed = 1588 total (was 1588) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 37s | | root: The patch generated
0 new + 568 unchanged - 1 fixed = 568 total (was 569) |
| +1 :green_heart: | mvnsite | 2m 28s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 1s | | the patch passed |
| +1 :green_heart: | spotbugs | 5m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 40s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 15m 41s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 198m 18s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| +1 :green_heart: | asflicense | 0m 53s | | The patch does not
generate ASF License warnings. |
| | | 343m 54s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3200 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 25db101e175e 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.2 / 9516ddf5488d512bd55f539363b0befe575b893a |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/4/testReport/ |
| Max. process+thread count | 1991 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/4/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail:
[GitHub] [hadoop] hadoop-yetus commented on pull request #3320: HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Hetero
hadoop-yetus commented on pull request #3320:
URL: https://github.com/apache/hadoop/pull/3320#issuecomment-905536745
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 13s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 5s | | trunk passed |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 41s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 20s | | the patch passed |
| +1 :green_heart: | compile | 1m 26s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 1m 20s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 58s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3320/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 153 unchanged
- 2 fixed = 155 total (was 155) |
| +1 :green_heart: | mvnsite | 1m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 26s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 38s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 344m 21s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3320/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 440m 16s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3320/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3320 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 90ed77414ee5 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 53e9dcb4e3e96a55627a7c8c2825c5e575c52d1e |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3320/2/testReport/ |
| Max. process+thread count | 2179 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Work logged] (HADOOP-17833) Improve Magic Committer Performance
[
https://issues.apache.org/jira/browse/HADOOP-17833?focusedWorklogId=641680=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641680
]
ASF GitHub Bot logged work on HADOOP-17833:
---
Author: ASF GitHub Bot
Created on: 25/Aug/21 13:25
Start Date: 25/Aug/21 13:25
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3289:
URL: https://github.com/apache/hadoop/pull/3289#issuecomment-904031779
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 12s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 20 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 55s | | trunk passed |
| +1 :green_heart: | compile | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 5s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/5/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt)
| hadoop-tools/hadoop-aws: The patch generated 5 new + 25 unchanged - 1 fixed
= 30 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 14s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 24s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/5/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
|
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 2 new +
63 unchanged - 0 fixed = 65 total (was 63) |
| +1 :green_heart: | spotbugs | 1m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 56s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 30s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 89m 47s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3289 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 0aac55373e8b 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e3dc6e416309b07ae015f5ec7c07a5fb0784cc5d |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
|
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3289: HADOOP-17833. Improve Magic Committer performance
hadoop-yetus removed a comment on pull request #3289:
URL: https://github.com/apache/hadoop/pull/3289#issuecomment-904031779
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 12s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 20 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 55s | | trunk passed |
| +1 :green_heart: | compile | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 5s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/5/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt)
| hadoop-tools/hadoop-aws: The patch generated 5 new + 25 unchanged - 1 fixed
= 30 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 14s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 24s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/5/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
|
hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 2 new +
63 unchanged - 0 fixed = 65 total (was 63) |
| +1 :green_heart: | spotbugs | 1m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 56s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 30s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 89m 47s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3289 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 0aac55373e8b 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e3dc6e416309b07ae015f5ec7c07a5fb0784cc5d |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/5/testReport/ |
| Max. process+thread count | 522 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console
[jira] [Work logged] (HADOOP-17863) ABFS: Fix compiler deprecation warning in TextFileBasedIdentityHandler
[
https://issues.apache.org/jira/browse/HADOOP-17863?focusedWorklogId=641675=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641675
]
ASF GitHub Bot logged work on HADOOP-17863:
---
Author: ASF GitHub Bot
Created on: 25/Aug/21 13:12
Start Date: 25/Aug/21 13:12
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3332:
URL: https://github.com/apache/hadoop/pull/3332#issuecomment-905489502
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 15s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 21s | | trunk passed |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 26s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 32s | |
hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 0 new + 26 unchanged - 1 fixed
= 26 total (was 27) |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 27s | |
hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 0 new
+ 20 unchanged - 1 fixed = 20 total (was 21) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 5s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 82m 14s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3332/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3332 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 90ede46dceaf 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 737c7f3a3f669a8e9b2aaa681012c5554923c81e |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test
[GitHub] [hadoop] hadoop-yetus commented on pull request #3332: HADOOP-17863. ABFS: Fix compiler deprecation warning in TextFileBasedIdentityHandler
hadoop-yetus commented on pull request #3332:
URL: https://github.com/apache/hadoop/pull/3332#issuecomment-905489502
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 15s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 21s | | trunk passed |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 26s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 32s | |
hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 0 new + 26 unchanged - 1 fixed
= 26 total (was 27) |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 27s | |
hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 0 new
+ 20 unchanged - 1 fixed = 20 total (was 21) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 5s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 82m 14s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3332/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3332 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 90ede46dceaf 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 737c7f3a3f669a8e9b2aaa681012c5554923c81e |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3332/1/testReport/ |
| Max. process+thread count | 520 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3332/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT
[jira] [Commented] (HADOOP-14693) Upgrade JUnit from 4 to 5
[ https://issues.apache.org/jira/browse/HADOOP-14693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17404429#comment-17404429 ] Akira Ajisaka commented on HADOOP-14693: If we are going to create JUnit 5 based KerberosSecurityTestCase, I think it should be placed in the test directory. > Upgrade JUnit from 4 to 5 > - > > Key: HADOOP-14693 > URL: https://issues.apache.org/jira/browse/HADOOP-14693 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > JUnit 4 does not support Java 9. We need to upgrade this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14693) Upgrade JUnit from 4 to 5
[ https://issues.apache.org/jira/browse/HADOOP-14693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17404427#comment-17404427 ] Akira Ajisaka commented on HADOOP-14693: Hi [~smeng] I appreciated that you are trying to upgrade JUnit in [https://github.com/apache/hadoop/pull/3316] I tried the plugin in some other modules and found a problem: * KerberosSecurityTestCase in hadoop-minikdc module has JUnit 4 dependency in compile (not test) scope, and it is used from many modules. Therefore we need to upgrade all the unit tests that extend KerberosSecurityTestCase at once. To mitigate the situation, we can create JUnit 5 based KerberosSecurityTestCase so that we don't need to upgrade the unit tests at the same time. There are many other test utils, and most of them have the same problem. Hi [~ste...@apache.org] I'm +1 for gradually moving to AssertJ. The error message in AssertJ is more verbose and it has many useful methods. > Upgrade JUnit from 4 to 5 > - > > Key: HADOOP-14693 > URL: https://issues.apache.org/jira/browse/HADOOP-14693 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > JUnit 4 does not support Java 9. We need to upgrade this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a change in pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
aajisaka commented on a change in pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#discussion_r695701899
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java
##
@@ -329,6 +340,25 @@ private void setNumberThreads(String numberThreadsString) {
}
}
+private void setThreadPoolQueueSize(String numThreadPoolQueueSize) {
+ if (numThreadPoolQueueSize != null) {
+int parsedValue = Integer.parseInt(numThreadPoolQueueSize);
+if (parsedValue < 1) {
+ LOG.warn("The value of the thread pool queue size cannot be " +
+ "less than 1, and the default value is used here. " +
+ "The default size is 1024.");
+ threadPoolQueueSize = 1024;
+} else {
+ threadPoolQueueSize = parsedValue;
+}
+ }
+}
+
+@VisibleForTesting
+public int getThreadPoolQueueSize() {
Review comment:
This method can be package-private
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15939) Filter overlapping objenesis class in hadoop-client-minicluster
[
https://issues.apache.org/jira/browse/HADOOP-15939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-15939:
---
Fix Version/s: 3.2.3
Backported to branch-3.2 and branch-3.2.3.
> Filter overlapping objenesis class in hadoop-client-minicluster
>
>
> Key: HADOOP-15939
> URL: https://issues.apache.org/jira/browse/HADOOP-15939
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Minor
> Fix For: 3.3.0, 3.2.3
>
> Attachments: HADOOP-15939.001.patch
>
>
> As mentioned here and found in with latest Jenkins shadedclient.
> Jenkins does not provide a detailed output file for the failure though. But
> it can be reproed with the following cmd:
> {code:java}
> mvn verify -fae --batch-mode -am -pl
> hadoop-client-modules/hadoop-client-check-invariants -pl
> hadoop-client-modules/hadoop-client-check-test-invariants -pl
> hadoop-client-modules/hadoop-client-integration-tests -Dtest=NoUnitTests
> -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true
> {code}
> Error Message:
> {code:java}
> [WARNING] objenesis-1.0.jar, mockito-all-1.8.5.jar define 30 overlapping
> classes:
> [WARNING] - org.objenesis.ObjenesisBase
> [WARNING] - org.objenesis.instantiator.gcj.GCJInstantiator
> [WARNING] - org.objenesis.ObjenesisHelper
> [WARNING] - org.objenesis.instantiator.jrockit.JRockitLegacyInstantiator
> [WARNING] - org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator
> [WARNING] - org.objenesis.instantiator.ObjectInstantiator
> [WARNING] - org.objenesis.instantiator.gcj.GCJInstantiatorBase$DummyStream
> [WARNING] - org.objenesis.instantiator.basic.ObjectStreamClassInstantiator
> [WARNING] - org.objenesis.ObjenesisException
> [WARNING] - org.objenesis.Objenesis
> [WARNING] - 20 more...
> [WARNING] maven-shade-plugin has detected that some class files are
> [WARNING] present in two or more JARs. When this happens, only one
> [WARNING] single version of the class is copied to the uber jar.
> [WARNING] Usually this is not harmful and you can skip these warnings,
> [WARNING] otherwise try to manually exclude artifacts based on
> [WARNING] mvn dependency:tree -Ddetail=true and the above output.
> [WARNING] See [http://maven.apache.org/plugins/maven-shade-plugin/]
> [INFO] Replacing original artifact with shaded artifact.
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17864) ABFS: Fork AbfsHttpOperation to add alternate connection
Sneha Vijayarajan created HADOOP-17864: -- Summary: ABFS: Fork AbfsHttpOperation to add alternate connection Key: HADOOP-17864 URL: https://issues.apache.org/jira/browse/HADOOP-17864 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.4.0 Reporter: Sneha Vijayarajan Assignee: Sneha Vijayarajan This Jira is to facilitate upcoming work as part of adding an alternate connection : [HADOOP-17853] ABFS: Enable optional store connectivity over azure specific protocol for data egress - ASF JIRA (apache.org) The scope of the change is to make AbfsHttpOperation as abstract class and create a child class AbfsHttpConnection. Future connection types will be added as child of AbfsHttpOperation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bbeaudreault commented on a change in pull request #3271: HDFS-16155: Allow configurable exponential backoff in DFSInputStream refetchLocations
bbeaudreault commented on a change in pull request #3271:
URL: https://github.com/apache/hadoop/pull/3271#discussion_r695678157
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/client/impl/TestFetchBlockLocationsRetryer.java
##
@@ -0,0 +1,107 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.client.impl;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.client.HdfsClientConfigKeys;
+import org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.Retry;
+import
org.apache.hadoop.hdfs.client.impl.DfsClientConf.FetchBlockLocationsRetryer;
+import org.junit.Test;
+
+public class TestFetchBlockLocationsRetryer {
+
+ private static final double EPSILON = 0.001;
+
+ @Test
+ public void testIsMaxFailuresExceeded() {
+Configuration conf = new Configuration();
+
+
conf.setInt(HdfsClientConfigKeys.DFS_CLIENT_MAX_BLOCK_ACQUIRE_FAILURES_KEY, 3);
+FetchBlockLocationsRetryer retryer = new FetchBlockLocationsRetryer(conf);
+
+assertFalse(retryer.isMaxFailuresExceeded(1));
+assertTrue(retryer.isMaxFailuresExceeded(3));
+assertTrue(retryer.isMaxFailuresExceeded(5));
+ }
+
+ @Test
+ public void testDefaultRetryPolicy() {
Review comment:
Per the comment in the original backoff policy:
> // Introducing a random factor to the wait time before another retry.
> // The wait time is dependent on # of failures and a random factor.
> // At the first time of getting a BlockMissingException, the wait
time
> // is a random number between 0..3000 ms. If the first retry
> // still fails, we will wait 3000 ms grace period before the 2nd
retry.
> // Also at the second retry, the waiting window is expanded to 6000
ms
> // alleviating the request rate from the server. Similarly the 3rd
retry
> // will wait 6000ms grace period before retry and the waiting window
is
> // expanded to 9000ms.
- The first backoff should be between 0-3000ms.
- The second should be 3000 plus a random number between 0-6000ms. So the
full range is 3000-9000.
- The third retry should be 6000 plus a random number between 0-9000ms. So
the full range is 6000-15000ms.
This test proves that this original retry strategy continues to work with
the new code. It's hard to test with randomness, so the random factor is
disabled. We're left with only the worst case scenario (if `rand()` returned
1). See the assertions below to see that the results adhere to the original
description above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bbeaudreault commented on a change in pull request #3271: HDFS-16155: Allow configurable exponential backoff in DFSInputStream refetchLocations
bbeaudreault commented on a change in pull request #3271:
URL: https://github.com/apache/hadoop/pull/3271#discussion_r695678157
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/client/impl/TestFetchBlockLocationsRetryer.java
##
@@ -0,0 +1,107 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.client.impl;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.client.HdfsClientConfigKeys;
+import org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.Retry;
+import
org.apache.hadoop.hdfs.client.impl.DfsClientConf.FetchBlockLocationsRetryer;
+import org.junit.Test;
+
+public class TestFetchBlockLocationsRetryer {
+
+ private static final double EPSILON = 0.001;
+
+ @Test
+ public void testIsMaxFailuresExceeded() {
+Configuration conf = new Configuration();
+
+
conf.setInt(HdfsClientConfigKeys.DFS_CLIENT_MAX_BLOCK_ACQUIRE_FAILURES_KEY, 3);
+FetchBlockLocationsRetryer retryer = new FetchBlockLocationsRetryer(conf);
+
+assertFalse(retryer.isMaxFailuresExceeded(1));
+assertTrue(retryer.isMaxFailuresExceeded(3));
+assertTrue(retryer.isMaxFailuresExceeded(5));
+ }
+
+ @Test
+ public void testDefaultRetryPolicy() {
Review comment:
Per the comment in the original backoff policy:
> // Introducing a random factor to the wait time before another retry.
> // The wait time is dependent on # of failures and a random factor.
> // At the first time of getting a BlockMissingException, the wait
time
> // is a random number between 0..3000 ms. If the first retry
> // still fails, we will wait 3000 ms grace period before the 2nd
retry.
> // Also at the second retry, the waiting window is expanded to 6000
ms
> // alleviating the request rate from the server. Similarly the 3rd
retry
> // will wait 6000ms grace period before retry and the waiting window
is
> // expanded to 9000ms.
The first backoff should be between 0-3000ms. The second should be between
3000ms-9000ms (3000 + 6000). The third retry should be between 6000-15000ms
(6000 + 9000).
The random factor is applied to the 2nd part of that. So the second retry is
`3000 + rand() * 6000`, etc.
This test proves that this original retry strategy continues to work with
the new code. It's hard to test with randomness, so the random factor is
disabled. We're left with only the worst case scenario (if `rand()` returned
1). See the assertions below to see that the results adhere to the original
description above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17863) ABFS: Fix compiler deprecation warning in TextFileBasedIdentityHandler
[ https://issues.apache.org/jira/browse/HADOOP-17863?focusedWorklogId=641628=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641628 ] ASF GitHub Bot logged work on HADOOP-17863: --- Author: ASF GitHub Bot Created on: 25/Aug/21 11:48 Start Date: 25/Aug/21 11:48 Worklog Time Spent: 10m Work Description: sumangala-patki opened a new pull request #3332: URL: https://github.com/apache/hadoop/pull/3332 TextFileBasedIdentityHandler uses an instance of LineIterator whose closeQuietly method has been deprecated, resulting in compiler warnings during yetus runs. This PR fixes it by leveraging a try-with-resources block to avoid the explicit call to closeQuietly by the LineIterator instance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 641628) Remaining Estimate: 0h Time Spent: 10m > ABFS: Fix compiler deprecation warning in TextFileBasedIdentityHandler > -- > > Key: HADOOP-17863 > URL: https://issues.apache.org/jira/browse/HADOOP-17863 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > TextFileBasedIdentityHandler uses an instance of LineIterator whose > closeQuietly method has been deprecated, resulting in compiler warnings > during yetus runs. Fix by leveraging a try-with-resources block to avoid the > explicit call to closeQuietly by the LineIterator instance. > hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TextFileBasedIdentityHandler.java:192:18:[deprecation] > closeQuietly(LineIterator) in LineIterator has been deprecated -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17863) ABFS: Fix compiler deprecation warning in TextFileBasedIdentityHandler
[ https://issues.apache.org/jira/browse/HADOOP-17863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-17863: Labels: pull-request-available (was: ) > ABFS: Fix compiler deprecation warning in TextFileBasedIdentityHandler > -- > > Key: HADOOP-17863 > URL: https://issues.apache.org/jira/browse/HADOOP-17863 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > TextFileBasedIdentityHandler uses an instance of LineIterator whose > closeQuietly method has been deprecated, resulting in compiler warnings > during yetus runs. Fix by leveraging a try-with-resources block to avoid the > explicit call to closeQuietly by the LineIterator instance. > hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TextFileBasedIdentityHandler.java:192:18:[deprecation] > closeQuietly(LineIterator) in LineIterator has been deprecated -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki opened a new pull request #3332: HADOOP-17863. ABFS: Fix compiler deprecation warning in TextFileBasedIdentityHandler
sumangala-patki opened a new pull request #3332: URL: https://github.com/apache/hadoop/pull/3332 TextFileBasedIdentityHandler uses an instance of LineIterator whose closeQuietly method has been deprecated, resulting in compiler warnings during yetus runs. This PR fixes it by leveraging a try-with-resources block to avoid the explicit call to closeQuietly by the LineIterator instance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17862) ABFS: Fix unchecked cast compiler warning for AbfsListStatusRemoteIterator
[ https://issues.apache.org/jira/browse/HADOOP-17862?focusedWorklogId=641625=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641625 ] ASF GitHub Bot logged work on HADOOP-17862: --- Author: ASF GitHub Bot Created on: 25/Aug/21 11:43 Start Date: 25/Aug/21 11:43 Worklog Time Spent: 10m Work Description: sumangala-patki commented on pull request #3331: URL: https://github.com/apache/hadoop/pull/3331#issuecomment-905424970 TEST RESULTS HNS Account Location: East US 2 NonHNS Account Location: East US 2, Central US ``` AppendBlob HNS OAuth [WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 1 [ERROR] Failures: [ERROR] ITestAbfsStreamStatistics.testAbfsStreamOps:140->Assert.assertTrue:42->Assert.fail:89 The actual value of 99 was not equal to the expected value [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendNoInfiniteLease:178->twoWriters:166 » IO [ERROR] Tests run: 558, Failures: 1, Errors: 1, Skipped: 98 [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads:77 » TestTimedOut [ERROR] ITestSmallWriteOptimization>AbstractAbfsScaleTest.setup:49->AbstractAbfsIntegrationTest.setup:174->AbstractAbfsIntegrationTest.createFileSystem:303 » FileAlreadyExists [ERROR] Tests run: 259, Failures: 0, Errors: 2, Skipped: 75 HNS OAuth [ERROR] Failures: [ERROR] TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 The actual value 14 is not within the expected range: [5.60, 8.40]. [ERROR] Tests run: 106, Failures: 1, Errors: 0, Skipped: 1 [WARNING] Tests run: 558, Failures: 0, Errors: 0, Skipped: 98 [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemBasics>FileSystemContractBaseTest.testRenameDirectoryMoveToExistingDirectory:526 » TestTimedOut [ERROR] Tests run: 259, Failures: 0, Errors: 1, Skipped: 52 HNS SharedKey [WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 2 [WARNING] Tests run: 558, Failures: 0, Errors: 0, Skipped: 54 [WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 40 NonHNS SharedKey [WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 2 [ERROR] Failures: [ERROR] ITestAbfsRestOperationException.testCustomTokenFetchRetryCount:93->testWithDifferentCustomTokenFetchRetry:122->Assert.assertTrue:42->Assert.fail:89 Number of token fetch retries (4) done, does not match with fs.azure.custom.token.fetch.retry.count configured (0) [ERROR] Tests run: 558, Failures: 1, Errors: 0, Skipped: 276 [WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 40 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 641625) Time Spent: 50m (was: 40m) > ABFS: Fix unchecked cast compiler warning for AbfsListStatusRemoteIterator > -- > > Key: HADOOP-17862 > URL: https://issues.apache.org/jira/browse/HADOOP-17862 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Hadoop yetus run shows a java compiler warning for unchecked casting of > Object to Iterator in a method of AbfsListStatusRemoteIterator > class. This can be resolved by introducing a new class to hold the iterator > and exception thrown when applicable. > > hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsListStatusRemoteIterator.java:97:38:[unchecked] > unchecked cast > > This task will also add logging to facilitate better investigation of the > transient failure tracked in HADOOP-17797. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on pull request #3331: HADOOP-17862. ABFS: Fix unchecked cast compiler warning for AbfsListStatusRemoteIterator
sumangala-patki commented on pull request #3331: URL: https://github.com/apache/hadoop/pull/3331#issuecomment-905424970 TEST RESULTS HNS Account Location: East US 2 NonHNS Account Location: East US 2, Central US ``` AppendBlob HNS OAuth [WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 1 [ERROR] Failures: [ERROR] ITestAbfsStreamStatistics.testAbfsStreamOps:140->Assert.assertTrue:42->Assert.fail:89 The actual value of 99 was not equal to the expected value [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendNoInfiniteLease:178->twoWriters:166 » IO [ERROR] Tests run: 558, Failures: 1, Errors: 1, Skipped: 98 [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemE2EScale.testWriteHeavyBytesToFileAcrossThreads:77 » TestTimedOut [ERROR] ITestSmallWriteOptimization>AbstractAbfsScaleTest.setup:49->AbstractAbfsIntegrationTest.setup:174->AbstractAbfsIntegrationTest.createFileSystem:303 » FileAlreadyExists [ERROR] Tests run: 259, Failures: 0, Errors: 2, Skipped: 75 HNS OAuth [ERROR] Failures: [ERROR] TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 The actual value 14 is not within the expected range: [5.60, 8.40]. [ERROR] Tests run: 106, Failures: 1, Errors: 0, Skipped: 1 [WARNING] Tests run: 558, Failures: 0, Errors: 0, Skipped: 98 [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemBasics>FileSystemContractBaseTest.testRenameDirectoryMoveToExistingDirectory:526 » TestTimedOut [ERROR] Tests run: 259, Failures: 0, Errors: 1, Skipped: 52 HNS SharedKey [WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 2 [WARNING] Tests run: 558, Failures: 0, Errors: 0, Skipped: 54 [WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 40 NonHNS SharedKey [WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 2 [ERROR] Failures: [ERROR] ITestAbfsRestOperationException.testCustomTokenFetchRetryCount:93->testWithDifferentCustomTokenFetchRetry:122->Assert.assertTrue:42->Assert.fail:89 Number of token fetch retries (4) done, does not match with fs.azure.custom.token.fetch.retry.count configured (0) [ERROR] Tests run: 558, Failures: 1, Errors: 0, Skipped: 276 [WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 40 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17863) ABFS: Fix compiler deprecation warning in TextFileBasedIdentityHandler
Sumangala Patki created HADOOP-17863: Summary: ABFS: Fix compiler deprecation warning in TextFileBasedIdentityHandler Key: HADOOP-17863 URL: https://issues.apache.org/jira/browse/HADOOP-17863 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.1 Reporter: Sumangala Patki Assignee: Sumangala Patki TextFileBasedIdentityHandler uses an instance of LineIterator whose closeQuietly method has been deprecated, resulting in compiler warnings during yetus runs. Fix by leveraging a try-with-resources block to avoid the explicit call to closeQuietly by the LineIterator instance. hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TextFileBasedIdentityHandler.java:192:18:[deprecation] closeQuietly(LineIterator) in LineIterator has been deprecated -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17862) ABFS: Fix unchecked cast compiler warning for AbfsListStatusRemoteIterator
[
https://issues.apache.org/jira/browse/HADOOP-17862?focusedWorklogId=641622=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641622
]
ASF GitHub Bot logged work on HADOOP-17862:
---
Author: ASF GitHub Bot
Created on: 25/Aug/21 11:33
Start Date: 25/Aug/21 11:33
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3331:
URL: https://github.com/apache/hadoop/pull/3331#issuecomment-905419314
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 30s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 27m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 0m 38s |
[/results-compile-javac-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/3/artifact/out/results-compile-javac-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with
JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 24 unchanged - 1
fixed = 25 total (was 25) |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 25s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 6s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 97m 7s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3331 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux bce30df808f8 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 98fb5bc8d7b299f224ab29a2195c9e7957eb087a |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
[GitHub] [hadoop] hadoop-yetus commented on pull request #3331: HADOOP-17862. ABFS: Fix unchecked cast compiler warning for AbfsListStatusRemoteIterator
hadoop-yetus commented on pull request #3331:
URL: https://github.com/apache/hadoop/pull/3331#issuecomment-905419314
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 30s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 27m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 0m 38s |
[/results-compile-javac-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/3/artifact/out/results-compile-javac-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with
JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 24 unchanged - 1
fixed = 25 total (was 25) |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 25s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 6s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 97m 7s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3331 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux bce30df808f8 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 98fb5bc8d7b299f224ab29a2195c9e7957eb087a |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/3/testReport/ |
| Max. process+thread count | 589 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was
[GitHub] [hadoop] bbeaudreault commented on a change in pull request #3271: HDFS-16155: Allow configurable exponential backoff in DFSInputStream refetchLocations
bbeaudreault commented on a change in pull request #3271: URL: https://github.com/apache/hadoop/pull/3271#discussion_r695662268 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml ## @@ -4440,8 +4440,29 @@ 3000 Base time window in ms for DFSClient retries. For each retry attempt, -this value is extended linearly (e.g. 3000 ms for first attempt and -first retry, 6000 ms for second retry, 9000 ms for third retry, etc.). +this value is extended exponentially based on dfs.client.retry.window.multiplier. + + + + + dfs.client.retry.window.multiplier + 1 + +Multiplier for extending the retry time window. For each retry attempt, +the retry time window is extended by multiplying dfs.client.retry.window.base +by this multiplier raised to the power of the current failure count. The default +value of 1 means the window will expand linearly (e.g. 3000 ms for first attempt +and first retry, 6000 ms for second retry, 9000 ms for third retry, etc.). + + + + + dfs.client.retry.window.max + 2147483647 Review comment: Thanks so much for the review @Hexiaoqiao. The reason I chose this value was that I wanted the changes in this PR to be totally transparent to existing users -- so the backoff should work exactly as it does today for anyone who upgrades. I don't know how people have tuned their backoffs today, so adding a lower max might affect their configured backoffs. The default case will be well-bounded by the default retries of 3. That said, I agree that there's very little utility in waiting many minutes on a backoff. What if I put this to 30s? Was this the only concern in terms of the default action? My test case [testDefaultRetryPolicy](https://github.com/apache/hadoop/pull/3271/files#diff-ce7234e2a352097945c5348774888466b48f9574d4764d9415559d975d380884R47) proves that the default case remains unchanged from trunk. The default case was determined based on the [comment in DFSInputStream](https://github.com/apache/hadoop/pull/3271/files#diff-f6cd7fbb9a909e1679c2d77df54b0b2fe661a66c8c89fb455fde313966d51a61L1018-L1026), the old implementation details, and my own testing of the backoff policy prior to this change. I also created this spreadsheet that helped me to determine how different multiplier values might affect the backoff: https://docs.google.com/spreadsheets/d/1I9ejqDtJ6-krSh-YBt0qHTf3JwZu5zRlrOhbzY0kJAg/edit?usp=sharing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3314: YARN-10891. Extend QueueInfo with max-parallel-apps in CS.
hadoop-yetus commented on pull request #3314:
URL: https://github.com/apache/hadoop/pull/3314#issuecomment-905414155
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 7s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 43s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 33s | | trunk passed |
| +1 :green_heart: | compile | 12m 6s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 8m 32s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 16s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 28s | |
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file
(spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 16m 29s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 45s | | the patch passed |
| +1 :green_heart: | compile | 9m 28s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | cc | 9m 28s | | the patch passed |
| +1 :green_heart: | javac | 9m 28s | | the patch passed |
| +1 :green_heart: | compile | 8m 31s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | cc | 8m 31s |
[/results-compile-cc-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3314/7/artifact/out/results-compile-cc-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
|
hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new
+ 181 unchanged - 1 fixed = 182 total (was 182) |
| +1 :green_heart: | javac | 8m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 38s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3314/7/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 4 new + 628 unchanged
- 0 fixed = 632 total (was 628) |
| +1 :green_heart: | mvnsite | 3m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 9s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 0s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 25s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs |
| +1 :green_heart: | shadedclient | 16m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 58s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 4m 42s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 99m 37s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 27m 19s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | unit | 0m 27s | | hadoop-yarn-site in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 294m 3s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
[jira] [Commented] (HADOOP-17847) S3AInstrumentation Closing output stream statistics while data is still marked as pending upload in OutputStreamStatistics
[
https://issues.apache.org/jira/browse/HADOOP-17847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17404352#comment-17404352
]
Chris Nix commented on HADOOP-17847:
Not wishing to divert the original report regarding spark, but I thought it
worth noting that I'm also getting this error bubbling up from Flink 1.8's
`StreamingFileSink` when using an s3a url. I think flink 1.8 is using hadoop
2.4.1. In this Flink case, I believe it might occur sometimes as the sink's
`RollingPolicy` rotates output files periodically.
We're streaming avro from Kafka into parquet files on S3. I've verified that we
have a small proportion of avro messages on Kafka that have no corresponding
parquet rows on S3. This might imply it's more than just an instrumentation
thing.
> S3AInstrumentation Closing output stream statistics while data is still
> marked as pending upload in OutputStreamStatistics
> --
>
> Key: HADOOP-17847
> URL: https://issues.apache.org/jira/browse/HADOOP-17847
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.2.1
> Environment: hadoop: 3.2.1
> spark: 3.0.2
> k8s server version: 1.18
> aws.java.sdk.bundle.version:1.11.1033
>Reporter: Li Rong
>Priority: Minor
> Attachments: logs.txt
>
>
> When using hadoop s3a file upload for spark event Logs, the logs were queued
> up and not uploaded before the process is shut down:
> {code:java}
> // 21/08/13 12:22:39 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client
> has been closed (this is expected if the application is shutting down.)
> 21/08/13 12:22:39 WARN S3AInstrumentation: Closing output stream statistics
> while data is still marked as pending upload in
> OutputStreamStatistics{blocksSubmitted=1, blocksInQueue=1, blocksActive=0,
> blockUploadsCompleted=0, blockUploadsFailed=0, bytesPendingUpload=106716,
> bytesUploaded=0, blocksAllocated=1, blocksReleased=1,
> blocksActivelyAllocated=0, exceptionsInMultipartFinalize=0,
> transferDuration=0 ms, queueDuration=0 ms, averageQueueTime=0 ms,
> totalUploadDuration=0 ms, effectiveBandwidth=0.0 bytes/s}{code}
> details see logs attached
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on a change in pull request #3271: HDFS-16155: Allow configurable exponential backoff in DFSInputStream refetchLocations
Hexiaoqiao commented on a change in pull request #3271: URL: https://github.com/apache/hadoop/pull/3271#discussion_r695628043 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml ## @@ -4440,8 +4440,29 @@ 3000 Base time window in ms for DFSClient retries. For each retry attempt, -this value is extended linearly (e.g. 3000 ms for first attempt and -first retry, 6000 ms for second retry, 9000 ms for third retry, etc.). +this value is extended exponentially based on dfs.client.retry.window.multiplier. + + + + + dfs.client.retry.window.multiplier + 1 + +Multiplier for extending the retry time window. For each retry attempt, +the retry time window is extended by multiplying dfs.client.retry.window.base +by this multiplier raised to the power of the current failure count. The default +value of 1 means the window will expand linearly (e.g. 3000 ms for first attempt +and first retry, 6000 ms for second retry, 9000 ms for third retry, etc.). + + + + + dfs.client.retry.window.max + 2147483647 Review comment: The default value for `dfs.client.retry.window.max` is too high here. In some corner case, it will sleep very long time? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17861) improve YARN Registry DNS Server qps
[
https://issues.apache.org/jira/browse/HADOOP-17861?focusedWorklogId=641608=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641608
]
ASF GitHub Bot logged work on HADOOP-17861:
---
Author: ASF GitHub Bot
Created on: 25/Aug/21 10:43
Start Date: 25/Aug/21 10:43
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3326:
URL: https://github.com/apache/hadoop/pull/3326#issuecomment-905386948
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 45s | | trunk passed |
| +1 :green_heart: | compile | 24m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 32s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 23m 27s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 27s | | the patch passed |
| +1 :green_heart: | compile | 20m 30s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 20m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 40s |
[/results-checkstyle-hadoop-common-project_hadoop-registry.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3326/2/artifact/out/results-checkstyle-hadoop-common-project_hadoop-registry.txt)
| hadoop-common-project/hadoop-registry: The patch generated 1 new + 2
unchanged - 0 fixed = 3 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | spotbugs | 1m 20s |
[/new-spotbugs-hadoop-common-project_hadoop-registry.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3326/2/artifact/out/new-spotbugs-hadoop-common-project_hadoop-registry.html)
| hadoop-common-project/hadoop-registry generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 16m 16s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | | hadoop-registry in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 168m 44s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-common-project/hadoop-registry |
| | Exceptional return value of
java.util.concurrent.ExecutorService.submit(Callable) ignored in
org.apache.hadoop.registry.server.dns.RegistryDNS.serveNIOUDP(DatagramChannel)
At RegistryDNS.java:ignored in
org.apache.hadoop.registry.server.dns.RegistryDNS.serveNIOUDP(DatagramChannel)
At RegistryDNS.java:[line 1025] |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3326/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3326 |
| Optional Tests | dupname
[GitHub] [hadoop] hadoop-yetus commented on pull request #3326: HADOOP-17861. improve YARN Registry DNS Server qps
hadoop-yetus commented on pull request #3326:
URL: https://github.com/apache/hadoop/pull/3326#issuecomment-905386948
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 45s | | trunk passed |
| +1 :green_heart: | compile | 24m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 32s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 23m 27s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 27s | | the patch passed |
| +1 :green_heart: | compile | 20m 30s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 20m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 40s |
[/results-checkstyle-hadoop-common-project_hadoop-registry.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3326/2/artifact/out/results-checkstyle-hadoop-common-project_hadoop-registry.txt)
| hadoop-common-project/hadoop-registry: The patch generated 1 new + 2
unchanged - 0 fixed = 3 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | spotbugs | 1m 20s |
[/new-spotbugs-hadoop-common-project_hadoop-registry.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3326/2/artifact/out/new-spotbugs-hadoop-common-project_hadoop-registry.html)
| hadoop-common-project/hadoop-registry generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 16m 16s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | | hadoop-registry in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 168m 44s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-common-project/hadoop-registry |
| | Exceptional return value of
java.util.concurrent.ExecutorService.submit(Callable) ignored in
org.apache.hadoop.registry.server.dns.RegistryDNS.serveNIOUDP(DatagramChannel)
At RegistryDNS.java:ignored in
org.apache.hadoop.registry.server.dns.RegistryDNS.serveNIOUDP(DatagramChannel)
At RegistryDNS.java:[line 1025] |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3326/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3326 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 47619be6e376 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9
23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5d6fa72bd0c117c57e3debf817d8acb923d3b687 |
| Default Java | Private
[GitHub] [hadoop] jojochuang commented on a change in pull request #3322: HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS.
jojochuang commented on a change in pull request #3322:
URL: https://github.com/apache/hadoop/pull/3322#discussion_r695375296
##
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
##
@@ -908,4 +911,51 @@ private static SnapshotStatus toSnapshotStatus(
SnapshotStatus.getParentPath(fullPath)));
return snapshotStatus;
}
+
+ static BlockLocation[] toBlockLocationArray(Map json)
Review comment:
TODO: need test
##
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
##
@@ -908,4 +911,51 @@ private static SnapshotStatus toSnapshotStatus(
SnapshotStatus.getParentPath(fullPath)));
return snapshotStatus;
}
+
+ static BlockLocation[] toBlockLocationArray(Map json)
+ throws IOException {
+final Map rootmap =
+(Map) json.get(BlockLocation.class.getSimpleName() + "s");
+final List array =
+JsonUtilClient.getList(rootmap, BlockLocation.class.getSimpleName());
+Preconditions.checkNotNull(array);
+final BlockLocation[] locations = new BlockLocation[array.size()];
+int i = 0;
+for (Object object : array) {
+ final Map m = (Map) object;
+ locations[i++] = JsonUtilClient.toBlockLocation(m);
+}
+return locations;
+ }
+
+ /** Convert a Json map to BlockLocation. **/
+ static BlockLocation toBlockLocation(Map m) throws IOException {
+if (m == null) {
+ return null;
+}
+long length = ((Number) m.get("length")).longValue();
+long offset = ((Number) m.get("offset")).longValue();
+boolean corrupt = Boolean.getBoolean(m.get("corrupt").toString());
+String[] storageIds = toStringArray(getList(m, "storageIds"));
+String[] cachedHosts = toStringArray(getList(m, "cachedHosts"));
+String[] hosts = toStringArray(getList(m, "hosts"));
+String[] names = toStringArray(getList(m, "names"));
+String[] topologyPaths = toStringArray(getList(m, "topologyPaths"));
+StorageType[] storageTypes = toStorageTypeArray(getList(m,
"storageTypes"));
+return new BlockLocation(names, hosts, cachedHosts, topologyPaths,
+storageIds, storageTypes, offset, length, corrupt);
+ }
+
+ static String[] toStringArray(List list) {
Review comment:
VisibleForTesting
##
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
##
@@ -908,4 +911,51 @@ private static SnapshotStatus toSnapshotStatus(
SnapshotStatus.getParentPath(fullPath)));
return snapshotStatus;
}
+
+ static BlockLocation[] toBlockLocationArray(Map json)
+ throws IOException {
+final Map rootmap =
+(Map) json.get(BlockLocation.class.getSimpleName() + "s");
+final List array =
+JsonUtilClient.getList(rootmap, BlockLocation.class.getSimpleName());
+Preconditions.checkNotNull(array);
+final BlockLocation[] locations = new BlockLocation[array.size()];
+int i = 0;
+for (Object object : array) {
+ final Map m = (Map) object;
+ locations[i++] = JsonUtilClient.toBlockLocation(m);
+}
+return locations;
+ }
+
+ /** Convert a Json map to BlockLocation. **/
+ static BlockLocation toBlockLocation(Map m) throws IOException {
Review comment:
need test.
##
File path:
hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java
##
@@ -2002,4 +2003,38 @@ public void testContentType() throws Exception {
() -> HttpFSUtils.jsonParse(conn));
conn.disconnect();
}
+
+ @Test
+ @TestDir
+ @TestJetty
+ @TestHdfs
+ public void testGetFileBlockLocations() throws Exception {
+createHttpFSServer(false, false);
+// Create a test directory
+String pathStr = "/tmp/tmp-snap-diff-test";
+createDirWithHttp(pathStr, "700", null);
+
+Path path = new Path(pathStr);
+DistributedFileSystem dfs = (DistributedFileSystem) FileSystem
+.get(path.toUri(), TestHdfsHelper.getHdfsConf());
+// Enable snapshot
+dfs.allowSnapshot(path);
+Assert.assertTrue(dfs.getFileStatus(path).isSnapshotEnabled());
+// Create a file and take a snapshot
+String file1 = pathStr + "/file1";
+createWithHttp(file1, null);
+HttpURLConnection conn = sendRequestToHttpFSServer(file1,
+"GETFILEBLOCKLOCATIONS", "length=10");
+Assert.assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+BlockLocation[] locations1 =
+dfs.getFileBlockLocations(new Path(file1), 0, 1);
+Assert.assertNotNull(locations1);
+
+HttpURLConnection conn1 = sendRequestToHttpFSServer(file1,
+"GET_BLOCK_LOCATIONS", "length=10");
+Assert.assertEquals(HttpURLConnection.HTTP_OK, conn1.getResponseCode());
+BlockLocation[] locations2 =
+dfs.getFileBlockLocations(new Path(file1), 0, 1);
+
[GitHub] [hadoop] hadoop-yetus commented on pull request #3330: YARN-10522. Document for Flexible Auto Queue Creation in Capacity Scheduler.
hadoop-yetus commented on pull request #3330:
URL: https://github.com/apache/hadoop/pull/3330#issuecomment-905382812
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 9s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 49m 46s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 68m 18s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3330/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3330 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint |
| uname | Linux 1fd267351106 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17
17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 97759a26a42eb445da128c16c1173b95ebb24b98 |
| Max. process+thread count | 611 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3330/3/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on a change in pull request #3320: HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail w
Hexiaoqiao commented on a change in pull request #3320:
URL: https://github.com/apache/hadoop/pull/3320#discussion_r695602966
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
##
@@ -1337,6 +1337,50 @@ public void testChooseSsdOverDisk() throws Exception {
Assert.assertEquals(StorageType.DISK, targets[1].getStorageType());
}
+ @Test
Review comment:
Add some javadoc here will be more friendly here.
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
##
@@ -469,7 +469,7 @@ private Node chooseTarget(int numOfReplicas,
LOG.trace("storageTypes={}", storageTypes);
try {
- if ((numOfReplicas = requiredStorageTypes.size()) == 0) {
Review comment:
Just wonder if this is common bug for every `chooseTarget` invokes when
assign `requiredStorageTypes.size()` to `numOfReplicas` here? or just for set
storagePolicy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
jianghuazhu commented on pull request #3302: URL: https://github.com/apache/hadoop/pull/3302#issuecomment-905351084 Thanks @jojochuang for the comment. I will continue to work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang merged pull request #3307: HDFS-16175.Improve the configurable value of Server #PURGE_INTERVAL_NANOS.
jojochuang merged pull request #3307: URL: https://github.com/apache/hadoop/pull/3307 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on pull request #3307: HDFS-16175.Improve the configurable value of Server #PURGE_INTERVAL_NANOS.
jojochuang commented on pull request #3307: URL: https://github.com/apache/hadoop/pull/3307#issuecomment-905340718 Merging it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
jojochuang commented on pull request #3302: URL: https://github.com/apache/hadoop/pull/3302#issuecomment-905339585 @jianghuazhu could you help verify TestCLI isn't broken by this change? We'll have to verify it manually. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17848) Hadoop NativeAzureFileSystem append removes ownership set on the file
[
https://issues.apache.org/jira/browse/HADOOP-17848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17404300#comment-17404300
]
Prabhu Joseph commented on HADOOP-17848:
[~anoop.hbase]
bq. The call fs.setPermission(filePath, new
FsPermission(FILE_LOG_PERMISSIONS)); is removing the owner/group details?
No. The write call using Append stream is removing the owner/group details.
{code}
stream = fs.append(filePath);
stream.write(888);
{code}
> Hadoop NativeAzureFileSystem append removes ownership set on the file
> -
>
> Key: HADOOP-17848
> URL: https://issues.apache.org/jira/browse/HADOOP-17848
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.1
>Reporter: Prabhu Joseph
>Priority: Major
>
> *Repro:* Create Operation sets ownership whereas append operation removes the
> same.
> Create:
> *// -rw-r--r-- 1 root supergroup 1 2021-08-15 11:02 /tmp/dummyfile*
> Append:
> *// -rwxrwxrwx 1 2 2021-08-15 11:04 /tmp/dummyfile*
> {code:java}
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FSDataOutputStream;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.fs.permission.FsPermission;
> public class Wasb {
> private static final short FILE_LOG_PERMISSIONS = 0640;
>
> public static void main(String[] args) throws Exception {
>
> Configuration fsConf = new Configuration();
> fsConf.set("fs.azure.enable.append.support", "true");
> Path filePath = new Path("/tmp/dummyfile");
> FileSystem fs = FileSystem.newInstance(filePath.toUri(), fsConf);
> FSDataOutputStream stream = fs.create(filePath, false);
> stream.write(12345);
> stream.close();
> stream = fs.append(filePath);
> stream.write(888);
> stream.close();
> fs.close();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17848) Hadoop NativeAzureFileSystem append removes ownership set on the file
[
https://issues.apache.org/jira/browse/HADOOP-17848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated HADOOP-17848:
---
Description:
*Repro:* Create Operation sets ownership whereas append operation removes the
same.
Create:
*// -rw-r--r-- 1 root supergroup 1 2021-08-15 11:02 /tmp/dummyfile*
Append:
*// -rwxrwxrwx 1 2 2021-08-15 11:04 /tmp/dummyfile*
{code:java}
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.permission.FsPermission;
public class Wasb {
private static final short FILE_LOG_PERMISSIONS = 0640;
public static void main(String[] args) throws Exception {
Configuration fsConf = new Configuration();
fsConf.set("fs.azure.enable.append.support", "true");
Path filePath = new Path("/tmp/dummyfile");
FileSystem fs = FileSystem.newInstance(filePath.toUri(), fsConf);
FSDataOutputStream stream = fs.create(filePath, false);
stream.write(12345);
stream.close();
stream = fs.append(filePath);
stream.write(888);
stream.close();
fs.close();
}
}
{code}
was:
*Repro:* Create Operation sets ownership whereas append operation removes the
same.
Create:
*// -rw-r--r-- 1 root supergroup 1 2021-08-15 11:02 /tmp/dummyfile*
Append:
*// -rwxrwxrwx 1 2 2021-08-15 11:04 /tmp/dummyfile*
{code:java}
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.permission.FsPermission;
public class Wasb {
private static final short FILE_LOG_PERMISSIONS = 0640;
public static void main(String[] args) throws Exception {
Configuration fsConf = new Configuration();
fsConf.set("fs.azure.enable.append.support", "true");
Path filePath = new Path("/tmp/dummyfile");
FileSystem fs = FileSystem.newInstance(filePath.toUri(), fsConf);
FSDataOutputStream stream = fs.create(filePath, false);
stream.write(12345);
stream.close();
stream = fs.append(filePath);
stream.write(888);
stream.close();
fs.setPermission(filePath, new FsPermission(FILE_LOG_PERMISSIONS));
fs.close();
}
}
{code}
> Hadoop NativeAzureFileSystem append removes ownership set on the file
> -
>
> Key: HADOOP-17848
> URL: https://issues.apache.org/jira/browse/HADOOP-17848
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.1
>Reporter: Prabhu Joseph
>Priority: Major
>
> *Repro:* Create Operation sets ownership whereas append operation removes the
> same.
> Create:
> *// -rw-r--r-- 1 root supergroup 1 2021-08-15 11:02 /tmp/dummyfile*
> Append:
> *// -rwxrwxrwx 1 2 2021-08-15 11:04 /tmp/dummyfile*
> {code:java}
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FSDataOutputStream;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.fs.permission.FsPermission;
> public class Wasb {
> private static final short FILE_LOG_PERMISSIONS = 0640;
>
> public static void main(String[] args) throws Exception {
>
> Configuration fsConf = new Configuration();
> fsConf.set("fs.azure.enable.append.support", "true");
> Path filePath = new Path("/tmp/dummyfile");
> FileSystem fs = FileSystem.newInstance(filePath.toUri(), fsConf);
> FSDataOutputStream stream = fs.create(filePath, false);
> stream.write(12345);
> stream.close();
> stream = fs.append(filePath);
> stream.write(888);
> stream.close();
> fs.close();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on pull request #3307: HDFS-16175.Improve the configurable value of Server #PURGE_INTERVAL_NANOS.
jianghuazhu commented on pull request #3307: URL: https://github.com/apache/hadoop/pull/3307#issuecomment-905322240 @ayushtkn , thanks for your help. I don't know if this PR request can be merged into the trunk branch. If can, can you help me deal with it? thank you very much. (I'm not sure if I should make such request.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
jianghuazhu commented on pull request #3302: URL: https://github.com/apache/hadoop/pull/3302#issuecomment-905321805 @ayushtkn , thanks for your help. I don't know if this PR request can be merged into the trunk branch. If can, can you help me deal with it? thank you very much. (I'm not sure if I should make such request.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3200: HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.
hadoop-yetus commented on pull request #3200: URL: https://github.com/apache/hadoop/pull/3200#issuecomment-905303834 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 23s | | HDFS-15160 does not apply to branch-3.3. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | JIRA Issue | HDFS-15160 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13022920/HDFS-15160-branch-3.3-001.patch | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/3/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tasanuma commented on pull request #3329: HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile
tasanuma commented on pull request #3329: URL: https://github.com/apache/hadoop/pull/3329#issuecomment-905302618 Merged. Thanks for your contribution, @virajjasani. Thanks for your review, @ayushtkn. > Also, it's good to see two consecutive +1s from QA bot for hadoop-hdfs. Haven't seen more than single +1 from QA build for long time on trunk. That's true. QA results seem pretty good recently. Really thanks for fixing several flaky tests, @virajjasani. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tasanuma merged pull request #3329: HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile
tasanuma merged pull request #3329: URL: https://github.com/apache/hadoop/pull/3329 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17862) ABFS: Fix unchecked cast compiler warning for AbfsListStatusRemoteIterator
[
https://issues.apache.org/jira/browse/HADOOP-17862?focusedWorklogId=641555=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641555
]
ASF GitHub Bot logged work on HADOOP-17862:
---
Author: ASF GitHub Bot
Created on: 25/Aug/21 08:30
Start Date: 25/Aug/21 08:30
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3331:
URL: https://github.com/apache/hadoop/pull/3331#issuecomment-905296286
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 7s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 52s | | trunk passed |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 0m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 16s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/2/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 29s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 56s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 79m 27s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3331 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux c3c8a2d50df9 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e18b91dfd7b1a761f5aef67d14887935aa495173 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/2/testReport/ |
| Max. process+thread count | 592 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
[GitHub] [hadoop] hadoop-yetus commented on pull request #3331: HADOOP-17862. ABFS: Fix unchecked cast compiler warning for AbfsListStatusRemoteIterator
hadoop-yetus commented on pull request #3331:
URL: https://github.com/apache/hadoop/pull/3331#issuecomment-905296286
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 7s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 52s | | trunk passed |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 0m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 16s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/2/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 29s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 56s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 79m 27s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3331 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux c3c8a2d50df9 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e18b91dfd7b1a761f5aef67d14887935aa495173 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/2/testReport/ |
| Max. process+thread count | 592 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above
[GitHub] [hadoop] virajjasani commented on pull request #3329: HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile
virajjasani commented on pull request #3329: URL: https://github.com/apache/hadoop/pull/3329#issuecomment-905248069 Also, it's good to see two consecutive +1s from QA bot for hadoop-hdfs. Haven't seen more than single +1 from QA build for long time on trunk. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #3329: HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile
virajjasani commented on pull request #3329: URL: https://github.com/apache/hadoop/pull/3329#issuecomment-905246751 Thanks to both of you for quick reviews @tasanuma @ayushtkn !! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17862) ABFS: Fix unchecked cast compiler warning for AbfsListStatusRemoteIterator
[
https://issues.apache.org/jira/browse/HADOOP-17862?focusedWorklogId=641533=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641533
]
ASF GitHub Bot logged work on HADOOP-17862:
---
Author: ASF GitHub Bot
Created on: 25/Aug/21 07:15
Start Date: 25/Aug/21 07:15
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3331:
URL: https://github.com/apache/hadoop/pull/3331#issuecomment-905245795
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 52s | | trunk passed |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 0m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 0m 30s |
[/results-compile-javac-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/1/artifact/out/results-compile-javac-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with
JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 24 unchanged - 1
fixed = 25 total (was 25) |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 16s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 2s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 83m 8s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3331 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 6d42215e8c38 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision |
[GitHub] [hadoop] hadoop-yetus commented on pull request #3331: HADOOP-17862. ABFS: Fix unchecked cast compiler warning for AbfsListStatusRemoteIterator
hadoop-yetus commented on pull request #3331:
URL: https://github.com/apache/hadoop/pull/3331#issuecomment-905245795
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 52s | | trunk passed |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 0m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 0m 30s |
[/results-compile-javac-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/1/artifact/out/results-compile-javac-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with
JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 24 unchanged - 1
fixed = 25 total (was 25) |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 16s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 2s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 83m 8s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3331 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 6d42215e8c38 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b3eaa6ceca8cfa2490b6c45d146a9e0c57f23d77 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3331/1/testReport/ |
| Max.
[jira] [Work logged] (HADOOP-17858) Avoid possible class loading deadlock with VerifierNone initialization
[ https://issues.apache.org/jira/browse/HADOOP-17858?focusedWorklogId=641530=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641530 ] ASF GitHub Bot logged work on HADOOP-17858: --- Author: ASF GitHub Bot Created on: 25/Aug/21 07:13 Start Date: 25/Aug/21 07:13 Worklog Time Spent: 10m Work Description: virajjasani commented on pull request #3321: URL: https://github.com/apache/hadoop/pull/3321#issuecomment-905244311 Thank you @tasanuma! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 641530) Time Spent: 1h (was: 50m) > Avoid possible class loading deadlock with VerifierNone initialization > -- > > Key: HADOOP-17858 > URL: https://issues.apache.org/jira/browse/HADOOP-17858 > Project: Hadoop Common > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.2 > > Time Spent: 1h > Remaining Estimate: 0h > > Superclass Verifier has a static initializer VERIFIER_NONE that initializes > sub-class VerifierNone. This reference can result in deadlock during class > loading as per > [https://docs.oracle.com/javase/specs/jls/se8/html/jls-12.html#jls-12.4.2]. > As of today, only RpcProgram use this instance and hence it is safe but if > more clients start using this (specifically static ones), it has potential to > bring deadlock. We should break this referencing before it is late. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #3321: HADOOP-17858. Avoid possible class loading deadlock with VerifierNone initialization
virajjasani commented on pull request #3321: URL: https://github.com/apache/hadoop/pull/3321#issuecomment-905244311 Thank you @tasanuma! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #3329: HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile
virajjasani commented on pull request #3329: URL: https://github.com/apache/hadoop/pull/3329#issuecomment-905241473 Thanks @ayushtkn for detailed review with cross-verification. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Neilxzn commented on a change in pull request #3320: HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with
Neilxzn commented on a change in pull request #3320:
URL: https://github.com/apache/hadoop/pull/3320#discussion_r695451621
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
##
@@ -469,7 +469,7 @@ private Node chooseTarget(int numOfReplicas,
LOG.trace("storageTypes={}", storageTypes);
try {
- if ((numOfReplicas = requiredStorageTypes.size()) == 0) {
Review comment:
done. Declare numOfReplicas a final variable at line 438.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
