[GitHub] [hadoop] imbajin commented on a change in pull request #2265: HDFS-15551. Tiny Improve for DeadNode detector
imbajin commented on a change in pull request #2265:
URL: https://github.com/apache/hadoop/pull/2265#discussion_r484620762
##
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DeadNodeDetector.java
##
@@ -396,13 +395,13 @@ private void probeCallBack(Probe probe, boolean success) {
probe.getDatanodeInfo());
removeDeadNode(probe.getDatanodeInfo());
} else if (probe.getType() == ProbeType.CHECK_SUSPECT) {
-LOG.debug("Remove the node out from suspect node list: {}.",
+LOG.info("Remove the node out from suspect node list: {}.",
Review comment:
ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] leosunli commented on pull request #2265: HDFS-15551. Tiny Improve for DeadNode detector
leosunli commented on pull request #2265: URL: https://github.com/apache/hadoop/pull/2265#issuecomment-688575038 +1 LGFM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jiwq commented on pull request #2277: HADOOP-17246. Fix build the hadoop-build Docker image failed
jiwq commented on pull request #2277: URL: https://github.com/apache/hadoop/pull/2277#issuecomment-688566323 @aajisaka Could you help to review? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17246) Fix build the hadoop-build Docker image failed
[
https://issues.apache.org/jira/browse/HADOOP-17246?focusedWorklogId=479818=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479818
]
ASF GitHub Bot logged work on HADOOP-17246:
---
Author: ASF GitHub Bot
Created on: 08/Sep/20 01:24
Start Date: 08/Sep/20 01:24
Worklog Time Spent: 10m
Work Description: jiwq commented on pull request #2277:
URL: https://github.com/apache/hadoop/pull/2277#issuecomment-688566323
@aajisaka Could you help to review?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479818)
Time Spent: 1h (was: 50m)
> Fix build the hadoop-build Docker image failed
> --
>
> Key: HADOOP-17246
> URL: https://issues.apache.org/jira/browse/HADOOP-17246
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Wanqiang Ji
>Assignee: Wanqiang Ji
>Priority: Major
> Labels: dockerfile, pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> When I build the docker-build image under macOS, it failed caused by:
> {code:java}
>
> Command "/usr/bin/python -u -c "import setuptools,
> tokenize;__file__='/tmp/pip-build-vKHcWu/isort/setup.py';exec(compile(getattr(tokenize,
> 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
> install --record /tmp/pip-odL0bY-record/install-record.txt
> --single-version-externally-managed --compile" failed with error code 1 in
> /tmp/pip-build-vKHcWu/isort/
> You are using pip version 8.1.1, however version 20.2.2 is available.
> You should consider upgrading via the 'pip install --upgrade pip' command.
> The command '/bin/bash -o pipefail -c pip2 install configparser==4.0.2
> pylint==1.9.2' returned a non-zero code: 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2266: [RBF] HDFS-15554 Force router check file existence in destinations before adding/updating mount points
hadoop-yetus commented on pull request #2266: URL: https://github.com/apache/hadoop/pull/2266#issuecomment-688553113 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 63m 50s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 44m 2s | trunk passed | | +1 :green_heart: | compile | 1m 5s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 47s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 34s | trunk passed | | +1 :green_heart: | mvnsite | 0m 58s | trunk passed | | +1 :green_heart: | shadedclient | 23m 6s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 59s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 16s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 2m 7s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 59s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 49s | the patch passed | | +1 :green_heart: | compile | 0m 49s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 49s | hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 0 new + 31 unchanged - 1 fixed = 31 total (was 32) | | +1 :green_heart: | compile | 0m 42s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 42s | hadoop-hdfs-project_hadoop-hdfs-rbf-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 0 new + 31 unchanged - 1 fixed = 31 total (was 32) | | -0 :warning: | checkstyle | 0m 22s | hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) | | +1 :green_heart: | mvnsite | 0m 46s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 21m 6s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 52s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 7s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 2m 8s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 16m 20s | hadoop-hdfs-rbf in the patch passed. | | +1 :green_heart: | asflicense | 0m 56s | The patch does not generate ASF License warnings. | | | | 188m 52s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterMountTableCacheRefresh | | | hadoop.hdfs.server.federation.router.TestRouterClientRejectOverload | | | hadoop.hdfs.server.federation.router.TestRouterRPCClientRetries | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2266/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2266 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml markdownlint | | uname | Linux 21b056ea9aaa 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / ac7d4623aef | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | checkstyle |
[GitHub] [hadoop] hadoop-yetus commented on pull request #2280: HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
hadoop-yetus commented on pull request #2280: URL: https://github.com/apache/hadoop/pull/2280#issuecomment-688527384 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 28m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 26m 37s | trunk passed | | +1 :green_heart: | compile | 19m 30s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 17m 0s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 2m 46s | trunk passed | | +1 :green_heart: | mvnsite | 2m 23s | trunk passed | | +1 :green_heart: | shadedclient | 19m 52s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 15s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 2m 17s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 15s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 28s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 22s | the patch passed | | +1 :green_heart: | compile | 18m 49s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 18m 49s | the patch passed | | +1 :green_heart: | compile | 16m 58s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 16m 58s | the patch passed | | -0 :warning: | checkstyle | 2m 42s | root: The patch generated 2 new + 171 unchanged - 0 fixed = 173 total (was 171) | | +1 :green_heart: | mvnsite | 2m 21s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 14m 8s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 13s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 2m 19s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 3m 47s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 9m 32s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 1m 35s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 55s | The patch does not generate ASF License warnings. | | | | 203m 41s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.TestLocalFSFileContextMainOperations | | | hadoop.fs.viewfs.TestFcMainOperationsLocalFs | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2280/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2280 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 8ed0a65527dd 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / ac7d4623aef | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2280/1/artifact/out/diff-checkstyle-root.txt | | unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2280/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2280/1/testReport/ | | Max. process+thread count | 2745 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | |
[jira] [Work logged] (HADOOP-17244) HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
[ https://issues.apache.org/jira/browse/HADOOP-17244?focusedWorklogId=479805=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479805 ] ASF GitHub Bot logged work on HADOOP-17244: --- Author: ASF GitHub Bot Created on: 07/Sep/20 22:23 Start Date: 07/Sep/20 22:23 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2280: URL: https://github.com/apache/hadoop/pull/2280#issuecomment-688527384 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 28m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 26m 37s | trunk passed | | +1 :green_heart: | compile | 19m 30s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 17m 0s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 2m 46s | trunk passed | | +1 :green_heart: | mvnsite | 2m 23s | trunk passed | | +1 :green_heart: | shadedclient | 19m 52s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 15s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 2m 17s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 15s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 28s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 22s | the patch passed | | +1 :green_heart: | compile | 18m 49s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 18m 49s | the patch passed | | +1 :green_heart: | compile | 16m 58s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 16m 58s | the patch passed | | -0 :warning: | checkstyle | 2m 42s | root: The patch generated 2 new + 171 unchanged - 0 fixed = 173 total (was 171) | | +1 :green_heart: | mvnsite | 2m 21s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 14m 8s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 13s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 2m 19s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 3m 47s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 9m 32s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 1m 35s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 55s | The patch does not generate ASF License warnings. | | | | 203m 41s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.TestLocalFSFileContextMainOperations | | | hadoop.fs.viewfs.TestFcMainOperationsLocalFs | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2280/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2280 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 8ed0a65527dd 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / ac7d4623aef | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | checkstyle |
[GitHub] [hadoop] fengnanli commented on a change in pull request #2266: [RBF] HDFS-15554 Force router check file existence in destinations before adding/updating mount points
fengnanli commented on a change in pull request #2266:
URL: https://github.com/apache/hadoop/pull/2266#discussion_r484569373
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterAdminServer.java
##
@@ -562,11 +595,35 @@ public GetDestinationResponse getDestination(
LOG.error("Cannot get location for {}: {}",
src, ioe.getMessage());
}
-if (nsIds.isEmpty() && !locations.isEmpty()) {
- String nsId = locations.get(0).getNameserviceId();
- nsIds.add(nsId);
+return nsIds;
+ }
+
+ /**
+ * Verify the file exists in destination nameservices to avoid dangling
+ * mount points.
+ *
+ * @param entry the new mount points added, could be from add or update.
+ * @return destination nameservices where the file doesn't exist.
+ * @throws IOException
+ */
+ private List verifyFileInDestinations(MountTable entry)
Review comment:
Had a discussion with @ayushtkn in the jira ticket and he suggested just
making the config to preserve the current workflow and enable this feature
optionally. I updated the diff as that and it is not necessary to fix a lot of
tests in that case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17191) ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
[ https://issues.apache.org/jira/browse/HADOOP-17191?focusedWorklogId=479758=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479758 ] ASF GitHub Bot logged work on HADOOP-17191: --- Author: ASF GitHub Bot Created on: 07/Sep/20 19:17 Start Date: 07/Sep/20 19:17 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2278: URL: https://github.com/apache/hadoop/pull/2278#issuecomment-688480976 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 29m 35s | trunk passed | | +1 :green_heart: | compile | 0m 42s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 38s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 32s | trunk passed | | +1 :green_heart: | mvnsite | 0m 44s | trunk passed | | +1 :green_heart: | shadedclient | 14m 18s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 36s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 1s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 31s | the patch passed | | +1 :green_heart: | compile | 0m 32s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 32s | the patch passed | | +1 :green_heart: | compile | 0m 27s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 27s | the patch passed | | +1 :green_heart: | checkstyle | 0m 19s | the patch passed | | +1 :green_heart: | mvnsite | 0m 30s | the patch passed | | -1 :x: | shellcheck | 0m 0s | The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | shelldocs | 0m 17s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) | | -1 :x: | whitespace | 0m 0s | The patch 2 line(s) with tabs. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 13m 53s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 29s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 26s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 1s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 31s | hadoop-azure in the patch passed. | | -1 :x: | asflicense | 0m 35s | The patch generated 1 ASF License warnings. | | | | 73m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2278 | | Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml | | uname | Linux d00bb145dfad 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 34fe74da0e9 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | shellcheck | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/6/artifact/out/diff-patch-shellcheck.txt | | whitespace |
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
hadoop-yetus commented on pull request #2278: URL: https://github.com/apache/hadoop/pull/2278#issuecomment-688480976 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 29m 35s | trunk passed | | +1 :green_heart: | compile | 0m 42s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 38s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 32s | trunk passed | | +1 :green_heart: | mvnsite | 0m 44s | trunk passed | | +1 :green_heart: | shadedclient | 14m 18s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 36s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 1s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 31s | the patch passed | | +1 :green_heart: | compile | 0m 32s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 32s | the patch passed | | +1 :green_heart: | compile | 0m 27s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 27s | the patch passed | | +1 :green_heart: | checkstyle | 0m 19s | the patch passed | | +1 :green_heart: | mvnsite | 0m 30s | the patch passed | | -1 :x: | shellcheck | 0m 0s | The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | shelldocs | 0m 17s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) | | -1 :x: | whitespace | 0m 0s | The patch 2 line(s) with tabs. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 13m 53s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 29s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 26s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 1s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 31s | hadoop-azure in the patch passed. | | -1 :x: | asflicense | 0m 35s | The patch generated 1 ASF License warnings. | | | | 73m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2278 | | Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml | | uname | Linux d00bb145dfad 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 34fe74da0e9 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | shellcheck | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/6/artifact/out/diff-patch-shellcheck.txt | | whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/6/artifact/out/whitespace-tabs.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/6/testReport/ | | asflicense | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/6/artifact/out/patch-asflicense-problems.txt | | Max. process+thread count | 424 (vs. ulimit of 5500) | | modules | C:
[GitHub] [hadoop] steveloughran commented on pull request #2280: HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
steveloughran commented on pull request #2280: URL: https://github.com/apache/hadoop/pull/2280#issuecomment-688475543 Tests in progress This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17244) HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
[
https://issues.apache.org/jira/browse/HADOOP-17244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HADOOP-17244:
Labels: pull-request-available (was: )
> HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
> --
>
> Key: HADOOP-17244
> URL: https://issues.apache.org/jira/browse/HADOOP-17244
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Test failure:
> {{ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory}}
> This is repeatable on -Dauth runs (we haven't been running them, have we?)
> Either its from the recent dir marker changes (initial hypothesis) or its
> been lurking a while and not been picked up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17244) HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
[
https://issues.apache.org/jira/browse/HADOOP-17244?focusedWorklogId=479755=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479755
]
ASF GitHub Bot logged work on HADOOP-17244:
---
Author: ASF GitHub Bot
Created on: 07/Sep/20 18:58
Start Date: 07/Sep/20 18:58
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2280:
URL: https://github.com/apache/hadoop/pull/2280#issuecomment-688475543
Tests in progress
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479755)
Time Spent: 20m (was: 10m)
> HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
> --
>
> Key: HADOOP-17244
> URL: https://issues.apache.org/jira/browse/HADOOP-17244
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Test failure:
> {{ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory}}
> This is repeatable on -Dauth runs (we haven't been running them, have we?)
> Either its from the recent dir marker changes (initial hypothesis) or its
> been lurking a while and not been picked up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17244) HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
[
https://issues.apache.org/jira/browse/HADOOP-17244?focusedWorklogId=479754=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479754
]
ASF GitHub Bot logged work on HADOOP-17244:
---
Author: ASF GitHub Bot
Created on: 07/Sep/20 18:58
Start Date: 07/Sep/20 18:58
Worklog Time Spent: 10m
Work Description: steveloughran opened a new pull request #2280:
URL: https://github.com/apache/hadoop/pull/2280
This changes directory tree deletion so that only files are incrementally
deleted
from S3Guard after the objects are deleted; the directories are left alone
until metadataStore.deleteSubtree(path) is invoked
This avoids directory tombstones being added above files/child directories,
which stop that subtree operation from actually finishing.
Also:
* callback to delete objects splits files and dirs so that
any problems deleting the dirs doesn't trigger s3guard updates
* new statistic to measure #of objects deleted, alongside request count.
* callback listFilesAndEmptyDirectories renamed listFilesAndDirectoryMarkers
to clarify behavior.
Test enhancements in ITestDirectoryOperationCost, but I didn't manage to
replicate it there (didn't quite understand the cause); the FileContext
test works for regression testing.
Change-Id: I8f96536bc6ffaf548b3da1068b9bfc897d34a993
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479754)
Remaining Estimate: 0h
Time Spent: 10m
> HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
> --
>
> Key: HADOOP-17244
> URL: https://issues.apache.org/jira/browse/HADOOP-17244
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Test failure:
> {{ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory}}
> This is repeatable on -Dauth runs (we haven't been running them, have we?)
> Either its from the recent dir marker changes (initial hypothesis) or its
> been lurking a while and not been picked up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran opened a new pull request #2280: HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
steveloughran opened a new pull request #2280: URL: https://github.com/apache/hadoop/pull/2280 This changes directory tree deletion so that only files are incrementally deleted from S3Guard after the objects are deleted; the directories are left alone until metadataStore.deleteSubtree(path) is invoked This avoids directory tombstones being added above files/child directories, which stop that subtree operation from actually finishing. Also: * callback to delete objects splits files and dirs so that any problems deleting the dirs doesn't trigger s3guard updates * new statistic to measure #of objects deleted, alongside request count. * callback listFilesAndEmptyDirectories renamed listFilesAndDirectoryMarkers to clarify behavior. Test enhancements in ITestDirectoryOperationCost, but I didn't manage to replicate it there (didn't quite understand the cause); the FileContext test works for regression testing. Change-Id: I8f96536bc6ffaf548b3da1068b9bfc897d34a993 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17244) HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
[
https://issues.apache.org/jira/browse/HADOOP-17244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-17244:
Summary: HADOOP-17244. S3A directory delete tombstones dir markers
prematurely. (was:
ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory
test failure on -Dauth)
> HADOOP-17244. S3A directory delete tombstones dir markers prematurely.
> --
>
> Key: HADOOP-17244
> URL: https://issues.apache.org/jira/browse/HADOOP-17244
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
>
> Test failure:
> {{ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory}}
> This is repeatable on -Dauth runs (we haven't been running them, have we?)
> Either its from the recent dir marker changes (initial hypothesis) or its
> been lurking a while and not been picked up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17244) ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory test failure on -Dauth
[
https://issues.apache.org/jira/browse/HADOOP-17244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191840#comment-17191840
]
Steve Loughran commented on HADOOP-17244:
-
This bug is is in {{DeleteOperation.deleteDirectoryTree()}} BTW...not any of
the new code.
Currently we naively list all entries and queue for delete in pages, and
there's clearly a big assumption there: dir markers don't have entries under
them.
What to change?
I'm going to to go with being less incremental
-files get deleted from S3Guard incrementally
-dir markers are not, instead at end of delete the existing cleanup will do its
thing
> ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory
> test failure on -Dauth
> --
>
> Key: HADOOP-17244
> URL: https://issues.apache.org/jira/browse/HADOOP-17244
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
>
> Test failure:
> {{ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory}}
> This is repeatable on -Dauth runs (we haven't been running them, have we?)
> Either its from the recent dir marker changes (initial hypothesis) or its
> been lurking a while and not been picked up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17191) ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
[ https://issues.apache.org/jira/browse/HADOOP-17191?focusedWorklogId=479749=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479749 ] ASF GitHub Bot logged work on HADOOP-17191: --- Author: ASF GitHub Bot Created on: 07/Sep/20 18:48 Start Date: 07/Sep/20 18:48 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2278: URL: https://github.com/apache/hadoop/pull/2278#issuecomment-688472439 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 29m 31s | trunk passed | | +1 :green_heart: | compile | 0m 40s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 38s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 32s | trunk passed | | +1 :green_heart: | mvnsite | 0m 43s | trunk passed | | +1 :green_heart: | shadedclient | 14m 24s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 35s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 2s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 59s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 31s | the patch passed | | +1 :green_heart: | compile | 0m 31s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 31s | the patch passed | | +1 :green_heart: | compile | 0m 26s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 26s | the patch passed | | +1 :green_heart: | checkstyle | 0m 19s | the patch passed | | +1 :green_heart: | mvnsite | 0m 32s | the patch passed | | -1 :x: | shellcheck | 0m 0s | The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | shelldocs | 0m 17s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) | | -1 :x: | whitespace | 0m 0s | The patch 2 line(s) with tabs. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 14m 19s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 30s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 27s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 1s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 31s | hadoop-azure in the patch passed. | | -1 :x: | asflicense | 0m 36s | The patch generated 1 ASF License warnings. | | | | 73m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2278 | | Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml | | uname | Linux 5dc09c5795e2 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 34fe74da0e9 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | shellcheck | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/5/artifact/out/diff-patch-shellcheck.txt | | whitespace |
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
hadoop-yetus commented on pull request #2278: URL: https://github.com/apache/hadoop/pull/2278#issuecomment-688472439 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 29m 31s | trunk passed | | +1 :green_heart: | compile | 0m 40s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 38s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 32s | trunk passed | | +1 :green_heart: | mvnsite | 0m 43s | trunk passed | | +1 :green_heart: | shadedclient | 14m 24s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 35s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 2s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 59s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 31s | the patch passed | | +1 :green_heart: | compile | 0m 31s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 31s | the patch passed | | +1 :green_heart: | compile | 0m 26s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 26s | the patch passed | | +1 :green_heart: | checkstyle | 0m 19s | the patch passed | | +1 :green_heart: | mvnsite | 0m 32s | the patch passed | | -1 :x: | shellcheck | 0m 0s | The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | shelldocs | 0m 17s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) | | -1 :x: | whitespace | 0m 0s | The patch 2 line(s) with tabs. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 14m 19s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 30s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 27s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 1s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 31s | hadoop-azure in the patch passed. | | -1 :x: | asflicense | 0m 36s | The patch generated 1 ASF License warnings. | | | | 73m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2278 | | Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml | | uname | Linux 5dc09c5795e2 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 34fe74da0e9 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | shellcheck | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/5/artifact/out/diff-patch-shellcheck.txt | | whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/5/artifact/out/whitespace-tabs.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/5/testReport/ | | asflicense | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/5/artifact/out/patch-asflicense-problems.txt | | Max. process+thread count | 414 (vs. ulimit of 5500) | | modules | C:
[GitHub] [hadoop] umamaheswararao merged pull request #2275: HDFS-15558: ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured
umamaheswararao merged pull request #2275: URL: https://github.com/apache/hadoop/pull/2275 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17191) ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
[ https://issues.apache.org/jira/browse/HADOOP-17191?focusedWorklogId=479744=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479744 ] ASF GitHub Bot logged work on HADOOP-17191: --- Author: ASF GitHub Bot Created on: 07/Sep/20 18:31 Start Date: 07/Sep/20 18:31 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2278: URL: https://github.com/apache/hadoop/pull/2278#issuecomment-688466988 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 31m 59s | trunk passed | | +1 :green_heart: | compile | 0m 41s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 26s | trunk passed | | +1 :green_heart: | mvnsite | 0m 38s | trunk passed | | +1 :green_heart: | shadedclient | 15m 58s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 30s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 26s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 0m 58s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 56s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 30s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 28s | the patch passed | | +1 :green_heart: | compile | 0m 24s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 24s | the patch passed | | +1 :green_heart: | checkstyle | 0m 17s | the patch passed | | +1 :green_heart: | mvnsite | 0m 29s | the patch passed | | -1 :x: | shellcheck | 0m 1s | The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | shelldocs | 0m 13s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) | | -1 :x: | whitespace | 0m 0s | The patch 2 line(s) with tabs. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 15m 54s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 22s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 0m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 17s | hadoop-azure in the patch passed. | | -1 :x: | asflicense | 0m 32s | The patch generated 1 ASF License warnings. | | | | 78m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2278 | | Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml | | uname | Linux f42098b5f563 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 34fe74da0e9 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | shellcheck | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/4/artifact/out/diff-patch-shellcheck.txt | | whitespace |
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
hadoop-yetus commented on pull request #2278: URL: https://github.com/apache/hadoop/pull/2278#issuecomment-688466988 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 31m 59s | trunk passed | | +1 :green_heart: | compile | 0m 41s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 26s | trunk passed | | +1 :green_heart: | mvnsite | 0m 38s | trunk passed | | +1 :green_heart: | shadedclient | 15m 58s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 30s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 26s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 0m 58s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 56s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 30s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 28s | the patch passed | | +1 :green_heart: | compile | 0m 24s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 24s | the patch passed | | +1 :green_heart: | checkstyle | 0m 17s | the patch passed | | +1 :green_heart: | mvnsite | 0m 29s | the patch passed | | -1 :x: | shellcheck | 0m 1s | The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | shelldocs | 0m 13s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) | | -1 :x: | whitespace | 0m 0s | The patch 2 line(s) with tabs. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 15m 54s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 22s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 0m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 17s | hadoop-azure in the patch passed. | | -1 :x: | asflicense | 0m 32s | The patch generated 1 ASF License warnings. | | | | 78m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2278 | | Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml | | uname | Linux f42098b5f563 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 34fe74da0e9 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | shellcheck | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/4/artifact/out/diff-patch-shellcheck.txt | | whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/4/artifact/out/whitespace-tabs.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/4/testReport/ | | asflicense | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/4/artifact/out/patch-asflicense-problems.txt | | Max. process+thread count | 311 (vs. ulimit of 5500) | | modules | C:
[jira] [Work logged] (HADOOP-17222) Create socket address combined with cache to speed up hdfs client choose DataNode
[
https://issues.apache.org/jira/browse/HADOOP-17222?focusedWorklogId=479727=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479727
]
ASF GitHub Bot logged work on HADOOP-17222:
---
Author: ASF GitHub Bot
Created on: 07/Sep/20 17:38
Start Date: 07/Sep/20 17:38
Worklog Time Spent: 10m
Work Description: liuml07 commented on pull request #2241:
URL: https://github.com/apache/hadoop/pull/2241#issuecomment-688451138
I'm now on vacation. I will merge it this week.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479727)
Time Spent: 0.5h (was: 20m)
> Create socket address combined with cache to speed up hdfs client choose
> DataNode
> -
>
> Key: HADOOP-17222
> URL: https://issues.apache.org/jira/browse/HADOOP-17222
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, hdfs-client
> Environment: HBase version: 2.1.0
> JVM: -Xmx2g -Xms2g
> hadoop hdfs version: 2.7.4
> disk:SSD
> OS:CentOS Linux release 7.4.1708 (Core)
> JMH Benchmark: @Fork(value = 1)
> @Warmup(iterations = 300)
> @Measurement(iterations = 300)
>Reporter: fanrui
>Assignee: fanrui
>Priority: Major
> Labels: pull-request-available
> Attachments: After Optimization remark.png, After optimization.svg,
> Before Optimization remark.png, Before optimization.svg
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Note：Not only the hdfs client can get the current benefit, all callers of
> NetUtils.createSocketAddr will get the benefit. Just use hdfs client as an
> example.
>
> Hdfs client selects best DN for hdfs Block. method call stack:
> DFSInputStream.chooseDataNode -> getBestNodeDNAddrPair ->
> NetUtils.createSocketAddr
> NetUtils.createSocketAddr creates the corresponding InetSocketAddress based
> on the host and port. There are some heavier operations in the
> NetUtils.createSocketAddr method, for example: URI.create(target), so
> NetUtils.createSocketAddr takes more time to execute.
> The following is my performance report. The report is based on HBase calling
> hdfs. HBase is a high-frequency access client for hdfs, because HBase read
> operations often access a small DataBlock (about 64k) instead of the entire
> HFile. In the case of high frequency access, the NetUtils.createSocketAddr
> method is time-consuming.
> h3. Test Environment：
>
> {code:java}
> HBase version: 2.1.0
> JVM: -Xmx2g -Xms2g
> hadoop hdfs version: 2.7.4
> disk:SSD
> OS:CentOS Linux release 7.4.1708 (Core)
> JMH Benchmark: @Fork(value = 1)
> @Warmup(iterations = 300)
> @Measurement(iterations = 300)
> {code}
> h4. Before Optimization FlameGraph:
> In the figure, we can see that DFSInputStream.getBestNodeDNAddrPair accounts
> for 4.86% of the entire CPU, and the creation of URIs accounts for a larger
> proportion.
> !Before Optimization remark.png!
> h3. Optimization ideas:
> NetUtils.createSocketAddr creates InetSocketAddress based on host and port.
> Here we can add Cache to InetSocketAddress. The key of Cache is host and
> port, and the value is InetSocketAddress.
> h4. After Optimization FlameGraph:
> In the figure, we can see that DFSInputStream.getBestNodeDNAddrPair accounts
> for 0.54% of the entire CPU. Here, ConcurrentHashMap is used as the Cache,
> and the ConcurrentHashMap.get() method gets data from the Cache. The CPU
> usage of DFSInputStream.getBestNodeDNAddrPair has been optimized from 4.86%
> to 0.54%.
> !After Optimization remark.png!
> h3. Original FlameGraph link:
> [Before
> Optimization|https://drive.google.com/file/d/133L5m75u2tu_KgKfGHZLEUzGR0XAfUl6/view?usp=sharing]
> [After Optimization
> FlameGraph|https://drive.google.com/file/d/133L5m75u2tu_KgKfGHZLEUzGR0XAfUl6/view?usp=sharing]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on pull request #2241: HADOOP-17222. Create socket address combined with cache
liuml07 commented on pull request #2241: URL: https://github.com/apache/hadoop/pull/2241#issuecomment-688451138 I'm now on vacation. I will merge it this week. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13230) S3A to optionally retain directory markers
[ https://issues.apache.org/jira/browse/HADOOP-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191816#comment-17191816 ] Steve Loughran commented on HADOOP-13230: - First followup JIRA to this. HADOOP-17244 > S3A to optionally retain directory markers > -- > > Key: HADOOP-13230 > URL: https://issues.apache.org/jira/browse/HADOOP-13230 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0 >Reporter: Aaron Fabbri >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Attachments: 2020-02-Fixing the S3A directory marker problem.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Users of s3a may not realize that, in some cases, it does not interoperate > well with other s3 tools, such as the AWS CLI. (See HIVE-13778, IMPALA-3558). > Specifically, if a user: > - Creates an empty directory with hadoop fs -mkdir s3a://bucket/path > - Copies data into that directory via another tool, i.e. aws cli. > - Tries to access the data in that directory with any Hadoop software. > Then the last step fails because the fake empty directory blob that s3a wrote > in the first step, causes s3a (listStatus() etc.) to continue to treat that > directory as empty, even though the second step was supposed to populate the > directory with data. > I wanted to document this fact for users. We may mark this as not-fix, "by > design".. May also be interesting to brainstorm solutions and/or a config > option to change the behavior if folks care. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17244) ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory test failure on -Dauth
[
https://issues.apache.org/jira/browse/HADOOP-17244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191814#comment-17191814
]
Steve Loughran commented on HADOOP-17244:
-
Working hypothesis
The presence of a marker with child entries in s3guard *which don't have S3
entries of their own* causes the marker to be deleted in the final batch of
paths deleted (maybe also at an interim point), which then places a tombstone
at that point in the table.
the subsequent scan to delete all s3guard entries hits the tombstone, so
doesn't bother looking underneath.
When you create the new dir back again, then the orphan entry is reinstated.so
it reappears as a directory.
Now, why only in auth mode? The bucket is set up with auth mode retention
(albeit for only a different subdir of the store). Once the -Dauth test run
puts the bucket into auth mode, its retained everywhere, and this test setup
finds it.
{code}
fs.s3a.bucket.stevel-london.directory.marker.retention
authoritative
fs.s3a.bucket.stevel-london.authoritative.path
/tables
{code}
Now #2: so why didn't -Dkeep find it? No idea whatsoever. Maybe it's also that
the auth flag changes listing too
> ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory
> test failure on -Dauth
> --
>
> Key: HADOOP-17244
> URL: https://issues.apache.org/jira/browse/HADOOP-17244
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
>
> Test failure:
> {{ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory}}
> This is repeatable on -Dauth runs (we haven't been running them, have we?)
> Either its from the recent dir marker changes (initial hypothesis) or its
> been lurking a while and not been picked up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16830) Add public IOStatistics API; S3A to support
[ https://issues.apache.org/jira/browse/HADOOP-16830?focusedWorklogId=479637=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479637 ] ASF GitHub Bot logged work on HADOOP-16830: --- Author: ASF GitHub Bot Created on: 07/Sep/20 13:23 Start Date: 07/Sep/20 13:23 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2069: URL: https://github.com/apache/hadoop/pull/2069#issuecomment-688323653 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 3s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 2s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 38 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 28m 16s | trunk passed | | +1 :green_heart: | compile | 20m 50s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 17m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 3m 1s | trunk passed | | +1 :green_heart: | mvnsite | 2m 54s | trunk passed | | +1 :green_heart: | shadedclient | 22m 4s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 24s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 2m 26s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 9s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 41s | trunk passed | | -0 :warning: | patch | 1m 28s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 55s | the patch passed | | +1 :green_heart: | compile | 20m 9s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | javac | 20m 9s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 2 new + 2061 unchanged - 1 fixed = 2063 total (was 2062) | | +1 :green_heart: | compile | 17m 37s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | -1 :x: | javac | 17m 37s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 2 new + 1955 unchanged - 1 fixed = 1957 total (was 1956) | | -0 :warning: | checkstyle | 2m 56s | root: The patch generated 16 new + 266 unchanged - 26 fixed = 282 total (was 292) | | +1 :green_heart: | mvnsite | 2m 53s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 14 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 15m 34s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 22s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | javadoc | 1m 27s | hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) | | +1 :green_heart: | javadoc | 0m 26s | hadoop-mapreduce-client-core in the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | +1 :green_heart: | javadoc | 0m 34s | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 0 new + 0 unchanged - 4 fixed = 0 total (was 4) | | -1 :x: | findbugs | 2m 21s | hadoop-common-project/hadoop-common generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | unit | 9m 58s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 6m 56s |
[GitHub] [hadoop] hadoop-yetus commented on pull request #2069: HADOOP-16830. IOStatistics API.
hadoop-yetus commented on pull request #2069: URL: https://github.com/apache/hadoop/pull/2069#issuecomment-688323653 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 3s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 2s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 38 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 28m 16s | trunk passed | | +1 :green_heart: | compile | 20m 50s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 17m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 3m 1s | trunk passed | | +1 :green_heart: | mvnsite | 2m 54s | trunk passed | | +1 :green_heart: | shadedclient | 22m 4s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 24s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 2m 26s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 9s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 41s | trunk passed | | -0 :warning: | patch | 1m 28s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 55s | the patch passed | | +1 :green_heart: | compile | 20m 9s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | javac | 20m 9s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 2 new + 2061 unchanged - 1 fixed = 2063 total (was 2062) | | +1 :green_heart: | compile | 17m 37s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | -1 :x: | javac | 17m 37s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 2 new + 1955 unchanged - 1 fixed = 1957 total (was 1956) | | -0 :warning: | checkstyle | 2m 56s | root: The patch generated 16 new + 266 unchanged - 26 fixed = 282 total (was 292) | | +1 :green_heart: | mvnsite | 2m 53s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 14 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 15m 34s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 22s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | javadoc | 1m 27s | hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) | | +1 :green_heart: | javadoc | 0m 26s | hadoop-mapreduce-client-core in the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | +1 :green_heart: | javadoc | 0m 34s | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 0 new + 0 unchanged - 4 fixed = 0 total (was 4) | | -1 :x: | findbugs | 2m 21s | hadoop-common-project/hadoop-common generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | unit | 9m 58s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 6m 56s | hadoop-mapreduce-client-core in the patch passed. | | +1 :green_heart: | unit | 1m 39s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 47s | The patch does not generate ASF License warnings. | | | | 196m 42s | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-common-project/hadoop-common | | | Inconsistent synchronization of
[jira] [Work logged] (HADOOP-17023) Tune listStatus() api of s3a.
[ https://issues.apache.org/jira/browse/HADOOP-17023?focusedWorklogId=479630=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479630 ] ASF GitHub Bot logged work on HADOOP-17023: --- Author: ASF GitHub Bot Created on: 07/Sep/20 13:08 Start Date: 07/Sep/20 13:08 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2257: URL: https://github.com/apache/hadoop/pull/2257#issuecomment-688314591 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 33m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. | ||| _ trunk Compile Tests _ | | -1 :x: | mvninstall | 0m 22s | root in trunk failed. | | -1 :x: | compile | 0m 11s | hadoop-aws in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | compile | 0m 25s | hadoop-aws in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -0 :warning: | checkstyle | 0m 20s | The patch fails to run checkstyle in hadoop-aws | | -1 :x: | mvnsite | 0m 10s | hadoop-aws in trunk failed. | | -1 :x: | shadedclient | 0m 49s | branch has errors when building and testing our client artifacts. | | -1 :x: | javadoc | 0m 16s | hadoop-aws in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javadoc | 0m 22s | hadoop-aws in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | +0 :ok: | spotbugs | 1m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 :x: | findbugs | 0m 22s | hadoop-aws in trunk failed. | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 0m 41s | hadoop-aws in the patch failed. | | -1 :x: | compile | 0m 23s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javac | 0m 23s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | compile | 0m 22s | hadoop-aws in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -1 :x: | javac | 0m 22s | hadoop-aws in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -0 :warning: | checkstyle | 0m 5s | The patch fails to run checkstyle in hadoop-aws | | -1 :x: | mvnsite | 0m 12s | hadoop-aws in the patch failed. | | -1 :x: | whitespace | 0m 0s | The patch has 38 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | -1 :x: | whitespace | 0m 0s | The patch 600 line(s) with tabs. | | +1 :green_heart: | shadedclient | 0m 21s | patch has no errors when building and testing our client artifacts. | | -1 :x: | javadoc | 0m 22s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javadoc | 0m 23s | hadoop-aws in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -1 :x: | findbugs | 0m 23s | hadoop-aws in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 22s | hadoop-aws in the patch failed. | | +0 :ok: | asflicense | 0m 22s | ASF License check generated no output? | | | | 43m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2257 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 667bf71456f4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 34fe74da0e9 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | mvninstall | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-mvninstall-root.txt | | compile |
[GitHub] [hadoop] hadoop-yetus commented on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
hadoop-yetus commented on pull request #2257: URL: https://github.com/apache/hadoop/pull/2257#issuecomment-688314591 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 33m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. | ||| _ trunk Compile Tests _ | | -1 :x: | mvninstall | 0m 22s | root in trunk failed. | | -1 :x: | compile | 0m 11s | hadoop-aws in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | compile | 0m 25s | hadoop-aws in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -0 :warning: | checkstyle | 0m 20s | The patch fails to run checkstyle in hadoop-aws | | -1 :x: | mvnsite | 0m 10s | hadoop-aws in trunk failed. | | -1 :x: | shadedclient | 0m 49s | branch has errors when building and testing our client artifacts. | | -1 :x: | javadoc | 0m 16s | hadoop-aws in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javadoc | 0m 22s | hadoop-aws in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | +0 :ok: | spotbugs | 1m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 :x: | findbugs | 0m 22s | hadoop-aws in trunk failed. | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 0m 41s | hadoop-aws in the patch failed. | | -1 :x: | compile | 0m 23s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javac | 0m 23s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | compile | 0m 22s | hadoop-aws in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -1 :x: | javac | 0m 22s | hadoop-aws in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -0 :warning: | checkstyle | 0m 5s | The patch fails to run checkstyle in hadoop-aws | | -1 :x: | mvnsite | 0m 12s | hadoop-aws in the patch failed. | | -1 :x: | whitespace | 0m 0s | The patch has 38 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | -1 :x: | whitespace | 0m 0s | The patch 600 line(s) with tabs. | | +1 :green_heart: | shadedclient | 0m 21s | patch has no errors when building and testing our client artifacts. | | -1 :x: | javadoc | 0m 22s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javadoc | 0m 23s | hadoop-aws in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -1 :x: | findbugs | 0m 23s | hadoop-aws in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 22s | hadoop-aws in the patch failed. | | +0 :ok: | asflicense | 0m 22s | ASF License check generated no output? | | | | 43m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2257 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 667bf71456f4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 34fe74da0e9 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | mvninstall | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt | | compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt | | checkstyle |
[jira] [Work logged] (HADOOP-17242) S3A (async) ObjectListingIterator to block in hasNext() for results
[ https://issues.apache.org/jira/browse/HADOOP-17242?focusedWorklogId=479620=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479620 ] ASF GitHub Bot logged work on HADOOP-17242: --- Author: ASF GitHub Bot Created on: 07/Sep/20 12:23 Start Date: 07/Sep/20 12:23 Worklog Time Spent: 10m Work Description: mukund-thakur commented on pull request #2273: URL: https://github.com/apache/hadoop/pull/2273#issuecomment-688290398 CC @mukund-thakur. Just to get subscribed. I will review as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 479620) Time Spent: 50m (was: 40m) > S3A (async) ObjectListingIterator to block in hasNext() for results > --- > > Key: HADOOP-17242 > URL: https://issues.apache.org/jira/browse/HADOOP-17242 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > HADOOP-17074 made listing async in S3A, but the iterator's hasNext Call > doesn't wait for the result. If invoked on an empty path it *may* return when > it should be failing. > Note: surfaced in code review, not seen in the wild and all our tests were > happy -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on pull request #2273: HADOOP-17242. S3A ObjectListingIterator to block in hasNext() for results
mukund-thakur commented on pull request #2273: URL: https://github.com/apache/hadoop/pull/2273#issuecomment-688290398 CC @mukund-thakur. Just to get subscribed. I will review as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17244) ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory test failure on -Dauth
[
https://issues.apache.org/jira/browse/HADOOP-17244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191674#comment-17191674
]
Steve Loughran commented on HADOOP-17244:
-
It's actually in the probe for an empty dir; just so happens the exception is
currently being lost in the test
{code}
if (dstStatus.isDirectory()) {
RemoteIterator list = listStatusIterator(dst);
if (list != null && list.hasNext()) {
throw new IOException(
"Rename cannot overwrite non empty destination directory " + dst);
}
}
{code}
This could expand the root cause to the listing changes, though as it is still
going through the getFileStatus changes, I'm not yet pointing blame there
> ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory
> test failure on -Dauth
> --
>
> Key: HADOOP-17244
> URL: https://issues.apache.org/jira/browse/HADOOP-17244
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
>
> Test failure:
> {{ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory}}
> This is repeatable on -Dauth runs (we haven't been running them, have we?)
> Either its from the recent dir marker changes (initial hypothesis) or its
> been lurking a while and not been picked up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
hadoop-yetus commented on pull request #2278: URL: https://github.com/apache/hadoop/pull/2278#issuecomment-688262400 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 31m 39s | trunk passed | | +1 :green_heart: | compile | 0m 38s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 25s | trunk passed | | +1 :green_heart: | mvnsite | 0m 35s | trunk passed | | +1 :green_heart: | shadedclient | 15m 54s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 28s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 9s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 6s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 33s | the patch passed | | +1 :green_heart: | compile | 0m 33s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 33s | the patch passed | | +1 :green_heart: | compile | 0m 27s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 27s | the patch passed | | +1 :green_heart: | checkstyle | 0m 16s | the patch passed | | +1 :green_heart: | mvnsite | 0m 28s | the patch passed | | -1 :x: | shellcheck | 0m 0s | The patch generated 10 new + 0 unchanged - 0 fixed = 10 total (was 0) | | +1 :green_heart: | shelldocs | 0m 13s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 17m 9s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 28s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 9s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 27s | hadoop-azure in the patch passed. | | +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. | | | | 80m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2278 | | Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml | | uname | Linux 6368d9d69464 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 34fe74da0e9 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | shellcheck | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/3/artifact/out/diff-patch-shellcheck.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/3/testReport/ | | Max. process+thread count | 306 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/3/console | | versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.0.6 | | Powered
[jira] [Work logged] (HADOOP-17191) ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
[ https://issues.apache.org/jira/browse/HADOOP-17191?focusedWorklogId=479594=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479594 ] ASF GitHub Bot logged work on HADOOP-17191: --- Author: ASF GitHub Bot Created on: 07/Sep/20 11:25 Start Date: 07/Sep/20 11:25 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2278: URL: https://github.com/apache/hadoop/pull/2278#issuecomment-688262400 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 31m 39s | trunk passed | | +1 :green_heart: | compile | 0m 38s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 25s | trunk passed | | +1 :green_heart: | mvnsite | 0m 35s | trunk passed | | +1 :green_heart: | shadedclient | 15m 54s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 28s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 9s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 6s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 33s | the patch passed | | +1 :green_heart: | compile | 0m 33s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 33s | the patch passed | | +1 :green_heart: | compile | 0m 27s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 27s | the patch passed | | +1 :green_heart: | checkstyle | 0m 16s | the patch passed | | +1 :green_heart: | mvnsite | 0m 28s | the patch passed | | -1 :x: | shellcheck | 0m 0s | The patch generated 10 new + 0 unchanged - 0 fixed = 10 total (was 0) | | +1 :green_heart: | shelldocs | 0m 13s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 17m 9s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 28s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 9s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 27s | hadoop-azure in the patch passed. | | +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. | | | | 80m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2278 | | Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml | | uname | Linux 6368d9d69464 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 34fe74da0e9 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | shellcheck |
[jira] [Work logged] (HADOOP-16202) Enhance S3A openFile()
[
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=479568=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479568
]
ASF GitHub Bot logged work on HADOOP-16202:
---
Author: ASF GitHub Bot
Created on: 07/Sep/20 10:24
Start Date: 07/Sep/20 10:24
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-688229908
tests
```
[ERROR] Errors:
[ERROR]
TestLocalFSContractOpen>AbstractContractOpenTest.testOpenFileNullStatus:358 ?
IllegalArgument
[ERROR]
TestRawlocalContractOpen>AbstractContractOpenTest.testOpenFileNullStatus:358 ?
IllegalArgument
[INFO]
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479568)
Time Spent: 1h (was: 50m)
> Enhance S3A openFile()
> --
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows
> the length of the file to be declared. If set, *no check for the existence of
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than
> only S3AFileStatus -and not check that the path matches the path being
> opened. Needed to support viewFS-style wrapping and mounting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2168: HADOOP-16202. Enhance openFile()
steveloughran commented on pull request #2168: URL: https://github.com/apache/hadoop/pull/2168#issuecomment-688229908 tests ``` [ERROR] Errors: [ERROR] TestLocalFSContractOpen>AbstractContractOpenTest.testOpenFileNullStatus:358 ? IllegalArgument [ERROR] TestRawlocalContractOpen>AbstractContractOpenTest.testOpenFileNullStatus:358 ? IllegalArgument [INFO] ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2168: HADOOP-16202. Enhance openFile()
steveloughran commented on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-688229627
```
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/OpenFileOptions.java:31:@InterfaceAudience.Public:1:
Utility classes should not have a public or default constructor.
[HideUtilityClassConstructor]
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/OpenFileParameters.java:22:import
java.util.stream.Collectors;:8: Unused import - java.util.stream.Collectors.
[UnusedImports]
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/OpenFileParameters.java:23:import
java.util.stream.Stream;:8: Unused import - java.util.stream.Stream.
[UnusedImports]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AOpenFileOperation.java:52:
public class S3AOpenFileOperation extends AbstractStoreOperation {: 'class
def modifier' has incorrect indentation level 2, expected level should be 0.
[Indentation]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AOpenFileOperation.java:106:
/** SQL string if this is a SQL select file*/: First sentence should end
with a period. [JavadocStyle]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AOpenFileOperation.java:280:
/**: First sentence should end with a period. [JavadocStyle]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestOpenFileHelper.java:215:
/**: First sentence should end with a period. [JavadocStyle]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestS3AOpenCost.java:169:
byte[] out = new byte[(int) longLen];: 'block' child has incorrect
indentation level 10, expected level should be 6. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestS3AOpenCost.java:170:
intercept(EOFException.class,: 'block' child has incorrect
indentation level 10, expected level should be 6. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestS3AOpenCost.java:172:
in3.seek(longLen - 1);: 'block' child has incorrect indentation level
10, expected level should be 6. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestS3AOpenCost.java:173:
assertEquals("read past real EOF on " + in3,: 'block' child has
incorrect indentation level 10, expected level should be 6. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestS3AOpenCost.java:175:
in3.close();: 'block' child has incorrect indentation level 10,
expected level should be 6. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestS3AOpenCost.java:176:
return null;: 'block' child has incorrect indentation level 10,
expected level should be 6. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestS3AOpenCost.java:177:
},: 'block rcurly' has incorrect indentation level 8, expected level
should be 4. [Indentation]
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16202) Enhance S3A openFile()
[
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=479567=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479567
]
ASF GitHub Bot logged work on HADOOP-16202:
---
Author: ASF GitHub Bot
Created on: 07/Sep/20 10:23
Start Date: 07/Sep/20 10:23
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-688229627
```
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/OpenFileOptions.java:31:@InterfaceAudience.Public:1:
Utility classes should not have a public or default constructor.
[HideUtilityClassConstructor]
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/OpenFileParameters.java:22:import
java.util.stream.Collectors;:8: Unused import - java.util.stream.Collectors.
[UnusedImports]
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/OpenFileParameters.java:23:import
java.util.stream.Stream;:8: Unused import - java.util.stream.Stream.
[UnusedImports]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AOpenFileOperation.java:52:
public class S3AOpenFileOperation extends AbstractStoreOperation {: 'class
def modifier' has incorrect indentation level 2, expected level should be 0.
[Indentation]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AOpenFileOperation.java:106:
/** SQL string if this is a SQL select file*/: First sentence should end
with a period. [JavadocStyle]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AOpenFileOperation.java:280:
/**: First sentence should end with a period. [JavadocStyle]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestOpenFileHelper.java:215:
/**: First sentence should end with a period. [JavadocStyle]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestS3AOpenCost.java:169:
byte[] out = new byte[(int) longLen];: 'block' child has incorrect
indentation level 10, expected level should be 6. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestS3AOpenCost.java:170:
intercept(EOFException.class,: 'block' child has incorrect
indentation level 10, expected level should be 6. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestS3AOpenCost.java:172:
in3.seek(longLen - 1);: 'block' child has incorrect indentation level
10, expected level should be 6. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestS3AOpenCost.java:173:
assertEquals("read past real EOF on " + in3,: 'block' child has
incorrect indentation level 10, expected level should be 6. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestS3AOpenCost.java:175:
in3.close();: 'block' child has incorrect indentation level 10,
expected level should be 6. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestS3AOpenCost.java:176:
return null;: 'block' child has incorrect indentation level 10,
expected level should be 6. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestS3AOpenCost.java:177:
},: 'block rcurly' has incorrect indentation level 8, expected level
should be 4. [Indentation]
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479567)
Time Spent: 50m (was: 40m)
> Enhance S3A openFile()
> --
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows
> the length of the file to be declared. If set, *no check for the existence of
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than
> only S3AFileStatus -and not check that
[jira] [Work logged] (HADOOP-16202) Enhance S3A openFile()
[
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=479566=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479566
]
ASF GitHub Bot logged work on HADOOP-16202:
---
Author: ASF GitHub Bot
Created on: 07/Sep/20 10:20
Start Date: 07/Sep/20 10:20
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-669933296
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479566)
Time Spent: 40m (was: 0.5h)
> Enhance S3A openFile()
> --
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows
> the length of the file to be declared. If set, *no check for the existence of
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than
> only S3AFileStatus -and not check that the path matches the path being
> opened. Needed to support viewFS-style wrapping and mounting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2168: HADOOP-16202. Enhance openFile()
hadoop-yetus removed a comment on pull request #2168: URL: https://github.com/apache/hadoop/pull/2168#issuecomment-669933296 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2168: HADOOP-16202. Enhance openFile()
hadoop-yetus removed a comment on pull request #2168: URL: https://github.com/apache/hadoop/pull/2168#issuecomment-663534484 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16202) Enhance S3A openFile()
[
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=479565=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479565
]
ASF GitHub Bot logged work on HADOOP-16202:
---
Author: ASF GitHub Bot
Created on: 07/Sep/20 10:19
Start Date: 07/Sep/20 10:19
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #2168:
URL: https://github.com/apache/hadoop/pull/2168#issuecomment-663534484
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479565)
Time Spent: 0.5h (was: 20m)
> Enhance S3A openFile()
> --
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows
> the length of the file to be declared. If set, *no check for the existence of
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than
> only S3AFileStatus -and not check that the path matches the path being
> opened. Needed to support viewFS-style wrapping and mounting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-17244) ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory test failure on -Dauth
[
https://issues.apache.org/jira/browse/HADOOP-17244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-17244 started by Steve Loughran.
---
> ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory
> test failure on -Dauth
> --
>
> Key: HADOOP-17244
> URL: https://issues.apache.org/jira/browse/HADOOP-17244
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
>
> Test failure:
> {{ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory}}
> This is repeatable on -Dauth runs (we haven't been running them, have we?)
> Either its from the recent dir marker changes (initial hypothesis) or its
> been lurking a while and not been picked up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17242) S3A (async) ObjectListingIterator to block in hasNext() for results
[ https://issues.apache.org/jira/browse/HADOOP-17242?focusedWorklogId=479564=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479564 ] ASF GitHub Bot logged work on HADOOP-17242: --- Author: ASF GitHub Bot Created on: 07/Sep/20 10:15 Start Date: 07/Sep/20 10:15 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2273: URL: https://github.com/apache/hadoop/pull/2273#issuecomment-688225319 ``` Unread field: org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.firstListing -- | Bug type URF_UNREAD_FIELD (click for details)In class org.apache.hadoop.fs.s3a.Listing$ObjectListingIteratorField org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.firstListingAt Listing.java:[line 731] ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 479564) Time Spent: 40m (was: 0.5h) > S3A (async) ObjectListingIterator to block in hasNext() for results > --- > > Key: HADOOP-17242 > URL: https://issues.apache.org/jira/browse/HADOOP-17242 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > HADOOP-17074 made listing async in S3A, but the iterator's hasNext Call > doesn't wait for the result. If invoked on an empty path it *may* return when > it should be failing. > Note: surfaced in code review, not seen in the wild and all our tests were > happy -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2273: HADOOP-17242. S3A ObjectListingIterator to block in hasNext() for results
steveloughran commented on pull request #2273: URL: https://github.com/apache/hadoop/pull/2273#issuecomment-688225319 ``` Unread field: org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.firstListing -- | Bug type URF_UNREAD_FIELD (click for details)In class org.apache.hadoop.fs.s3a.Listing$ObjectListingIteratorField org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.firstListingAt Listing.java:[line 731] ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17247) Support Non-Path Based FileSystem API in Regex Mount Points
[ https://issues.apache.org/jira/browse/HADOOP-17247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191612#comment-17191612 ] Steve Loughran commented on HADOOP-17247: - usual request: tag with affects version, components, target fix > Support Non-Path Based FileSystem API in Regex Mount Points > --- > > Key: HADOOP-17247 > URL: https://issues.apache.org/jira/browse/HADOOP-17247 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: zhenzhao wang >Priority: Major > > Regex mount points create ChoRootedFileSystem while accessing. And we > couldn't know the underlying filesystems ahead. This won't work with non-path > based APIs such as getAdditionalTokenIssuers. Instead of totaly unsuport it, > we should support APIs to some extend. The could be done recording the > FileSystem created and perform APIs for FileSystem instances created for > ViewFileSystem. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17222) Create socket address combined with cache to speed up hdfs client choose DataNode
[
https://issues.apache.org/jira/browse/HADOOP-17222?focusedWorklogId=479535=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479535
]
ASF GitHub Bot logged work on HADOOP-17222:
---
Author: ASF GitHub Bot
Created on: 07/Sep/20 08:29
Start Date: 07/Sep/20 08:29
Worklog Time Spent: 10m
Work Description: 1996fanrui commented on pull request #2241:
URL: https://github.com/apache/hadoop/pull/2241#issuecomment-688160009
Hi @liuml07 ,Could you have time to help merge it? Thank you.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479535)
Time Spent: 20m (was: 10m)
> Create socket address combined with cache to speed up hdfs client choose
> DataNode
> -
>
> Key: HADOOP-17222
> URL: https://issues.apache.org/jira/browse/HADOOP-17222
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, hdfs-client
> Environment: HBase version: 2.1.0
> JVM: -Xmx2g -Xms2g
> hadoop hdfs version: 2.7.4
> disk:SSD
> OS:CentOS Linux release 7.4.1708 (Core)
> JMH Benchmark: @Fork(value = 1)
> @Warmup(iterations = 300)
> @Measurement(iterations = 300)
>Reporter: fanrui
>Assignee: fanrui
>Priority: Major
> Attachments: After Optimization remark.png, After optimization.svg,
> Before Optimization remark.png, Before optimization.svg
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Note：Not only the hdfs client can get the current benefit, all callers of
> NetUtils.createSocketAddr will get the benefit. Just use hdfs client as an
> example.
>
> Hdfs client selects best DN for hdfs Block. method call stack:
> DFSInputStream.chooseDataNode -> getBestNodeDNAddrPair ->
> NetUtils.createSocketAddr
> NetUtils.createSocketAddr creates the corresponding InetSocketAddress based
> on the host and port. There are some heavier operations in the
> NetUtils.createSocketAddr method, for example: URI.create(target), so
> NetUtils.createSocketAddr takes more time to execute.
> The following is my performance report. The report is based on HBase calling
> hdfs. HBase is a high-frequency access client for hdfs, because HBase read
> operations often access a small DataBlock (about 64k) instead of the entire
> HFile. In the case of high frequency access, the NetUtils.createSocketAddr
> method is time-consuming.
> h3. Test Environment：
>
> {code:java}
> HBase version: 2.1.0
> JVM: -Xmx2g -Xms2g
> hadoop hdfs version: 2.7.4
> disk:SSD
> OS:CentOS Linux release 7.4.1708 (Core)
> JMH Benchmark: @Fork(value = 1)
> @Warmup(iterations = 300)
> @Measurement(iterations = 300)
> {code}
> h4. Before Optimization FlameGraph:
> In the figure, we can see that DFSInputStream.getBestNodeDNAddrPair accounts
> for 4.86% of the entire CPU, and the creation of URIs accounts for a larger
> proportion.
> !Before Optimization remark.png!
> h3. Optimization ideas:
> NetUtils.createSocketAddr creates InetSocketAddress based on host and port.
> Here we can add Cache to InetSocketAddress. The key of Cache is host and
> port, and the value is InetSocketAddress.
> h4. After Optimization FlameGraph:
> In the figure, we can see that DFSInputStream.getBestNodeDNAddrPair accounts
> for 0.54% of the entire CPU. Here, ConcurrentHashMap is used as the Cache,
> and the ConcurrentHashMap.get() method gets data from the Cache. The CPU
> usage of DFSInputStream.getBestNodeDNAddrPair has been optimized from 4.86%
> to 0.54%.
> !After Optimization remark.png!
> h3. Original FlameGraph link:
> [Before
> Optimization|https://drive.google.com/file/d/133L5m75u2tu_KgKfGHZLEUzGR0XAfUl6/view?usp=sharing]
> [After Optimization
> FlameGraph|https://drive.google.com/file/d/133L5m75u2tu_KgKfGHZLEUzGR0XAfUl6/view?usp=sharing]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17222) Create socket address combined with cache to speed up hdfs client choose DataNode
[
https://issues.apache.org/jira/browse/HADOOP-17222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HADOOP-17222:
Labels: pull-request-available (was: )
> Create socket address combined with cache to speed up hdfs client choose
> DataNode
> -
>
> Key: HADOOP-17222
> URL: https://issues.apache.org/jira/browse/HADOOP-17222
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, hdfs-client
> Environment: HBase version: 2.1.0
> JVM: -Xmx2g -Xms2g
> hadoop hdfs version: 2.7.4
> disk:SSD
> OS:CentOS Linux release 7.4.1708 (Core)
> JMH Benchmark: @Fork(value = 1)
> @Warmup(iterations = 300)
> @Measurement(iterations = 300)
>Reporter: fanrui
>Assignee: fanrui
>Priority: Major
> Labels: pull-request-available
> Attachments: After Optimization remark.png, After optimization.svg,
> Before Optimization remark.png, Before optimization.svg
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Note：Not only the hdfs client can get the current benefit, all callers of
> NetUtils.createSocketAddr will get the benefit. Just use hdfs client as an
> example.
>
> Hdfs client selects best DN for hdfs Block. method call stack:
> DFSInputStream.chooseDataNode -> getBestNodeDNAddrPair ->
> NetUtils.createSocketAddr
> NetUtils.createSocketAddr creates the corresponding InetSocketAddress based
> on the host and port. There are some heavier operations in the
> NetUtils.createSocketAddr method, for example: URI.create(target), so
> NetUtils.createSocketAddr takes more time to execute.
> The following is my performance report. The report is based on HBase calling
> hdfs. HBase is a high-frequency access client for hdfs, because HBase read
> operations often access a small DataBlock (about 64k) instead of the entire
> HFile. In the case of high frequency access, the NetUtils.createSocketAddr
> method is time-consuming.
> h3. Test Environment：
>
> {code:java}
> HBase version: 2.1.0
> JVM: -Xmx2g -Xms2g
> hadoop hdfs version: 2.7.4
> disk:SSD
> OS:CentOS Linux release 7.4.1708 (Core)
> JMH Benchmark: @Fork(value = 1)
> @Warmup(iterations = 300)
> @Measurement(iterations = 300)
> {code}
> h4. Before Optimization FlameGraph:
> In the figure, we can see that DFSInputStream.getBestNodeDNAddrPair accounts
> for 4.86% of the entire CPU, and the creation of URIs accounts for a larger
> proportion.
> !Before Optimization remark.png!
> h3. Optimization ideas:
> NetUtils.createSocketAddr creates InetSocketAddress based on host and port.
> Here we can add Cache to InetSocketAddress. The key of Cache is host and
> port, and the value is InetSocketAddress.
> h4. After Optimization FlameGraph:
> In the figure, we can see that DFSInputStream.getBestNodeDNAddrPair accounts
> for 0.54% of the entire CPU. Here, ConcurrentHashMap is used as the Cache,
> and the ConcurrentHashMap.get() method gets data from the Cache. The CPU
> usage of DFSInputStream.getBestNodeDNAddrPair has been optimized from 4.86%
> to 0.54%.
> !After Optimization remark.png!
> h3. Original FlameGraph link:
> [Before
> Optimization|https://drive.google.com/file/d/133L5m75u2tu_KgKfGHZLEUzGR0XAfUl6/view?usp=sharing]
> [After Optimization
> FlameGraph|https://drive.google.com/file/d/133L5m75u2tu_KgKfGHZLEUzGR0XAfUl6/view?usp=sharing]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] 1996fanrui commented on pull request #2241: HADOOP-17222. Create socket address combined with cache
1996fanrui commented on pull request #2241: URL: https://github.com/apache/hadoop/pull/2241#issuecomment-688160009 Hi @liuml07 ,Could you have time to help merge it? Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
mukund-thakur commented on a change in pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#discussion_r484259633
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileOperationCost.java
##
@@ -176,6 +176,77 @@ public void testCostOfListFilesOnNonExistingDir() throws
Throwable {
.plus(GET_FILE_STATUS_FNFE)));
}
+ @Test
+ public void testCostOfListStatusOnFile() throws Throwable {
+describe("Performing listStatus() on a file");
+Path file = path(getMethodName() + ".txt");
+S3AFileSystem fs = getFileSystem();
+touch(fs, file);
+verifyMetrics(() ->
+fs.listStatus(file),
+whenRaw(LIST_STATUS_LIST_OP
+.plus(GET_FILE_STATUS_ON_FILE)),
+whenAuthoritative(LIST_STATUS_LIST_OP),
+whenNonauth(LIST_STATUS_LIST_OP));
+//resetMetricDiffs();
+//fs.listStatus(file);
+//if (!fs.hasMetadataStore()) {
+// metadataRequests.assertDiffEquals(1);
+//}
+//listRequests.assertDiffEquals(1);
+ }
+
+ @Test
+ public void testCostOfListStatusOnEmptyDir() throws Throwable {
+describe("Performing listStatus() on an empty dir");
+Path dir = path(getMethodName());
+S3AFileSystem fs = getFileSystem();
+fs.mkdirs(dir);
+verifyMetrics(() ->
+fs.listStatus(dir),
+whenRaw(LIST_STATUS_LIST_OP
Review comment:
Sure Thanks :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17023) Tune listStatus() api of s3a.
[
https://issues.apache.org/jira/browse/HADOOP-17023?focusedWorklogId=479517=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-479517
]
ASF GitHub Bot logged work on HADOOP-17023:
---
Author: ASF GitHub Bot
Created on: 07/Sep/20 08:02
Start Date: 07/Sep/20 08:02
Worklog Time Spent: 10m
Work Description: mukund-thakur commented on a change in pull request
#2257:
URL: https://github.com/apache/hadoop/pull/2257#discussion_r484259633
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileOperationCost.java
##
@@ -176,6 +176,77 @@ public void testCostOfListFilesOnNonExistingDir() throws
Throwable {
.plus(GET_FILE_STATUS_FNFE)));
}
+ @Test
+ public void testCostOfListStatusOnFile() throws Throwable {
+describe("Performing listStatus() on a file");
+Path file = path(getMethodName() + ".txt");
+S3AFileSystem fs = getFileSystem();
+touch(fs, file);
+verifyMetrics(() ->
+fs.listStatus(file),
+whenRaw(LIST_STATUS_LIST_OP
+.plus(GET_FILE_STATUS_ON_FILE)),
+whenAuthoritative(LIST_STATUS_LIST_OP),
+whenNonauth(LIST_STATUS_LIST_OP));
+//resetMetricDiffs();
+//fs.listStatus(file);
+//if (!fs.hasMetadataStore()) {
+// metadataRequests.assertDiffEquals(1);
+//}
+//listRequests.assertDiffEquals(1);
+ }
+
+ @Test
+ public void testCostOfListStatusOnEmptyDir() throws Throwable {
+describe("Performing listStatus() on an empty dir");
+Path dir = path(getMethodName());
+S3AFileSystem fs = getFileSystem();
+fs.mkdirs(dir);
+verifyMetrics(() ->
+fs.listStatus(dir),
+whenRaw(LIST_STATUS_LIST_OP
Review comment:
Sure Thanks :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 479517)
Time Spent: 50m (was: 40m)
> Tune listStatus() api of s3a.
> -
>
> Key: HADOOP-17023
> URL: https://issues.apache.org/jira/browse/HADOOP-17023
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.1
>Reporter: Mukund Thakur
>Assignee: Mukund Thakur
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Similar optimisation which was done for listLocatedSttaus()
> https://issues.apache.org/jira/browse/HADOOP-16465 can done for listStatus()
> api as well.
> This is going to reduce the number of remote calls in case of directory
> listing.
>
> CC [~ste...@apache.org] [~shwethags]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
