[GitHub] [hadoop] snvijaya commented on pull request #2368: Hadoop-17296. ABFS: Force reads to be always of buffer size
snvijaya commented on pull request #2368: URL: https://github.com/apache/hadoop/pull/2368#issuecomment-734069035 @mukund-thakur Have synced with latest trunk. Please have a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2368: Hadoop-17296. ABFS: Force reads to be always of buffer size
snvijaya commented on a change in pull request #2368:
URL: https://github.com/apache/hadoop/pull/2368#discussion_r530769116
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferManager.java
##
@@ -49,21 +51,37 @@
private Queue readAheadQueue = new LinkedList<>(); // queue of
requests that are not picked up by any worker thread yet
private LinkedList inProgressList = new LinkedList<>(); //
requests being processed by worker threads
private LinkedList completedReadList = new LinkedList<>(); //
buffers available for reading
- private static final ReadBufferManager BUFFER_MANAGER; // singleton,
initialized in static initialization block
+ private static ReadBufferManager bufferManager; // singleton, initialized in
static initialization block
+ private static final ReentrantLock LOCK = new ReentrantLock();
- static {
-BUFFER_MANAGER = new ReadBufferManager();
-BUFFER_MANAGER.init();
+ static ReadBufferManager getBufferManager() {
Review comment:
Retaining the change as current change has no functional issues.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2368: Hadoop-17296. ABFS: Force reads to be always of buffer size
snvijaya commented on a change in pull request #2368: URL: https://github.com/apache/hadoop/pull/2368#discussion_r530768877 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java ## @@ -223,16 +244,19 @@ private int readInternal(final long position, final byte[] b, final int offset, // queue read-aheads int numReadAheads = this.readAheadQueueDepth; - long nextSize; long nextOffset = position; + // First read to queue needs to be of readBufferSize and later Review comment: Have synced with @mukund-thakur over mail thread further to clear the understanding. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2368: Hadoop-17296. ABFS: Force reads to be always of buffer size
snvijaya commented on a change in pull request #2368: URL: https://github.com/apache/hadoop/pull/2368#discussion_r530768704 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java ## @@ -223,16 +244,19 @@ private int readInternal(final long position, final byte[] b, final int offset, // queue read-aheads int numReadAheads = this.readAheadQueueDepth; - long nextSize; long nextOffset = position; + // First read to queue needs to be of readBufferSize and later Review comment: Read buffer size config being available to be modified, fixed read ahead buffer size and issuing read aheads by buffer size is the current prod behaviour and will function as the picture attached. This will need fixing. And as for deprecating read buffer size config and only use the new read ahead buffer size config. The config has been available since GA, and hence deprecating it would not be feasible. (Also for clients who are disabling readAheads to use readahead buffer size for reads might be confusing too). As for the number of different configs present for read, 1 and 2 configs already were present while this PR is introducing 3 and 4. So total of 4 configs. 1. fs.azure.read.request.size 2. fs.azure.readaheadqueue.depth 3. fs.azure.read.alwaysReadBufferSize => For Gen1 migrating customers 4. fs.azure.read.readahead.blocksize => Was one that needed fixing long back as there is no validation on 4 MB being the right size for all workloads. Just the way read buffer size can be modified. All these changes are being added based on various customer issues and experiences that we are dealing with. Instead of spending our time in providing patches that can enable them to test various combinations, having these options over a config for their testing saves our dev time to improve the service. As you can see in the PR, the defaults introduced by these configs will retain the current prod behavior.  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2368: Hadoop-17296. ABFS: Force reads to be always of buffer size
hadoop-yetus commented on pull request #2368:
URL: https://github.com/apache/hadoop/pull/2368#issuecomment-734053500
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 31m 5s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 20s | | trunk passed |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 14s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 57s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | | trunk passed |
| -0 :warning: | patch | 1m 12s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 34s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 59s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 25s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 112m 53s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2368/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2368 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux fd8c2773496c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 235947e2822 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2368/7/testReport/ |
| Max. process+thread count | 622 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2368/7/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an automated
[jira] [Work logged] (HADOOP-17397) ABFS: SAS Test updates for version and permission update
[
https://issues.apache.org/jira/browse/HADOOP-17397?focusedWorklogId=516905=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516905
]
ASF GitHub Bot logged work on HADOOP-17397:
---
Author: ASF GitHub Bot
Created on: 26/Nov/20 02:52
Start Date: 26/Nov/20 02:52
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2492:
URL: https://github.com/apache/hadoop/pull/2492#issuecomment-734042721
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 30m 7s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 18s | | trunk passed |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 52s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 55s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 54s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 48s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 57s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 19s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 111m 27s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2492/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2492 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d81a2a34aade 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 235947e2822 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2492/1/testReport/ |
| Max. process+thread count | 515 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2492/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
[GitHub] [hadoop] hadoop-yetus commented on pull request #2492: HADOOP-17397. ABFS: SAS Test updates for version and permission update
hadoop-yetus commented on pull request #2492:
URL: https://github.com/apache/hadoop/pull/2492#issuecomment-734042721
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 30m 7s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 18s | | trunk passed |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 52s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 55s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 54s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 48s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 57s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 19s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 111m 27s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2492/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2492 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d81a2a34aade 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 235947e2822 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2492/1/testReport/ |
| Max. process+thread count | 515 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2492/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (HADOOP-17394) [JDK 11] mvn package -Pdocs fails
[
https://issues.apache.org/jira/browse/HADOOP-17394?focusedWorklogId=516901=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516901
]
ASF GitHub Bot logged work on HADOOP-17394:
---
Author: ASF GitHub Bot
Created on: 26/Nov/20 02:39
Start Date: 26/Nov/20 02:39
Worklog Time Spent: 10m
Work Description: aajisaka commented on pull request #2488:
URL: https://github.com/apache/hadoop/pull/2488#issuecomment-734039447
Thank you @tasanuma !
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 516901)
Time Spent: 50m (was: 40m)
> [JDK 11] mvn package -Pdocs fails
> -
>
> Key: HADOOP-17394
> URL: https://issues.apache.org/jira/browse/HADOOP-17394
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, documentation
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> "mvn package -Pdocs -DskipTests -DskipShade" fails with Java 11.
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-javadoc-plugin:3.0.1:javadoc (default) on
> project hadoop-common: An error has occurred in Javadoc report generation:
> [ERROR] Exit code: 1 - javadoc: warning - You have specified the HTML version
> as HTML 4.01 by using the -html4 option.
> [ERROR] The default is currently HTML5 and the support for HTML 4.01 will be
> removed
> [ERROR] in a future release. To suppress this warning, please ensure that any
> HTML constructs
> [ERROR] in your comments are valid in HTML5, and remove the -html4 option.
> [ERROR]
> /Users/aajisaka/git/hadoop/hadoop-common-project/hadoop-common/target/generated-sources/java/org/apache/hadoop/ipc/protobuf/RpcHeaderProtos.java:3467:
> error: cannot find symbol
> [ERROR] com.google.protobuf.GeneratedMessageV3 implements
> [ERROR] ^
> (snip)
> {noformat}
> The goal "javadoc" should be "javadoc-no-fork".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #2488: HADOOP-17394. [JDK 11] mvn package -Pdocs fails
aajisaka commented on pull request #2488: URL: https://github.com/apache/hadoop/pull/2488#issuecomment-734039447 Thank you @tasanuma ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17394) [JDK 11] mvn package -Pdocs fails
[
https://issues.apache.org/jira/browse/HADOOP-17394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka resolved HADOOP-17394.
Fix Version/s: 3.4.0
3.3.1
Resolution: Fixed
Cherry-picked to branch-3.3.
> [JDK 11] mvn package -Pdocs fails
> -
>
> Key: HADOOP-17394
> URL: https://issues.apache.org/jira/browse/HADOOP-17394
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, documentation
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> "mvn package -Pdocs -DskipTests -DskipShade" fails with Java 11.
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-javadoc-plugin:3.0.1:javadoc (default) on
> project hadoop-common: An error has occurred in Javadoc report generation:
> [ERROR] Exit code: 1 - javadoc: warning - You have specified the HTML version
> as HTML 4.01 by using the -html4 option.
> [ERROR] The default is currently HTML5 and the support for HTML 4.01 will be
> removed
> [ERROR] in a future release. To suppress this warning, please ensure that any
> HTML constructs
> [ERROR] in your comments are valid in HTML5, and remove the -html4 option.
> [ERROR]
> /Users/aajisaka/git/hadoop/hadoop-common-project/hadoop-common/target/generated-sources/java/org/apache/hadoop/ipc/protobuf/RpcHeaderProtos.java:3467:
> error: cannot find symbol
> [ERROR] com.google.protobuf.GeneratedMessageV3 implements
> [ERROR] ^
> (snip)
> {noformat}
> The goal "javadoc" should be "javadoc-no-fork".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17394) [JDK 11] mvn package -Pdocs fails
[
https://issues.apache.org/jira/browse/HADOOP-17394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239022#comment-17239022
]
Akira Ajisaka commented on HADOOP-17394:
Reverted the above commit and added the JIRA ID.
https://github.com/apache/hadoop/commit/2ce2198287c9acade501d2681475fac1e5c27c68
> [JDK 11] mvn package -Pdocs fails
> -
>
> Key: HADOOP-17394
> URL: https://issues.apache.org/jira/browse/HADOOP-17394
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, documentation
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> "mvn package -Pdocs -DskipTests -DskipShade" fails with Java 11.
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-javadoc-plugin:3.0.1:javadoc (default) on
> project hadoop-common: An error has occurred in Javadoc report generation:
> [ERROR] Exit code: 1 - javadoc: warning - You have specified the HTML version
> as HTML 4.01 by using the -html4 option.
> [ERROR] The default is currently HTML5 and the support for HTML 4.01 will be
> removed
> [ERROR] in a future release. To suppress this warning, please ensure that any
> HTML constructs
> [ERROR] in your comments are valid in HTML5, and remove the -html4 option.
> [ERROR]
> /Users/aajisaka/git/hadoop/hadoop-common-project/hadoop-common/target/generated-sources/java/org/apache/hadoop/ipc/protobuf/RpcHeaderProtos.java:3467:
> error: cannot find symbol
> [ERROR] com.google.protobuf.GeneratedMessageV3 implements
> [ERROR] ^
> (snip)
> {noformat}
> The goal "javadoc" should be "javadoc-no-fork".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17394) [JDK 11] mvn package -Pdocs fails
[
https://issues.apache.org/jira/browse/HADOOP-17394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239020#comment-17239020
]
Akira Ajisaka commented on HADOOP-17394:
Merged the PR into trunk. Sorry I forgot to add the JIRA ID when committing.
https://github.com/apache/hadoop/commit/ce5b3d7975520e013c17aef5b5fd3a5c29f0e381
> [JDK 11] mvn package -Pdocs fails
> -
>
> Key: HADOOP-17394
> URL: https://issues.apache.org/jira/browse/HADOOP-17394
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, documentation
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> "mvn package -Pdocs -DskipTests -DskipShade" fails with Java 11.
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-javadoc-plugin:3.0.1:javadoc (default) on
> project hadoop-common: An error has occurred in Javadoc report generation:
> [ERROR] Exit code: 1 - javadoc: warning - You have specified the HTML version
> as HTML 4.01 by using the -html4 option.
> [ERROR] The default is currently HTML5 and the support for HTML 4.01 will be
> removed
> [ERROR] in a future release. To suppress this warning, please ensure that any
> HTML constructs
> [ERROR] in your comments are valid in HTML5, and remove the -html4 option.
> [ERROR]
> /Users/aajisaka/git/hadoop/hadoop-common-project/hadoop-common/target/generated-sources/java/org/apache/hadoop/ipc/protobuf/RpcHeaderProtos.java:3467:
> error: cannot find symbol
> [ERROR] com.google.protobuf.GeneratedMessageV3 implements
> [ERROR] ^
> (snip)
> {noformat}
> The goal "javadoc" should be "javadoc-no-fork".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17394) [JDK 11] mvn package -Pdocs fails
[
https://issues.apache.org/jira/browse/HADOOP-17394?focusedWorklogId=516900=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516900
]
ASF GitHub Bot logged work on HADOOP-17394:
---
Author: ASF GitHub Bot
Created on: 26/Nov/20 02:27
Start Date: 26/Nov/20 02:27
Worklog Time Spent: 10m
Work Description: aajisaka merged pull request #2488:
URL: https://github.com/apache/hadoop/pull/2488
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 516900)
Time Spent: 40m (was: 0.5h)
> [JDK 11] mvn package -Pdocs fails
> -
>
> Key: HADOOP-17394
> URL: https://issues.apache.org/jira/browse/HADOOP-17394
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, documentation
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> "mvn package -Pdocs -DskipTests -DskipShade" fails with Java 11.
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-javadoc-plugin:3.0.1:javadoc (default) on
> project hadoop-common: An error has occurred in Javadoc report generation:
> [ERROR] Exit code: 1 - javadoc: warning - You have specified the HTML version
> as HTML 4.01 by using the -html4 option.
> [ERROR] The default is currently HTML5 and the support for HTML 4.01 will be
> removed
> [ERROR] in a future release. To suppress this warning, please ensure that any
> HTML constructs
> [ERROR] in your comments are valid in HTML5, and remove the -html4 option.
> [ERROR]
> /Users/aajisaka/git/hadoop/hadoop-common-project/hadoop-common/target/generated-sources/java/org/apache/hadoop/ipc/protobuf/RpcHeaderProtos.java:3467:
> error: cannot find symbol
> [ERROR] com.google.protobuf.GeneratedMessageV3 implements
> [ERROR] ^
> (snip)
> {noformat}
> The goal "javadoc" should be "javadoc-no-fork".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka merged pull request #2488: HADOOP-17394. [JDK 11] mvn package -Pdocs fails
aajisaka merged pull request #2488: URL: https://github.com/apache/hadoop/pull/2488 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on pull request #2368: Hadoop-17296. ABFS: Force reads to be always of buffer size
snvijaya commented on pull request #2368: URL: https://github.com/apache/hadoop/pull/2368#issuecomment-734024756 ### HNS-OAuth [INFO] Results: [INFO] [INFO] Tests run: 89, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 446, Failures: 0, Errors: 0, Skipped: 66 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 141 ### NonHNS-OAuth [INFO] Results: [INFO] [INFO] Tests run: 89, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 458, Failures: 0, Errors: 0, Skipped: 249 [INFO] Results: [INFO] [WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 141 ### HNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 89, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemDelegationSAS.testCheckAccess:94 » AbfsRestOperation O... [ERROR] ITestAzureBlobFileSystemDelegationSAS.testRootPath:343 » AbfsRestOperation Ope... [INFO] [ERROR] Tests run: 446, Failures: 0, Errors: 2, Skipped: 24 [INFO] Results: [INFO] [ERROR] Failures: [ERROR] ITestAbfsFileSystemContractRename>AbstractContractRenameTest.testRenameFileOverExistingFile:131->Assert.fail:88 expected rename(abfs://gen2contractt...@snvijayacontracttest.dfs.core.windows.net/fork-6/test/source-256.txt, abfs://gen2contractt...@snvijayacontracttest.dfs.core.windows.net/fork-6/test/dest-512.txt) to be rejected with exception, but got false [INFO] [ERROR] Tests run: 196, Failures: 1, Errors: 0, Skipped: 24 ### NonHNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 89, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 424, Failures: 0, Errors: 0, Skipped: 242 [INFO] Results: [INFO] [ERROR] Failures: [ERROR] ITestAbfsFileSystemContractRename>AbstractContractRenameTest.testRenameFileOverExistingFile:131->Assert.fail:88 expected rename(abfs://gen2contractt...@snvijayanonhnstest.dfs.core.windows.net/fork-7/test/source-256.txt, abfs://gen2contractt...@snvijayanonhnstest.dfs.core.windows.net/fork-7/test/dest-512.txt) to be rejected with exception, but got false [INFO] [ERROR] Tests run: 208, Failures: 1, Errors: 0, Skipped: 24 Rename Test failure is tracked in JIRA: https://issues.apache.org/jira/browse/HADOOP-17396 SAS Test failure is tracked in JIRA: https://issues.apache.org/jira/browse/HADOOP-17397 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17397) ABFS: SAS Test updates for version and permission update
[ https://issues.apache.org/jira/browse/HADOOP-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sneha Vijayarajan updated HADOOP-17397: --- Status: Patch Available (was: Open) > ABFS: SAS Test updates for version and permission update > > > Key: HADOOP-17397 > URL: https://issues.apache.org/jira/browse/HADOOP-17397 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Fix For: 3.3.0 > > Time Spent: 20m > Remaining Estimate: 0h > > This Jira will track the below 2 updates to SAS test code: > # Upgrading the SAS version in Service SAS generator (test code) > # Updating the permission in Delegation SAS to "op" from "p" for ACL > operation as identities added as suoid/saoid added by tests are not owners of > test path (Again test code). > [Relevant public documentation: > https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview|https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17397) ABFS: SAS Test updates for version and permission update
[ https://issues.apache.org/jira/browse/HADOOP-17397?focusedWorklogId=516878=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516878 ] ASF GitHub Bot logged work on HADOOP-17397: --- Author: ASF GitHub Bot Created on: 26/Nov/20 01:01 Start Date: 26/Nov/20 01:01 Worklog Time Spent: 10m Work Description: snvijaya commented on pull request #2492: URL: https://github.com/apache/hadoop/pull/2492#issuecomment-734014552 ### HNS-OAuth [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 445, Failures: 0, Errors: 0, Skipped: 66 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 141 ### NonHNS-OAuth [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 249 [INFO] Results: [INFO] [WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 141 ### HNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 445, Failures: 0, Errors: 0, Skipped: 24 [INFO] Results: [INFO] [ERROR] Failures: [ERROR] ITestAbfsFileSystemContractRename>AbstractContractRenameTest.testRenameFileOverExistingFile:131->Assert.fail:88 expected rename(abfs://gen2contractt...@snvijayacontracttest.dfs.core.windows.net/fork-8/test/source-256.txt, abfs://gen2contractt...@snvijayacontracttest.dfs.core.windows.net/fork-8/test/dest-512.txt) to be rejected with exception, but got false [INFO] [ERROR] Tests run: 198, Failures: 1, Errors: 0, Skipped: 24 ### NonHNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 440, Failures: 0, Errors: 0, Skipped: 243 [INFO] Results: [INFO] [ERROR] Failures: [ERROR] ITestAbfsFileSystemContractRename>AbstractContractRenameTest.testRenameFileOverExistingFile:131->Assert.fail:88 expected rename(abfs://gen2contractt...@snvijayanonhnstest.dfs.core.windows.net/fork-4/test/source-256.txt, abfs://gen2contractt...@snvijayanonhnstest.dfs.core.windows.net/fork-4/test/dest-512.txt) to be rejected with exception, but got false [INFO] [ERROR] Tests run: 208, Failures: 1, Errors: 0, Skipped: 24 Rename Test failure is tracked in JIRA: https://issues.apache.org/jira/browse/HADOOP-17396 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516878) Time Spent: 20m (was: 10m) > ABFS: SAS Test updates for version and permission update > > > Key: HADOOP-17397 > URL: https://issues.apache.org/jira/browse/HADOOP-17397 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Fix For: 3.3.0 > > Time Spent: 20m > Remaining Estimate: 0h > > This Jira will track the below 2 updates to SAS test code: > # Upgrading the SAS version in Service SAS generator (test code) > # Updating the permission in Delegation SAS to "op" from "p" for ACL > operation as identities added as suoid/saoid added by tests are not owners of > test path (Again test code). > [Relevant public documentation: > https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview|https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on pull request #2492: HADOOP-17397. ABFS: SAS Test updates for version and permission update
snvijaya commented on pull request #2492: URL: https://github.com/apache/hadoop/pull/2492#issuecomment-734014552 ### HNS-OAuth [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 445, Failures: 0, Errors: 0, Skipped: 66 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 141 ### NonHNS-OAuth [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 249 [INFO] Results: [INFO] [WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 141 ### HNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 445, Failures: 0, Errors: 0, Skipped: 24 [INFO] Results: [INFO] [ERROR] Failures: [ERROR] ITestAbfsFileSystemContractRename>AbstractContractRenameTest.testRenameFileOverExistingFile:131->Assert.fail:88 expected rename(abfs://gen2contractt...@snvijayacontracttest.dfs.core.windows.net/fork-8/test/source-256.txt, abfs://gen2contractt...@snvijayacontracttest.dfs.core.windows.net/fork-8/test/dest-512.txt) to be rejected with exception, but got false [INFO] [ERROR] Tests run: 198, Failures: 1, Errors: 0, Skipped: 24 ### NonHNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 440, Failures: 0, Errors: 0, Skipped: 243 [INFO] Results: [INFO] [ERROR] Failures: [ERROR] ITestAbfsFileSystemContractRename>AbstractContractRenameTest.testRenameFileOverExistingFile:131->Assert.fail:88 expected rename(abfs://gen2contractt...@snvijayanonhnstest.dfs.core.windows.net/fork-4/test/source-256.txt, abfs://gen2contractt...@snvijayanonhnstest.dfs.core.windows.net/fork-4/test/dest-512.txt) to be rejected with exception, but got false [INFO] [ERROR] Tests run: 208, Failures: 1, Errors: 0, Skipped: 24 Rename Test failure is tracked in JIRA: https://issues.apache.org/jira/browse/HADOOP-17396 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17397) ABFS: SAS Test updates for version and permission update
[ https://issues.apache.org/jira/browse/HADOOP-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-17397: Labels: pull-request-available (was: ) > ABFS: SAS Test updates for version and permission update > > > Key: HADOOP-17397 > URL: https://issues.apache.org/jira/browse/HADOOP-17397 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Fix For: 3.3.0 > > Time Spent: 10m > Remaining Estimate: 0h > > This Jira will track the below 2 updates to SAS test code: > # Upgrading the SAS version in Service SAS generator (test code) > # Updating the permission in Delegation SAS to "op" from "p" for ACL > operation as identities added as suoid/saoid added by tests are not owners of > test path (Again test code). > [Relevant public documentation: > https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview|https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17397) ABFS: SAS Test updates for version and permission update
[ https://issues.apache.org/jira/browse/HADOOP-17397?focusedWorklogId=516876=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516876 ] ASF GitHub Bot logged work on HADOOP-17397: --- Author: ASF GitHub Bot Created on: 26/Nov/20 00:59 Start Date: 26/Nov/20 00:59 Worklog Time Spent: 10m Work Description: snvijaya opened a new pull request #2492: URL: https://github.com/apache/hadoop/pull/2492 Below updates are made: 1. Upgrading the SAS version in Service SAS generator to Feb20 which is the latest server supported version (test code) 2. Updating the permission in Delegation SAS to "op" from "p" for ACL operation as identities added as suoid/saoid added by tests are not owners of test path (Again test code). Relevant public documentation: https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview Tests were re-run. Test results posted at end of conversation tab. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516876) Remaining Estimate: 0h Time Spent: 10m > ABFS: SAS Test updates for version and permission update > > > Key: HADOOP-17397 > URL: https://issues.apache.org/jira/browse/HADOOP-17397 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Fix For: 3.3.0 > > Time Spent: 10m > Remaining Estimate: 0h > > This Jira will track the below 2 updates to SAS test code: > # Upgrading the SAS version in Service SAS generator (test code) > # Updating the permission in Delegation SAS to "op" from "p" for ACL > operation as identities added as suoid/saoid added by tests are not owners of > test path (Again test code). > [Relevant public documentation: > https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview|https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya opened a new pull request #2492: HADOOP-17397. ABFS: SAS Test updates for version and permission update
snvijaya opened a new pull request #2492: URL: https://github.com/apache/hadoop/pull/2492 Below updates are made: 1. Upgrading the SAS version in Service SAS generator to Feb20 which is the latest server supported version (test code) 2. Updating the permission in Delegation SAS to "op" from "p" for ACL operation as identities added as suoid/saoid added by tests are not owners of test path (Again test code). Relevant public documentation: https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview Tests were re-run. Test results posted at end of conversation tab. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17397) ABFS: SAS Test updates for version and permission update
[ https://issues.apache.org/jira/browse/HADOOP-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sneha Vijayarajan updated HADOOP-17397: --- Description: This Jira will track the below 2 updates to SAS test code: # Upgrading the SAS version in Service SAS generator (test code) # Updating the permission in Delegation SAS to "op" from "p" for ACL operation as identities added as suoid/saoid added by tests are not owners of test path (Again test code). [Relevant public documentation: https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview|https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview] was: This Jira will track the below 2 updates to SAS test code: # Upgrading the SAS version # Updating the permission in SAS to "op" from "p" for ACL operation as identities added as suoid/saoid added by tests are not owners of test path [Relevant public documentation: https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview|https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview] > ABFS: SAS Test updates for version and permission update > > > Key: HADOOP-17397 > URL: https://issues.apache.org/jira/browse/HADOOP-17397 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Fix For: 3.3.0 > > > This Jira will track the below 2 updates to SAS test code: > # Upgrading the SAS version in Service SAS generator (test code) > # Updating the permission in Delegation SAS to "op" from "p" for ACL > operation as identities added as suoid/saoid added by tests are not owners of > test path (Again test code). > [Relevant public documentation: > https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview|https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17396) ABFS: testRenameFileOverExistingFile Fails after Contract test update
[ https://issues.apache.org/jira/browse/HADOOP-17396?focusedWorklogId=516874=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516874 ] ASF GitHub Bot logged work on HADOOP-17396: --- Author: ASF GitHub Bot Created on: 26/Nov/20 00:55 Start Date: 26/Nov/20 00:55 Worklog Time Spent: 10m Work Description: snvijaya edited a comment on pull request #2491: URL: https://github.com/apache/hadoop/pull/2491#issuecomment-733963193 ### HNS-OAuth [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 66 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 141 ### HNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemDelegationSAS.testCheckAccess:94 » AbfsRestOperation O... [ERROR] ITestAzureBlobFileSystemDelegationSAS.testRootPath:343 » AbfsRestOperation Ope... [INFO] [ERROR] Tests run: 457, Failures: 0, Errors: 2, Skipped: 24 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 ### NonHNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 245 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 SAS Test failure is tracked in JIRA: https://issues.apache.org/jira/browse/HADOOP-17397 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516874) Time Spent: 40m (was: 0.5h) > ABFS: testRenameFileOverExistingFile Fails after Contract test update > - > > Key: HADOOP-17396 > URL: https://issues.apache.org/jira/browse/HADOOP-17396 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Post updates to rename on existing file test, ABFS contract test is having > failure. > Updates were made in the AbstractContractTest class in > https://issues.apache.org/jira/browse/HADOOP-17365. > To align to test expectation, ABFS tests need config > "fs.contract.rename-returns-false-if-dest-exists" set to true. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17396) ABFS: testRenameFileOverExistingFile Fails after Contract test update
[ https://issues.apache.org/jira/browse/HADOOP-17396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sneha Vijayarajan updated HADOOP-17396: --- Status: Patch Available (was: Open) > ABFS: testRenameFileOverExistingFile Fails after Contract test update > - > > Key: HADOOP-17396 > URL: https://issues.apache.org/jira/browse/HADOOP-17396 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Post updates to rename on existing file test, ABFS contract test is having > failure. > Updates were made in the AbstractContractTest class in > https://issues.apache.org/jira/browse/HADOOP-17365. > To align to test expectation, ABFS tests need config > "fs.contract.rename-returns-false-if-dest-exists" set to true. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya edited a comment on pull request #2491: HADOOP-17396. ABFS: testRenameFileOverExistingFile Fails after Contract test update
snvijaya edited a comment on pull request #2491: URL: https://github.com/apache/hadoop/pull/2491#issuecomment-733963193 ### HNS-OAuth [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 66 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 141 ### HNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemDelegationSAS.testCheckAccess:94 » AbfsRestOperation O... [ERROR] ITestAzureBlobFileSystemDelegationSAS.testRootPath:343 » AbfsRestOperation Ope... [INFO] [ERROR] Tests run: 457, Failures: 0, Errors: 2, Skipped: 24 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 ### NonHNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 245 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 SAS Test failure is tracked in JIRA: https://issues.apache.org/jira/browse/HADOOP-17397 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17265) ABFS: Support for Client Correlation ID
[ https://issues.apache.org/jira/browse/HADOOP-17265?focusedWorklogId=516866=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516866 ] ASF GitHub Bot logged work on HADOOP-17265: --- Author: ASF GitHub Bot Created on: 26/Nov/20 00:02 Start Date: 26/Nov/20 00:02 Worklog Time Spent: 10m Work Description: sumangala-patki commented on pull request #2344: URL: https://github.com/apache/hadoop/pull/2344#issuecomment-733998187 TEST RESULTS HNS-Enabled Account (Location: East US 2) ``` Authentication type: SharedKey [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemDelegationSAS.testCheckAccess:94 ? AbfsRestOperation O... [ERROR] ITestAzureBlobFileSystemDelegationSAS.testRootPath:343 ? AbfsRestOperation Ope... [ERROR] Tests run: 458, Failures: 0, Errors: 2, Skipped: 24 [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24 (CheckAccess: [WARNING] Tests run: 12, Failures: 0, Errors: 0, Skipped: 1) (SAS: [ERROR] Tests run: 9, Failures: 0, Errors: 2, Skipped: 0) Authentication type: OAuth [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 458, Failures: 0, Errors: 0, Skipped: 66 [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 140 (CheckAccess: [WARNING] Tests run: 12, Failures: 0, Errors: 0, Skipped: 1) (SAS: [WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 9) ``` HNS-Disabled Account (Location: East US 2, Central US) ``` Authentication type: SharedKey [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 458, Failures: 0, Errors: 0, Skipped: 245 [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24 (CheckAccess: [WARNING] Tests run: 12, Failures: 0, Errors: 0, Skipped: 10) (SAS: [WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 9) Authentication type: OAuth [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 458, Failures: 0, Errors: 0, Skipped: 245 [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24 (CheckAccess: [WARNING] Tests run: 12, Failures: 0, Errors: 0, Skipped: 10) (SAS: [WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 9) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516866) Time Spent: 3h 10m (was: 3h) > ABFS: Support for Client Correlation ID > --- > > Key: HADOOP-17265 > URL: https://issues.apache.org/jira/browse/HADOOP-17265 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sumangala Patki >Priority: Major > Labels: abfsactive, pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > Introducing a client correlation ID that appears in the Azure Storage > diagnostic logs. This will modify the client request header to include the > client-provided identifier for correlating requests. A valid correlation ID > comprises alphanumeric characters and/or hyphens, and can be up to 72 > characters in length. Invalid or missing entries for this configuration are > substituted with the empty string default. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on pull request #2344: HADOOP-17265. ABFS: Support for Client Correlation ID
sumangala-patki commented on pull request #2344: URL: https://github.com/apache/hadoop/pull/2344#issuecomment-733998187 TEST RESULTS HNS-Enabled Account (Location: East US 2) ``` Authentication type: SharedKey [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemDelegationSAS.testCheckAccess:94 ? AbfsRestOperation O... [ERROR] ITestAzureBlobFileSystemDelegationSAS.testRootPath:343 ? AbfsRestOperation Ope... [ERROR] Tests run: 458, Failures: 0, Errors: 2, Skipped: 24 [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24 (CheckAccess: [WARNING] Tests run: 12, Failures: 0, Errors: 0, Skipped: 1) (SAS: [ERROR] Tests run: 9, Failures: 0, Errors: 2, Skipped: 0) Authentication type: OAuth [INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 458, Failures: 0, Errors: 0, Skipped: 66 [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 140 (CheckAccess: [WARNING] Tests run: 12, Failures: 0, Errors: 0, Skipped: 1) (SAS: [WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 9) ``` HNS-Disabled Account (Location: East US 2, Central US) ``` Authentication type: SharedKey [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 458, Failures: 0, Errors: 0, Skipped: 245 [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24 (CheckAccess: [WARNING] Tests run: 12, Failures: 0, Errors: 0, Skipped: 10) (SAS: [WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 9) Authentication type: OAuth [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 458, Failures: 0, Errors: 0, Skipped: 245 [WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24 (CheckAccess: [WARNING] Tests run: 12, Failures: 0, Errors: 0, Skipped: 10) (SAS: [WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 9) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17396) ABFS: testRenameFileOverExistingFile Fails after Contract test update
[
https://issues.apache.org/jira/browse/HADOOP-17396?focusedWorklogId=516858=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516858
]
ASF GitHub Bot logged work on HADOOP-17396:
---
Author: ASF GitHub Bot
Created on: 25/Nov/20 23:17
Start Date: 25/Nov/20 23:17
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2491:
URL: https://github.com/apache/hadoop/pull/2491#issuecomment-733986954
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 21s | | trunk passed |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 8s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 57s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 54s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 16m 53s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 56s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 23s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 83m 15s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2491/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2491 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 7105c78e7057 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 235947e2822 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2491/1/testReport/ |
| Max. process+thread count | 570 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2491/1/console |
| versions | git=2.17.1 maven=3.6.0
[GitHub] [hadoop] hadoop-yetus commented on pull request #2491: HADOOP-17396. ABFS: testRenameFileOverExistingFile Fails after Contract test update
hadoop-yetus commented on pull request #2491:
URL: https://github.com/apache/hadoop/pull/2491#issuecomment-733986954
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 21s | | trunk passed |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 8s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 57s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 54s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 16m 53s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 56s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 23s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 83m 15s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2491/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2491 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 7105c78e7057 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 235947e2822 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2491/1/testReport/ |
| Max. process+thread count | 570 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2491/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact
[jira] [Created] (HADOOP-17397) ABFS: SAS Test updates for version and permission update
Sneha Vijayarajan created HADOOP-17397: -- Summary: ABFS: SAS Test updates for version and permission update Key: HADOOP-17397 URL: https://issues.apache.org/jira/browse/HADOOP-17397 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.0 Reporter: Sneha Vijayarajan Assignee: Sneha Vijayarajan Fix For: 3.3.0 This Jira will track the below 2 updates to SAS test code: # Upgrading the SAS version # Updating the permission in SAS to "op" from "p" for ACL operation as identities added as suoid/saoid added by tests are not owners of test path [Relevant public documentation: https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview|https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17396) ABFS: testRenameFileOverExistingFile Fails after Contract test update
[ https://issues.apache.org/jira/browse/HADOOP-17396?focusedWorklogId=516847=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516847 ] ASF GitHub Bot logged work on HADOOP-17396: --- Author: ASF GitHub Bot Created on: 25/Nov/20 21:55 Start Date: 25/Nov/20 21:55 Worklog Time Spent: 10m Work Description: snvijaya commented on pull request #2491: URL: https://github.com/apache/hadoop/pull/2491#issuecomment-733963193 ### HNS-OAuth [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 66 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 141 ### HNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemDelegationSAS.testCheckAccess:94 » AbfsRestOperation O... [ERROR] ITestAzureBlobFileSystemDelegationSAS.testRootPath:343 » AbfsRestOperation Ope... [INFO] [ERROR] Tests run: 457, Failures: 0, Errors: 2, Skipped: 24 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 ### NonHNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 245 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516847) Time Spent: 20m (was: 10m) > ABFS: testRenameFileOverExistingFile Fails after Contract test update > - > > Key: HADOOP-17396 > URL: https://issues.apache.org/jira/browse/HADOOP-17396 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Post updates to rename on existing file test, ABFS contract test is having > failure. > Updates were made in the AbstractContractTest class in > https://issues.apache.org/jira/browse/HADOOP-17365. > To align to test expectation, ABFS tests need config > "fs.contract.rename-returns-false-if-dest-exists" set to true. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on pull request #2491: HADOOP-17396. ABFS: testRenameFileOverExistingFile Fails after Contract test update
snvijaya commented on pull request #2491: URL: https://github.com/apache/hadoop/pull/2491#issuecomment-733963193 ### HNS-OAuth [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 66 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 141 ### HNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemDelegationSAS.testCheckAccess:94 » AbfsRestOperation O... [ERROR] ITestAzureBlobFileSystemDelegationSAS.testRootPath:343 » AbfsRestOperation Ope... [INFO] [ERROR] Tests run: 457, Failures: 0, Errors: 2, Skipped: 24 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 ### NonHNS-SharedKey [INFO] Results: [INFO] [INFO] Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 [INFO] Results: [INFO] [WARNING] Tests run: 457, Failures: 0, Errors: 0, Skipped: 245 [INFO] Results: [INFO] [WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17396) ABFS: testRenameFileOverExistingFile Fails after Contract test update
[ https://issues.apache.org/jira/browse/HADOOP-17396?focusedWorklogId=516846=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516846 ] ASF GitHub Bot logged work on HADOOP-17396: --- Author: ASF GitHub Bot Created on: 25/Nov/20 21:53 Start Date: 25/Nov/20 21:53 Worklog Time Spent: 10m Work Description: snvijaya opened a new pull request #2491: URL: https://github.com/apache/hadoop/pull/2491 Post updates to rename on existing file test, ABFS contract test is having failure. Updates were made in the AbstractContractTest class in https://issues.apache.org/jira/browse/HADOOP-17365. To align to test expectation of no exception but a false return from rename, ABFS tests need config "fs.contract.rename-returns-false-if-dest-exists" set to true. Also the abstract class was reusing a static test folder path. Hence on test re-run, where same storage container is used, source and destination path creations will fail with FileAlreadyExists, leading to test failures. This is also fixed. [Test results pasted at the end of conversation tab]. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516846) Remaining Estimate: 0h Time Spent: 10m > ABFS: testRenameFileOverExistingFile Fails after Contract test update > - > > Key: HADOOP-17396 > URL: https://issues.apache.org/jira/browse/HADOOP-17396 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Post updates to rename on existing file test, ABFS contract test is having > failure. > Updates were made in the AbstractContractTest class in > https://issues.apache.org/jira/browse/HADOOP-17365. > To align to test expectation, ABFS tests need config > "fs.contract.rename-returns-false-if-dest-exists" set to true. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17396) ABFS: testRenameFileOverExistingFile Fails after Contract test update
[ https://issues.apache.org/jira/browse/HADOOP-17396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-17396: Labels: pull-request-available (was: ) > ABFS: testRenameFileOverExistingFile Fails after Contract test update > - > > Key: HADOOP-17396 > URL: https://issues.apache.org/jira/browse/HADOOP-17396 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Post updates to rename on existing file test, ABFS contract test is having > failure. > Updates were made in the AbstractContractTest class in > https://issues.apache.org/jira/browse/HADOOP-17365. > To align to test expectation, ABFS tests need config > "fs.contract.rename-returns-false-if-dest-exists" set to true. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya opened a new pull request #2491: HADOOP-17396. ABFS: testRenameFileOverExistingFile Fails after Contract test update
snvijaya opened a new pull request #2491: URL: https://github.com/apache/hadoop/pull/2491 Post updates to rename on existing file test, ABFS contract test is having failure. Updates were made in the AbstractContractTest class in https://issues.apache.org/jira/browse/HADOOP-17365. To align to test expectation of no exception but a false return from rename, ABFS tests need config "fs.contract.rename-returns-false-if-dest-exists" set to true. Also the abstract class was reusing a static test folder path. Hence on test re-run, where same storage container is used, source and destination path creations will fail with FileAlreadyExists, leading to test failures. This is also fixed. [Test results pasted at the end of conversation tab]. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17396) ABFS: testRenameFileOverExistingFile Fails after Contract test update
[ https://issues.apache.org/jira/browse/HADOOP-17396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sneha Vijayarajan updated HADOOP-17396: --- Description: Post updates to rename on existing file test, ABFS contract test is having failure. Updates were made in the AbstractContractTest class in https://issues.apache.org/jira/browse/HADOOP-17365. To align to test expectation, ABFS tests need config "fs.contract.rename-returns-false-if-dest-exists" set to true. was: Post updates to rename on existing file test, ABFS contract test is having failure. Updates made in https://issues.apache.org/jira/browse/HADOOP-17365. To align to expecation, ABFS test config needs "fs.contract.rename-returns-false-if-dest-exists" set to true. > ABFS: testRenameFileOverExistingFile Fails after Contract test update > - > > Key: HADOOP-17396 > URL: https://issues.apache.org/jira/browse/HADOOP-17396 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > > Post updates to rename on existing file test, ABFS contract test is having > failure. > Updates were made in the AbstractContractTest class in > https://issues.apache.org/jira/browse/HADOOP-17365. > To align to test expectation, ABFS tests need config > "fs.contract.rename-returns-false-if-dest-exists" set to true. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16080) hadoop-aws does not work with hadoop-client-api
[
https://issues.apache.org/jira/browse/HADOOP-16080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238930#comment-17238930
]
Dongjoon Hyun commented on HADOOP-16080:
Hi, All.
Is there any update on this JIRA? Otherwise, can we increase the `Priority`
from `Major` to `Blocker`?
> hadoop-aws does not work with hadoop-client-api
> ---
>
> Key: HADOOP-16080
> URL: https://issues.apache.org/jira/browse/HADOOP-16080
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.2.0, 3.1.1
>Reporter: Keith Turner
>Priority: Major
>
> I attempted to use Accumulo and S3a with the following jars on the classpath.
> * hadoop-client-api-3.1.1.jar
> * hadoop-client-runtime-3.1.1.jar
> * hadoop-aws-3.1.1.jar
> This failed with the following exception.
> {noformat}
> Exception in thread "init" java.lang.NoSuchMethodError:
> org.apache.hadoop.util.SemaphoredDelegatingExecutor.(Lcom/google/common/util/concurrent/ListeningExecutorService;IZ)V
> at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:769)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1108)
> at org.apache.hadoop.fs.FileSystem.createNewFile(FileSystem.java:1413)
> at
> org.apache.accumulo.server.fs.VolumeManagerImpl.createNewFile(VolumeManagerImpl.java:184)
> at
> org.apache.accumulo.server.init.Initialize.initDirs(Initialize.java:479)
> at
> org.apache.accumulo.server.init.Initialize.initFileSystem(Initialize.java:487)
> at
> org.apache.accumulo.server.init.Initialize.initialize(Initialize.java:370)
> at org.apache.accumulo.server.init.Initialize.doInit(Initialize.java:348)
> at org.apache.accumulo.server.init.Initialize.execute(Initialize.java:967)
> at org.apache.accumulo.start.Main.lambda$execKeyword$0(Main.java:129)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The problem is that {{S3AFileSystem.create()}} looks for
> {{SemaphoredDelegatingExecutor(com.google.common.util.concurrent.ListeningExecutorService)}}
> which does not exist in hadoop-client-api-3.1.1.jar. What does exist is
> {{SemaphoredDelegatingExecutor(org.apache.hadoop.shaded.com.google.common.util.concurrent.ListeningExecutorService)}}.
> To work around this issue I created a version of hadoop-aws-3.1.1.jar that
> relocated references to Guava.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16080) hadoop-aws does not work with hadoop-client-api
[
https://issues.apache.org/jira/browse/HADOOP-16080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated HADOOP-16080:
---
Affects Version/s: 3.2.0
> hadoop-aws does not work with hadoop-client-api
> ---
>
> Key: HADOOP-16080
> URL: https://issues.apache.org/jira/browse/HADOOP-16080
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.2.0, 3.1.1
>Reporter: Keith Turner
>Priority: Major
>
> I attempted to use Accumulo and S3a with the following jars on the classpath.
> * hadoop-client-api-3.1.1.jar
> * hadoop-client-runtime-3.1.1.jar
> * hadoop-aws-3.1.1.jar
> This failed with the following exception.
> {noformat}
> Exception in thread "init" java.lang.NoSuchMethodError:
> org.apache.hadoop.util.SemaphoredDelegatingExecutor.(Lcom/google/common/util/concurrent/ListeningExecutorService;IZ)V
> at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:769)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1108)
> at org.apache.hadoop.fs.FileSystem.createNewFile(FileSystem.java:1413)
> at
> org.apache.accumulo.server.fs.VolumeManagerImpl.createNewFile(VolumeManagerImpl.java:184)
> at
> org.apache.accumulo.server.init.Initialize.initDirs(Initialize.java:479)
> at
> org.apache.accumulo.server.init.Initialize.initFileSystem(Initialize.java:487)
> at
> org.apache.accumulo.server.init.Initialize.initialize(Initialize.java:370)
> at org.apache.accumulo.server.init.Initialize.doInit(Initialize.java:348)
> at org.apache.accumulo.server.init.Initialize.execute(Initialize.java:967)
> at org.apache.accumulo.start.Main.lambda$execKeyword$0(Main.java:129)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The problem is that {{S3AFileSystem.create()}} looks for
> {{SemaphoredDelegatingExecutor(com.google.common.util.concurrent.ListeningExecutorService)}}
> which does not exist in hadoop-client-api-3.1.1.jar. What does exist is
> {{SemaphoredDelegatingExecutor(org.apache.hadoop.shaded.com.google.common.util.concurrent.ListeningExecutorService)}}.
> To work around this issue I created a version of hadoop-aws-3.1.1.jar that
> relocated references to Guava.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17396) ABFS: testRenameFileOverExistingFile Fails after Contract test update
Sneha Vijayarajan created HADOOP-17396: -- Summary: ABFS: testRenameFileOverExistingFile Fails after Contract test update Key: HADOOP-17396 URL: https://issues.apache.org/jira/browse/HADOOP-17396 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.0 Reporter: Sneha Vijayarajan Assignee: Sneha Vijayarajan Post updates to rename on existing file test, ABFS contract test is having failure. Updates made in https://issues.apache.org/jira/browse/HADOOP-17365. To align to expecation, ABFS test config needs "fs.contract.rename-returns-false-if-dest-exists" set to true. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl edited a comment on pull request #2472: HDFS-15689. allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check
smengcl edited a comment on pull request #2472: URL: https://github.com/apache/hadoop/pull/2472#issuecomment-733275478 > Thanks @smengcl . I think, we will run into the same problem while creating an encryption zone on a snapshottable dir and may need a similar fix there. Can you plz check? The reverse won't be a problem because encryption zone can only be created on empty directories. Once snapshot is enabled (`dfs.namenode.snapshot.trashroot.enabled=true`), `.Trash` will prevent encryption zone to be enabled on that directory unless the admin manually removes it and then do it again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl merged pull request #2472: HDFS-15689. allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check
smengcl merged pull request #2472: URL: https://github.com/apache/hadoop/pull/2472 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl commented on pull request #2472: HDFS-15689. allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check
smengcl commented on pull request #2472: URL: https://github.com/apache/hadoop/pull/2472#issuecomment-733895032 Thanks for the review @bshashikant . Will commit shortly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17313) FileSystem.get to support slow-to-instantiate FS clients
[ https://issues.apache.org/jira/browse/HADOOP-17313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17313. - Fix Version/s: 3.3.1 Resolution: Fixed > FileSystem.get to support slow-to-instantiate FS clients > > > Key: HADOOP-17313 > URL: https://issues.apache.org/jira/browse/HADOOP-17313 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/azure, fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > A recurrent problem in processes with many worker threads (hive, spark etc) > is that calling `FileSystem.get(URI-to-object-store)` triggers the creation > and then discard of many FS clients -all but one for the same URL. As well as > the direct performance hit, this can exacerbate locking problems and make > instantiation a lot slower than it would otherwise be. > This has been observed with the S3A and ABFS connectors. > The ultimate solution here would probably be something more complicated to > ensure that only one thread was ever creating a connector for a given URL > -the rest would wait for it to be initialized. This would (a) reduce > contention & CPU, IO network load, and (b) reduce the time for all but the > first thread to resume processing to that of the remaining time in > .initialize(). This would also benefit the S3A connector. > We'd need something like > # A (per-user) map of filesystems being created > # split createFileSystem into two: instantiateFileSystem and > initializeFileSystem > # each thread to instantiate the FS, put() it into the new map > # If there was one already, discard the old one and wait for the new one to > be ready via a call to Object.wait() > # If there wasn't an entry, call initializeFileSystem) and then, finally, > call Object.notifyAll(), and move it from the map of filesystems being > initialized to the map of created filesystems > This sounds too straightforward to be that simple; the troublespots are > probably related to race conditions moving entries between the two maps and > making sure that no thread will block on the FS being initialized while it > has already been initialized (and so wait() will block forever). > Rather than seek perfection, it may be safest go for a best-effort > optimisation of the #of FS instances created/initialized. That is: its better > to maybe create a few more FS instances than needed than it is to block > forever. > Something is doable here, it's just not quick-and-dirty. Testing will be > "fun"; probably best to isolate this new logic somewhere where we can > simulate slow starts on one thread with many other threads waiting for it. > A simpler option would be to have a lock on the construction process: only > one FS can be instantiated per user at a a time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17311) ABFS: Logs should redact SAS signature
[ https://issues.apache.org/jira/browse/HADOOP-17311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17311: Fix Version/s: 3.3.1 Resolution: Fixed Status: Resolved (was: Patch Available) > ABFS: Logs should redact SAS signature > -- > > Key: HADOOP-17311 > URL: https://issues.apache.org/jira/browse/HADOOP-17311 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, security >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Bilahari T H >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 3h > Remaining Estimate: 0h > > Signature part of the SAS should be redacted for security purposes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17385) ITestS3ADeleteCost.testDirMarkersFileCreation failure
[
https://issues.apache.org/jira/browse/HADOOP-17385?focusedWorklogId=516719=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516719
]
ASF GitHub Bot logged work on HADOOP-17385:
---
Author: ASF GitHub Bot
Created on: 25/Nov/20 14:51
Start Date: 25/Nov/20 14:51
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2473:
URL: https://github.com/apache/hadoop/pull/2473#issuecomment-733753574
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 6s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 31s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 10s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 8s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 6s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 9s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 21s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 79m 16s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2473/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2473 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6570e183c2bb 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 08b2e285dbd |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2473/4/testReport/ |
| Max. process+thread count | 536 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2473/4/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
[GitHub] [hadoop] hadoop-yetus commented on pull request #2473: HADOOP-17385. ITestS3ADeleteCost.testDirMarkersFileCreation failure
hadoop-yetus commented on pull request #2473:
URL: https://github.com/apache/hadoop/pull/2473#issuecomment-733753574
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 6s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 31s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 10s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 8s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 6s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 9s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 21s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 79m 16s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2473/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2473 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6570e183c2bb 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 08b2e285dbd |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2473/4/testReport/ |
| Max. process+thread count | 536 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2473/4/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HADOOP-17313) FileSystem.get to support slow-to-instantiate FS clients
[ https://issues.apache.org/jira/browse/HADOOP-17313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17313: Release Note: The option "fs.creation.parallel.count" sets a a semaphore to throttle the number of FileSystem instances which can be created simultaneously. This is designed to reduce the impact of many threads in an application calling FileSystem.get() on a filesystem which takes time to instantiate -for example to an object where HTTPS connections are set up during initialization. Many threads trying to do this may create spurious delays by conflicting for access to synchronized blocks, when simply limiting the parallelism diminishes the conflict, so speeds up all threads trying to access the store. The default value, 64, is larger than is likely to deliver any speedup -but it does mean that there should be no adverse effects from the change. If a service appears to be blocking on all threads initializing connections to abfs, s3a or store, try a smaller (possibly significantly smaller) value. Description: A recurrent problem in processes with many worker threads (hive, spark etc) is that calling `FileSystem.get(URI-to-object-store)` triggers the creation and then discard of many FS clients -all but one for the same URL. As well as the direct performance hit, this can exacerbate locking problems and make instantiation a lot slower than it would otherwise be. This has been observed with the S3A and ABFS connectors. The ultimate solution here would probably be something more complicated to ensure that only one thread was ever creating a connector for a given URL -the rest would wait for it to be initialized. This would (a) reduce contention & CPU, IO network load, and (b) reduce the time for all but the first thread to resume processing to that of the remaining time in .initialize(). This would also benefit the S3A connector. We'd need something like # A (per-user) map of filesystems being created # split createFileSystem into two: instantiateFileSystem and initializeFileSystem # each thread to instantiate the FS, put() it into the new map # If there was one already, discard the old one and wait for the new one to be ready via a call to Object.wait() # If there wasn't an entry, call initializeFileSystem) and then, finally, call Object.notifyAll(), and move it from the map of filesystems being initialized to the map of created filesystems This sounds too straightforward to be that simple; the troublespots are probably related to race conditions moving entries between the two maps and making sure that no thread will block on the FS being initialized while it has already been initialized (and so wait() will block forever). Rather than seek perfection, it may be safest go for a best-effort optimisation of the #of FS instances created/initialized. That is: its better to maybe create a few more FS instances than needed than it is to block forever. Something is doable here, it's just not quick-and-dirty. Testing will be "fun"; probably best to isolate this new logic somewhere where we can simulate slow starts on one thread with many other threads waiting for it. A simpler option would be to have a lock on the construction process: only one FS can be instantiated per user at a a time. was: A recurrent problem in processes with many worker threads (hive, spark etc) is that calling `FileSystem.get(URI-to-object-store)` triggers the creation and then discard of many FS clients -all but one for the same URL. As well as the direct performance hit, this can exacerbate locking problems and make instantiation a lot slower than it would otherwise be. This has been observed with the S3A and ABFS connectors. The ultimate solution here would probably be something more complicated to ensure that only one thread was ever creating a connector for a given URL -the rest would wait for it to be initialized. This would (a) reduce contention & CPU, IO network load, and (b) reduce the time for all but the first thread to resume processing to that of the remaining time in .initialize(). This would also benefit the S3A connector. We'd need something like # A (per-user) map of filesystems being created # split createFileSystem into two: instantiateFileSystem and initializeFileSystem # each thread to instantiate the FS, put() it into the new map # If there was one already, discard the old one and wait for the new one to be ready via a call to Object.wait() # If there wasn't an entry, call initializeFileSystem) and then, finally, call Object.notifyAll(), and move it from the map of filesystems being initialized to the map of created filesystems This sounds too straightforward to be that simple; the troublespots are probably related to race conditions moving entries between the two maps and making sure that no thread will block on the FS being initialized while it has
[jira] [Work logged] (HADOOP-17313) FileSystem.get to support slow-to-instantiate FS clients
[ https://issues.apache.org/jira/browse/HADOOP-17313?focusedWorklogId=516708=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516708 ] ASF GitHub Bot logged work on HADOOP-17313: --- Author: ASF GitHub Bot Created on: 25/Nov/20 14:31 Start Date: 25/Nov/20 14:31 Worklog Time Spent: 10m Work Description: steveloughran merged pull request #2396: URL: https://github.com/apache/hadoop/pull/2396 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516708) Time Spent: 4h 50m (was: 4h 40m) > FileSystem.get to support slow-to-instantiate FS clients > > > Key: HADOOP-17313 > URL: https://issues.apache.org/jira/browse/HADOOP-17313 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/azure, fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 4h 50m > Remaining Estimate: 0h > > A recurrent problem in processes with many worker threads (hive, spark etc) > is that calling `FileSystem.get(URI-to-object-store)` triggers the creation > and then discard of many FS clients -all but one for the same URL. As well as > the direct performance hit, this can exacerbate locking problems and make > instantiation a lot slower than it would otherwise be. > This has been observed with the S3A and ABFS connectors. > The ultimate solution here would probably be something more complicated to > ensure that only one thread was ever creating a connector for a given URL > -the rest would wait for it to be initialized. This would (a) reduce > contention & CPU, IO network load, and (b) reduce the time for all but the > first thread to resume processing to that of the remaining time in > .initialize(). This would also benefit the S3A connector. > We'd need something like > # A (per-user) map of filesystems being created > # split createFileSystem into two: instantiateFileSystem and > initializeFileSystem > # each thread to instantiate the FS, put() it into the new map > # If there was one already, discard the old one and wait for the new one to > be ready via a call to Object.wait() > # If there wasn't an entry, call initializeFileSystem) and then, finally, > call Object.notifyAll(), and move it from the map of filesystems being > initialized to the map of created filesystems > This sounds too straightforward to be that simple; the troublespots are > probably related to race conditions moving entries between the two maps and > making sure that no thread will block on the FS being initialized while it > has already been initialized (and so wait() will block forever). > Rather than seek perfection, it may be safest go for a best-effort > optimisation of the #of FS instances created/initialized. That is: its better > to maybe create a few more FS instances than needed than it is to block > forever. > Something is doable here, it's just not quick-and-dirty. Testing will be > "fun"; probably best to isolate this new logic somewhere where we can > simulate slow starts on one thread with many other threads waiting for it. > A simpler option would be to have a lock on the construction process: only > one FS can be instantiated per user at a a time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #2396: HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients.
steveloughran merged pull request #2396: URL: https://github.com/apache/hadoop/pull/2396 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17313) FileSystem.get to support slow-to-instantiate FS clients
[ https://issues.apache.org/jira/browse/HADOOP-17313?focusedWorklogId=516705=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516705 ] ASF GitHub Bot logged work on HADOOP-17313: --- Author: ASF GitHub Bot Created on: 25/Nov/20 14:23 Start Date: 25/Nov/20 14:23 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2396: URL: https://github.com/apache/hadoop/pull/2396#issuecomment-733737440 thanks -merging to 3.3+! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516705) Time Spent: 4h 40m (was: 4.5h) > FileSystem.get to support slow-to-instantiate FS clients > > > Key: HADOOP-17313 > URL: https://issues.apache.org/jira/browse/HADOOP-17313 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/azure, fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 4h 40m > Remaining Estimate: 0h > > A recurrent problem in processes with many worker threads (hive, spark etc) > is that calling `FileSystem.get(URI-to-object-store)` triggers the creation > and then discard of many FS clients -all but one for the same URL. As well as > the direct performance hit, this can exacerbate locking problems and make > instantiation a lot slower than it would otherwise be. > This has been observed with the S3A and ABFS connectors. > The ultimate solution here would probably be something more complicated to > ensure that only one thread was ever creating a connector for a given URL > -the rest would wait for it to be initialized. This would (a) reduce > contention & CPU, IO network load, and (b) reduce the time for all but the > first thread to resume processing to that of the remaining time in > .initialize(). This would also benefit the S3A connector. > We'd need something like > # A (per-user) map of filesystems being created > # split createFileSystem into two: instantiateFileSystem and > initializeFileSystem > # each thread to instantiate the FS, put() it into the new map > # If there was one already, discard the old one and wait for the new one to > be ready via a call to Object.wait() > # If there wasn't an entry, call initializeFileSystem) and then, finally, > call Object.notifyAll(), and move it from the map of filesystems being > initialized to the map of created filesystems > This sounds too straightforward to be that simple; the troublespots are > probably related to race conditions moving entries between the two maps and > making sure that no thread will block on the FS being initialized while it > has already been initialized (and so wait() will block forever). > Rather than seek perfection, it may be safest go for a best-effort > optimisation of the #of FS instances created/initialized. That is: its better > to maybe create a few more FS instances than needed than it is to block > forever. > Something is doable here, it's just not quick-and-dirty. Testing will be > "fun"; probably best to isolate this new logic somewhere where we can > simulate slow starts on one thread with many other threads waiting for it. > A simpler option would be to have a lock on the construction process: only > one FS can be instantiated per user at a a time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2396: HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients.
steveloughran commented on pull request #2396: URL: https://github.com/apache/hadoop/pull/2396#issuecomment-733737440 thanks -merging to 3.3+! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17311) ABFS: Logs should redact SAS signature
[ https://issues.apache.org/jira/browse/HADOOP-17311?focusedWorklogId=516704=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516704 ] ASF GitHub Bot logged work on HADOOP-17311: --- Author: ASF GitHub Bot Created on: 25/Nov/20 14:22 Start Date: 25/Nov/20 14:22 Worklog Time Spent: 10m Work Description: steveloughran merged pull request #2422: URL: https://github.com/apache/hadoop/pull/2422 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 516704) Time Spent: 3h (was: 2h 50m) > ABFS: Logs should redact SAS signature > -- > > Key: HADOOP-17311 > URL: https://issues.apache.org/jira/browse/HADOOP-17311 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, security >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Bilahari T H >Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > Signature part of the SAS should be redacted for security purposes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #2422: HADOOP-17311. ABFS: Logs should redact SAS signature
steveloughran merged pull request #2422: URL: https://github.com/apache/hadoop/pull/2422 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17265) ABFS: Support for Client Correlation ID
[
https://issues.apache.org/jira/browse/HADOOP-17265?focusedWorklogId=516676=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516676
]
ASF GitHub Bot logged work on HADOOP-17265:
---
Author: ASF GitHub Bot
Created on: 25/Nov/20 13:23
Start Date: 25/Nov/20 13:23
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2344:
URL: https://github.com/apache/hadoop/pull/2344#issuecomment-733703307
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 9s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 38s | | trunk passed |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 10s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 59s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | | trunk passed |
| -0 :warning: | patch | 1m 14s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 56s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 0s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 26s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 90m 28s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2344/16/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2344 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux bfe4baa4b11d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 08b2e285dbd |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2344/16/testReport/ |
| Max. process+thread count | 510 (vs. ulimit of 5500) |
| modules | C:
[GitHub] [hadoop] hadoop-yetus commented on pull request #2344: HADOOP-17265. ABFS: Support for Client Correlation ID
hadoop-yetus commented on pull request #2344:
URL: https://github.com/apache/hadoop/pull/2344#issuecomment-733703307
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 9s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 38s | | trunk passed |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 10s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 0m 59s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | | trunk passed |
| -0 :warning: | patch | 1m 14s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 56s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 0s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 26s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 90m 28s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2344/16/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2344 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux bfe4baa4b11d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 08b2e285dbd |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2344/16/testReport/ |
| Max. process+thread count | 510 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2344/16/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an
[jira] [Work logged] (HADOOP-17385) ITestS3ADeleteCost.testDirMarkersFileCreation failure
[
https://issues.apache.org/jira/browse/HADOOP-17385?focusedWorklogId=516672=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516672
]
ASF GitHub Bot logged work on HADOOP-17385:
---
Author: ASF GitHub Bot
Created on: 25/Nov/20 13:07
Start Date: 25/Nov/20 13:07
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#2473:
URL: https://github.com/apache/hadoop/pull/2473#discussion_r530360751
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3ATestBase.java
##
@@ -60,6 +60,9 @@ public void setup() throws Exception {
// filesystems which add default configuration resources to do it before
// our tests start adding/removing options. See HADOOP-16626.
FileSystem.getLocal(new Configuration());
+// instantiate an S3A FS here here to force deprecated key load through the
+// static initializers. See: HADOOP-17385
+new S3AFileSystem();
Review comment:
I was about to say I don't know a way to trigger instantiation of static
class initializers except by a new() call, but actually we can invoke any
static method. How about I add a new static `initializeClass()` call and load
it.
FWIW, what I'd really like is to be able to disable forced reload of
configs, but I can't see a way to do that which wouldn't run the risk of damage
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 516672)
Time Spent: 2h 10m (was: 2h)
> ITestS3ADeleteCost.testDirMarkersFileCreation failure
> -
>
> Key: HADOOP-17385
> URL: https://issues.apache.org/jira/browse/HADOOP-17385
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Test runs after apply of recent patches HADOOP-17318 and HADOOP-17244 are
> showing a failure in ITestS3ADeleteCost.testDirMarkersFileCreation
> Either one of the patches is failing and wasn't picked up, or the combined
> changes are conflicting in some way. Not backporting HADOOP-17318 to
> branch-3.3 until this is addressed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2473: HADOOP-17385. ITestS3ADeleteCost.testDirMarkersFileCreation failure
steveloughran commented on a change in pull request #2473:
URL: https://github.com/apache/hadoop/pull/2473#discussion_r530360751
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3ATestBase.java
##
@@ -60,6 +60,9 @@ public void setup() throws Exception {
// filesystems which add default configuration resources to do it before
// our tests start adding/removing options. See HADOOP-16626.
FileSystem.getLocal(new Configuration());
+// instantiate an S3A FS here here to force deprecated key load through the
+// static initializers. See: HADOOP-17385
+new S3AFileSystem();
Review comment:
I was about to say I don't know a way to trigger instantiation of static
class initializers except by a new() call, but actually we can invoke any
static method. How about I add a new static `initializeClass()` call and load
it.
FWIW, what I'd really like is to be able to disable forced reload of
configs, but I can't see a way to do that which wouldn't run the risk of damage
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #2490: YARN-10499. TestRouterWebServiceREST fails
aajisaka commented on pull request #2490: URL: https://github.com/apache/hadoop/pull/2490#issuecomment-733637841 > hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps > hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation These tests failed by OOM in the precommit job. I ran the tests successfully on my local environment: ``` [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Running org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation [INFO] Tests run: 160, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 820.439 s - in org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation [INFO] Running org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps [INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.061 s - in org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps [INFO] [INFO] Results: [INFO] [INFO] Tests run: 200, Failures: 0, Errors: 0, Skipped: 0 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17385) ITestS3ADeleteCost.testDirMarkersFileCreation failure
[
https://issues.apache.org/jira/browse/HADOOP-17385?focusedWorklogId=516603=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-516603
]
ASF GitHub Bot logged work on HADOOP-17385:
---
Author: ASF GitHub Bot
Created on: 25/Nov/20 10:46
Start Date: 25/Nov/20 10:46
Worklog Time Spent: 10m
Work Description: bgaborg commented on a change in pull request #2473:
URL: https://github.com/apache/hadoop/pull/2473#discussion_r530276592
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3ATestBase.java
##
@@ -60,6 +60,9 @@ public void setup() throws Exception {
// filesystems which add default configuration resources to do it before
// our tests start adding/removing options. See HADOOP-16626.
FileSystem.getLocal(new Configuration());
+// instantiate an S3A FS here here to force deprecated key load through the
+// static initializers. See: HADOOP-17385
+new S3AFileSystem();
Review comment:
I think it would be cleaner if we could run the static initializers
instead of creating an object without storing the reference. I know that
there's a comment there stating that this is required because of a jira, but I
don't see the point why would we create an object just to trigger side effects.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 516603)
Time Spent: 2h (was: 1h 50m)
> ITestS3ADeleteCost.testDirMarkersFileCreation failure
> -
>
> Key: HADOOP-17385
> URL: https://issues.apache.org/jira/browse/HADOOP-17385
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Test runs after apply of recent patches HADOOP-17318 and HADOOP-17244 are
> showing a failure in ITestS3ADeleteCost.testDirMarkersFileCreation
> Either one of the patches is failing and wasn't picked up, or the combined
> changes are conflicting in some way. Not backporting HADOOP-17318 to
> branch-3.3 until this is addressed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #2473: HADOOP-17385. ITestS3ADeleteCost.testDirMarkersFileCreation failure
bgaborg commented on a change in pull request #2473:
URL: https://github.com/apache/hadoop/pull/2473#discussion_r530276592
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3ATestBase.java
##
@@ -60,6 +60,9 @@ public void setup() throws Exception {
// filesystems which add default configuration resources to do it before
// our tests start adding/removing options. See HADOOP-16626.
FileSystem.getLocal(new Configuration());
+// instantiate an S3A FS here here to force deprecated key load through the
+// static initializers. See: HADOOP-17385
+new S3AFileSystem();
Review comment:
I think it would be cleaner if we could run the static initializers
instead of creating an object without storing the reference. I know that
there's a comment there stating that this is required because of a jira, but I
don't see the point why would we create an object just to trigger side effects.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2489: HDFS-15695. NN should not let the balancer run in safemode
hadoop-yetus commented on pull request #2489:
URL: https://github.com/apache/hadoop/pull/2489#issuecomment-733537635
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 14s | | trunk passed |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 31s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 3m 51s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 47s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 28s | | the patch passed |
| +1 :green_heart: | compile | 1m 36s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 1m 36s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 53s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 56 unchanged - 10
fixed = 56 total (was 66) |
| +1 :green_heart: | mvnsite | 1m 32s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 19m 23s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | findbugs | 4m 12s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 127m 45s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2489/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 238m 21s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2489/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2489 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e67cdb121b2f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 08b2e285dbd |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2489/1/testReport/ |
| Max. process+thread count | 3097 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
