[GitHub] [hadoop] hadoop-yetus commented on pull request #3311: HDFS-16178. Make recursive rmdir in libhdfs++ cross platform
hadoop-yetus commented on pull request #3311:
URL: https://github.com/apache/hadoop/pull/3311#issuecomment-902450837
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 0s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 7s | | trunk passed |
| +1 :green_heart: | compile | 2m 54s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 2m 52s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | mvnsite | 0m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 45m 4s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 13s | | the patch passed |
| +1 :green_heart: | compile | 2m 41s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | cc | 2m 41s | | the patch passed |
| +1 :green_heart: | golang | 2m 41s | | the patch passed |
| +1 :green_heart: | javac | 2m 41s | | the patch passed |
| +1 :green_heart: | compile | 2m 48s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | cc | 2m 48s | | the patch passed |
| +1 :green_heart: | golang | 2m 48s | | the patch passed |
| +1 :green_heart: | javac | 2m 48s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 86m 19s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 156m 53s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3311 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux 7699f5deb741 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0ae3c13ceb40b6e8af2e8012e00dd9bec53ff94e |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/testReport/ |
| Max. process+thread count | 520 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on a change in pull request #3307: HDFS-16175.Improve the configurable value of Server #PURGE_INTERVAL_NANOS.
jianghuazhu commented on a change in pull request #3307: URL: https://github.com/apache/hadoop/pull/3307#discussion_r692683222 ## File path: hadoop-common-project/hadoop-common/src/main/resources/core-default.xml ## @@ -2460,6 +2460,14 @@ + + ipc.server.purge.interval + 15 + In Server, how often will Calls be cleaned up? Review comment: Thanks @ayushtkn for the comment. I will make some changes later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #3313: HDFS-16179. Update loglevel for BlockManager#chooseExcessRedundancySt…
tomscut commented on pull request #3313: URL: https://github.com/apache/hadoop/pull/3313#issuecomment-902436338 > @tomscut Thanks for comments. > > > but there are no redundancies in most cases > > I see that if shouldProcessExtraRedundancy returns true, It will enter this method. > If no redundancies, it will not enter this method, is it right? > Are there any reasons that it prints lots of logs, maybe there is a hidden reason ? Thanks @ferhui for your comments and careful consideration. I found this issue [HDFS-9876](https://issues.apache.org/jira/browse/HDFS-9876), and we can refer to it. Hi @Jing9 , could you please take a look at this. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17682) ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
[ https://issues.apache.org/jira/browse/HADOOP-17682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17402024#comment-17402024 ] Surendra Singh Lilhore commented on HADOOP-17682: - Committed to trunk. Back port pending for 3.3 branch > ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters > -- > > Key: HADOOP-17682 > URL: https://issues.apache.org/jira/browse/HADOOP-17682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 5h 40m > Remaining Estimate: 0h > > ABFS open methods require certain information (contentLength, eTag, etc) to > to create an InputStream for the file at the given path. This information is > retrieved via a GetFileStatus request to backend. > However, client applications may often have access to the FileStatus prior to > invoking the open API. Providing this FileStatus to the driver through the > OpenFileParameters argument of openFileWithOptions() can help avoid the call > to Store for FileStatus. > This PR adds handling for the FileStatus instance (if any) provided via the > OpenFileParameters argument. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17682) ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
[ https://issues.apache.org/jira/browse/HADOOP-17682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HADOOP-17682: Fix Version/s: 3.4.0 > ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters > -- > > Key: HADOOP-17682 > URL: https://issues.apache.org/jira/browse/HADOOP-17682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sumangala Patki >Assignee: Sumangala Patki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 5h 40m > Remaining Estimate: 0h > > ABFS open methods require certain information (contentLength, eTag, etc) to > to create an InputStream for the file at the given path. This information is > retrieved via a GetFileStatus request to backend. > However, client applications may often have access to the FileStatus prior to > invoking the open API. Providing this FileStatus to the driver through the > OpenFileParameters argument of openFileWithOptions() can help avoid the call > to Store for FileStatus. > This PR adds handling for the FileStatus instance (if any) provided via the > OpenFileParameters argument. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ferhui commented on pull request #3313: HDFS-16179. Update loglevel for BlockManager#chooseExcessRedundancySt…
ferhui commented on pull request #3313: URL: https://github.com/apache/hadoop/pull/3313#issuecomment-902402168 @tomscut Thanks for comments. > but there are no redundancies in most cases I see that if shouldProcessExtraRedundancy returns true, It will enter this method. If no redundancies, it will not enter this method, is it right? Are there any reasons that it prints lots of logs, maybe there is a hidden reason ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #3307: HDFS-16175.Improve the configurable value of Server #PURGE_INTERVAL_NANOS.
ayushtkn commented on a change in pull request #3307: URL: https://github.com/apache/hadoop/pull/3307#discussion_r692630502 ## File path: hadoop-common-project/hadoop-common/src/main/resources/core-default.xml ## @@ -2460,6 +2460,14 @@ + + ipc.server.purge.interval + 15 + In Server, how often will Calls be cleaned up? Review comment: nit: Can remove the question mark. Asking question in the description will not make sense. :-) Please rephrase it just tell this configuration defines how often the calls will get cleaned up in the server. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3311: HDFS-16178. Make recursive rmdir in libhdfs++ cross platform
hadoop-yetus commented on pull request #3311:
URL: https://github.com/apache/hadoop/pull/3311#issuecomment-902399733
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 15m 3s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 3s | | trunk passed |
| +1 :green_heart: | compile | 2m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 53m 16s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 14s | | the patch passed |
| +1 :green_heart: | compile | 2m 19s | | the patch passed |
| +1 :green_heart: | cc | 2m 19s | | the patch passed |
| +1 :green_heart: | golang | 2m 19s | | the patch passed |
| +1 :green_heart: | javac | 2m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 90m 48s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 184m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3311 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux 6a46954d984e 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0ae3c13ceb40b6e8af2e8012e00dd9bec53ff94e |
| Default Java | Debian-11.0.12+7-post-Debian-2 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/testReport/ |
| Max. process+thread count | 532 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/console |
| versions | git=2.32.0 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Neilxzn commented on pull request #3315: HDFS-16180.FsVolumeImpl.nextBlock should consider that the block meta file has been deleted
Neilxzn commented on pull request #3315: URL: https://github.com/apache/hadoop/pull/3315#issuecomment-902399137 The checks from jenkins has failed, but I can't find any error about the patch. Please check it again -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Neilxzn commented on pull request #3315: HDFS-16180.FsVolumeImpl.nextBlock should consider that the block meta file has been deleted
Neilxzn commented on pull request #3315: URL: https://github.com/apache/hadoop/pull/3315#issuecomment-902395807 cc @jojochuang. Thank you for the review. I think we should catch FileNotFoundException about the meta file in FsVolumeImpl.nextBlock. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut edited a comment on pull request #3313: HDFS-16179. Update loglevel for BlockManager#chooseExcessRedundancySt…
tomscut edited a comment on pull request #3313: URL: https://github.com/apache/hadoop/pull/3313#issuecomment-902367933 > @tomscut Thanks for contribution. > I'm confused here. The log is necessary or not. If it's necessary, many logs as you said drown out other logs. If it's not necessary, I think DEBUG level is OK. Thanks @ferhui for your review. I think this method is just detecting redundancies and deleting them, but there are no redundancies in most cases. IMO, the log is not necessary here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #3313: HDFS-16179. Update loglevel for BlockManager#chooseExcessRedundancySt…
tomscut commented on pull request #3313: URL: https://github.com/apache/hadoop/pull/3313#issuecomment-902367933 > @tomscut Thanks for contribution. > I'm confused here. The log is necessary or not. If it's necessary, many logs as you said drown out other logs. If it's not necessary, I think DEBUG level is OK. Thanks @ferhui for your review. I think this method is just detecting redundancies and deleting them, but there are no redundancies in most cases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ferhui commented on pull request #3313: HDFS-16179. Update loglevel for BlockManager#chooseExcessRedundancySt…
ferhui commented on pull request #3313: URL: https://github.com/apache/hadoop/pull/3313#issuecomment-902364073 @tomscut Thanks for contribution. I'm confused here. The log is necessary or not. If it's necessary, many logs as you said drown out other logs. If it's not necessary, I think DEBUG level is OK. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #3313: HDFS-16179. Update loglevel for BlockManager#chooseExcessRedundancySt…
tomscut commented on pull request #3313: URL: https://github.com/apache/hadoop/pull/3313#issuecomment-902348717 Hi @jojochuang @ferhui , could you please also review the code? Thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-14693) Upgrade JUnit from 4 to 5
[
https://issues.apache.org/jira/browse/HADOOP-14693?focusedWorklogId=640071=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-640071
]
ASF GitHub Bot logged work on HADOOP-14693:
---
Author: ASF GitHub Bot
Created on: 20/Aug/21 00:42
Start Date: 20/Aug/21 00:42
Worklog Time Spent: 10m
Work Description: smengcl edited a comment on pull request #3304:
URL: https://github.com/apache/hadoop/pull/3304#issuecomment-902345346
Thanks @aajisaka for trying out the _rewrite_ plugin.
Overall the tool seems to be doing a pretty good job.
1. In the case of `TestHDFSContractPathHandle.java`, It somehow attempts to
import `org.junit.jupiter.api.Assertions.super` and rearrange the argument
order to `super()` call, which is kinda amusing.
```diff
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
index c65a60b18b1..a5ea66782cf 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
@@ -21,11 +21,13 @@
...
+import static org.junit.jupiter.api.Assertions.super;
+
/**
* Verify HDFS compliance with {@link org.apache.hadoop.fs.PathHandle}
* semantics.
@@ -35,15 +37,15 @@
public TestHDFSContractPathHandle(String testname, Options.HandleOpt[]
opts,
boolean serialized) {
-super(testname, opts, serialized);
+ super(opts, serialized, testname);
}
...
```
2. In the case of `TestDiskBalancerRPC.java`, it does remove JUnit 4's
`ExpectedException` variables, but it doesn't seem to rewrite the
`ExpectedException.expect` usages.
For instance,
```java
@Rule
public ExpectedException thrown = ExpectedException.none();
...
@Test
public void testSubmitPlanWithInvalidVersion() throws Exception {
...
thrown.expect(DiskBalancerException.class);
thrown.expect(new
DiskBalancerResultVerifier(Result.INVALID_PLAN_VERSION));
dataNode.submitDiskBalancerPlan(planHash, planVersion, PLAN_FILE,
plan.toJson(), false);
```
should have been rewritten into something like:
```java
final DiskBalancerException thrown =
Assertions.assertThrows(DiskBalancerException.class, () -> {
dataNode.submitDiskBalancerPlan(planHash, planVersion, PLAN_FILE,
plan.toJson(), false);
});
Assertions.assertEquals(thrown.getResult(), Result.INVALID_PLAN_VERSION);
```
... and some other interesting changes by rewrite plugin.
I have included some fixes my fork of your branch that addresses some of
those rewrite errors I mentioned above (and others):
https://github.com/apache/hadoop/compare/aajisaka:rewrite-junit5-hdfs..smengcl:rewrite-junit5-hdfs?diff=split
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 640071)
Time Spent: 1h 10m (was: 1h)
> Upgrade JUnit from 4 to 5
> -
>
> Key: HADOOP-14693
> URL: https://issues.apache.org/jira/browse/HADOOP-14693
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> JUnit 4 does not support Java 9. We need to upgrade this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-14693) Upgrade JUnit from 4 to 5
[
https://issues.apache.org/jira/browse/HADOOP-14693?focusedWorklogId=640070=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-640070
]
ASF GitHub Bot logged work on HADOOP-14693:
---
Author: ASF GitHub Bot
Created on: 20/Aug/21 00:42
Start Date: 20/Aug/21 00:42
Worklog Time Spent: 10m
Work Description: smengcl edited a comment on pull request #3304:
URL: https://github.com/apache/hadoop/pull/3304#issuecomment-902345346
Thanks @aajisaka for trying out the _rewrite_ plugin.
Overall the tool seems to be doing a pretty good job.
1. In the case of `TestHDFSContractPathHandle.java`, It somehow attempts to
import `org.junit.jupiter.api.Assertions.super` and rearrange the argument
order to `super()` call, which is kinda amusing.
```diff
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
index c65a60b18b1..a5ea66782cf 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
@@ -21,11 +21,13 @@
...
+import static org.junit.jupiter.api.Assertions.super;
+
/**
* Verify HDFS compliance with {@link org.apache.hadoop.fs.PathHandle}
* semantics.
@@ -35,15 +37,15 @@
public TestHDFSContractPathHandle(String testname, Options.HandleOpt[]
opts,
boolean serialized) {
-super(testname, opts, serialized);
+ super(opts, serialized, testname);
}
...
```
2. In the case of `TestDiskBalancerRPC.java`, it does remove JUnit 4's
`ExpectedException` variables, but it doesn't seem to rewrite the
`ExpectedException.expect` usages.
For instance,
```java
@Rule
public ExpectedException thrown = ExpectedException.none();
...
@Test
public void testSubmitPlanWithInvalidVersion() throws Exception {
...
thrown.expect(DiskBalancerException.class);
thrown.expect(new
DiskBalancerResultVerifier(Result.INVALID_PLAN_VERSION));
dataNode.submitDiskBalancerPlan(planHash, planVersion, PLAN_FILE,
plan.toJson(), false);
```
should have been rewritten into something like:
```java
final DiskBalancerException thrown =
Assertions.assertThrows(DiskBalancerException.class, () -> {
dataNode.submitDiskBalancerPlan(planHash, planVersion, PLAN_FILE,
plan.toJson(), false);
});
Assertions.assertEquals(thrown.getResult(), Result.INVALID_PLAN_VERSION);
```
... and some other interesting changes by rewrite plugin.
I have included some fixes my fork of your branch that addresses some of
those rewrite errors I mentioned above (and others):
https://github.com/aajisaka/hadoop/compare/rewrite-junit5-hdfs..smengcl:rewrite-junit5-hdfs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 640070)
Time Spent: 1h (was: 50m)
> Upgrade JUnit from 4 to 5
> -
>
> Key: HADOOP-14693
> URL: https://issues.apache.org/jira/browse/HADOOP-14693
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> JUnit 4 does not support Java 9. We need to upgrade this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl edited a comment on pull request #3304: HADOOP-14693. Test PR: Ran rewrite plugin on hadoop-hdfs module to upgrade to JUnit 5
smengcl edited a comment on pull request #3304:
URL: https://github.com/apache/hadoop/pull/3304#issuecomment-902345346
Thanks @aajisaka for trying out the _rewrite_ plugin.
Overall the tool seems to be doing a pretty good job.
1. In the case of `TestHDFSContractPathHandle.java`, It somehow attempts to
import `org.junit.jupiter.api.Assertions.super` and rearrange the argument
order to `super()` call, which is kinda amusing.
```diff
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
index c65a60b18b1..a5ea66782cf 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
@@ -21,11 +21,13 @@
...
+import static org.junit.jupiter.api.Assertions.super;
+
/**
* Verify HDFS compliance with {@link org.apache.hadoop.fs.PathHandle}
* semantics.
@@ -35,15 +37,15 @@
public TestHDFSContractPathHandle(String testname, Options.HandleOpt[]
opts,
boolean serialized) {
-super(testname, opts, serialized);
+ super(opts, serialized, testname);
}
...
```
2. In the case of `TestDiskBalancerRPC.java`, it does remove JUnit 4's
`ExpectedException` variables, but it doesn't seem to rewrite the
`ExpectedException.expect` usages.
For instance,
```java
@Rule
public ExpectedException thrown = ExpectedException.none();
...
@Test
public void testSubmitPlanWithInvalidVersion() throws Exception {
...
thrown.expect(DiskBalancerException.class);
thrown.expect(new
DiskBalancerResultVerifier(Result.INVALID_PLAN_VERSION));
dataNode.submitDiskBalancerPlan(planHash, planVersion, PLAN_FILE,
plan.toJson(), false);
```
should have been rewritten into something like:
```java
final DiskBalancerException thrown =
Assertions.assertThrows(DiskBalancerException.class, () -> {
dataNode.submitDiskBalancerPlan(planHash, planVersion, PLAN_FILE,
plan.toJson(), false);
});
Assertions.assertEquals(thrown.getResult(), Result.INVALID_PLAN_VERSION);
```
... and some other interesting changes by rewrite plugin.
I have included some fixes my fork of your branch that addresses some of
those rewrite errors I mentioned above (and others):
https://github.com/apache/hadoop/compare/aajisaka:rewrite-junit5-hdfs..smengcl:rewrite-junit5-hdfs?diff=split
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl edited a comment on pull request #3304: HADOOP-14693. Test PR: Ran rewrite plugin on hadoop-hdfs module to upgrade to JUnit 5
smengcl edited a comment on pull request #3304:
URL: https://github.com/apache/hadoop/pull/3304#issuecomment-902345346
Thanks @aajisaka for trying out the _rewrite_ plugin.
Overall the tool seems to be doing a pretty good job.
1. In the case of `TestHDFSContractPathHandle.java`, It somehow attempts to
import `org.junit.jupiter.api.Assertions.super` and rearrange the argument
order to `super()` call, which is kinda amusing.
```diff
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
index c65a60b18b1..a5ea66782cf 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
@@ -21,11 +21,13 @@
...
+import static org.junit.jupiter.api.Assertions.super;
+
/**
* Verify HDFS compliance with {@link org.apache.hadoop.fs.PathHandle}
* semantics.
@@ -35,15 +37,15 @@
public TestHDFSContractPathHandle(String testname, Options.HandleOpt[]
opts,
boolean serialized) {
-super(testname, opts, serialized);
+ super(opts, serialized, testname);
}
...
```
2. In the case of `TestDiskBalancerRPC.java`, it does remove JUnit 4's
`ExpectedException` variables, but it doesn't seem to rewrite the
`ExpectedException.expect` usages.
For instance,
```java
@Rule
public ExpectedException thrown = ExpectedException.none();
...
@Test
public void testSubmitPlanWithInvalidVersion() throws Exception {
...
thrown.expect(DiskBalancerException.class);
thrown.expect(new
DiskBalancerResultVerifier(Result.INVALID_PLAN_VERSION));
dataNode.submitDiskBalancerPlan(planHash, planVersion, PLAN_FILE,
plan.toJson(), false);
```
should have been rewritten into something like:
```java
final DiskBalancerException thrown =
Assertions.assertThrows(DiskBalancerException.class, () -> {
dataNode.submitDiskBalancerPlan(planHash, planVersion, PLAN_FILE,
plan.toJson(), false);
});
Assertions.assertEquals(thrown.getResult(), Result.INVALID_PLAN_VERSION);
```
... and some other interesting changes by rewrite plugin.
I have included some fixes my fork of your branch that addresses some of
those rewrite errors I mentioned above (and others):
https://github.com/aajisaka/hadoop/compare/rewrite-junit5-hdfs..smengcl:rewrite-junit5-hdfs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-14693) Upgrade JUnit from 4 to 5
[
https://issues.apache.org/jira/browse/HADOOP-14693?focusedWorklogId=640068=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-640068
]
ASF GitHub Bot logged work on HADOOP-14693:
---
Author: ASF GitHub Bot
Created on: 20/Aug/21 00:41
Start Date: 20/Aug/21 00:41
Worklog Time Spent: 10m
Work Description: smengcl edited a comment on pull request #3304:
URL: https://github.com/apache/hadoop/pull/3304#issuecomment-902345346
Thanks @aajisaka for trying out the _rewrite_ plugin.
Overall the tool seems to be doing a pretty good job.
1. In the case of `TestHDFSContractPathHandle.java`, It somehow attempts to
import `org.junit.jupiter.api.Assertions.super` and rearrange the argument
order to `super()` call, which is kinda amusing.
```diff
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
index c65a60b18b1..a5ea66782cf 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
@@ -21,11 +21,13 @@
...
+import static org.junit.jupiter.api.Assertions.super;
+
/**
* Verify HDFS compliance with {@link org.apache.hadoop.fs.PathHandle}
* semantics.
@@ -35,15 +37,15 @@
public TestHDFSContractPathHandle(String testname, Options.HandleOpt[]
opts,
boolean serialized) {
-super(testname, opts, serialized);
+ super(opts, serialized, testname);
}
...
```
2. In the case of `TestDiskBalancerRPC.java`, it does remove JUnit 4's
`ExpectedException` variables, but it doesn't seem to rewrite the
`ExpectedException.expect` usages.
For instance,
```java
@Rule
public ExpectedException thrown = ExpectedException.none();
...
@Test
public void testSubmitPlanWithInvalidVersion() throws Exception {
...
thrown.expect(DiskBalancerException.class);
thrown.expect(new
DiskBalancerResultVerifier(Result.INVALID_PLAN_VERSION));
dataNode.submitDiskBalancerPlan(planHash, planVersion, PLAN_FILE,
plan.toJson(), false);
```
should have been rewritten into something like:
```java
final DiskBalancerException thrown =
Assertions.assertThrows(DiskBalancerException.class, () -> {
dataNode.submitDiskBalancerPlan(planHash, planVersion, PLAN_FILE,
plan.toJson(), false);
});
Assertions.assertEquals(thrown.getResult(), Result.INVALID_PLAN_VERSION);
```
... and some other interesting changes by rewrite plugin.
I have included some fixes my fork of your branch that addresses some of
those rewrite errors I mentioned above (and others):
https://github.com/smengcl/hadoop/compare/rewrite-junit5-hdfs..aajisaka:rewrite-junit5-hdfs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 640068)
Time Spent: 50m (was: 40m)
> Upgrade JUnit from 4 to 5
> -
>
> Key: HADOOP-14693
> URL: https://issues.apache.org/jira/browse/HADOOP-14693
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> JUnit 4 does not support Java 9. We need to upgrade this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-14693) Upgrade JUnit from 4 to 5
[
https://issues.apache.org/jira/browse/HADOOP-14693?focusedWorklogId=640067=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-640067
]
ASF GitHub Bot logged work on HADOOP-14693:
---
Author: ASF GitHub Bot
Created on: 20/Aug/21 00:40
Start Date: 20/Aug/21 00:40
Worklog Time Spent: 10m
Work Description: smengcl commented on pull request #3304:
URL: https://github.com/apache/hadoop/pull/3304#issuecomment-902345346
Thanks @aajisaka for trying out the _rewrite_ plugin.
Overall the tool seems to be doing a pretty good job.
1. In the case of `TestHDFSContractPathHandle.java`, It somehow attempts to
import `org.junit.jupiter.api.Assertions.super` and rearrange the argument
order to `super()` call, which is kinda amusing.
```diff
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
index c65a60b18b1..a5ea66782cf 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
@@ -21,11 +21,13 @@
...
+import static org.junit.jupiter.api.Assertions.super;
+
/**
* Verify HDFS compliance with {@link org.apache.hadoop.fs.PathHandle}
* semantics.
@@ -35,15 +37,15 @@
public TestHDFSContractPathHandle(String testname, Options.HandleOpt[]
opts,
boolean serialized) {
-super(testname, opts, serialized);
+ super(opts, serialized, testname);
}
...
```
2. In the case of `TestDiskBalancerRPC.java`, it does remove JUnit 4's
`ExpectedException` variables, but it doesn't seem to rewrite the
`ExpectedException.expect` usages.
For instance,
```java
@Rule
public ExpectedException thrown = ExpectedException.none();
...
@Test
public void testSubmitPlanWithInvalidVersion() throws Exception {
...
thrown.expect(DiskBalancerException.class);
thrown.expect(new
DiskBalancerResultVerifier(Result.INVALID_PLAN_VERSION));
dataNode.submitDiskBalancerPlan(planHash, planVersion, PLAN_FILE,
plan.toJson(), false);
```
should have been rewritten into something like:
```java
final DiskBalancerException thrown =
Assertions.assertThrows(DiskBalancerException.class, () -> {
dataNode.submitDiskBalancerPlan(planHash, planVersion, PLAN_FILE,
plan.toJson(), false);
});
Assertions.assertEquals(thrown.getResult(), Result.INVALID_PLAN_VERSION);
```
... and some other interesting changes by rewrite plugin.
I have included some fixes my fork of your branch that addresses some of
those rewrite errors I mentioned above:
https://github.com/smengcl/hadoop/compare/rewrite-junit5-hdfs..aajisaka:rewrite-junit5-hdfs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 640067)
Time Spent: 40m (was: 0.5h)
> Upgrade JUnit from 4 to 5
> -
>
> Key: HADOOP-14693
> URL: https://issues.apache.org/jira/browse/HADOOP-14693
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> JUnit 4 does not support Java 9. We need to upgrade this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl edited a comment on pull request #3304: HADOOP-14693. Test PR: Ran rewrite plugin on hadoop-hdfs module to upgrade to JUnit 5
smengcl edited a comment on pull request #3304:
URL: https://github.com/apache/hadoop/pull/3304#issuecomment-902345346
Thanks @aajisaka for trying out the _rewrite_ plugin.
Overall the tool seems to be doing a pretty good job.
1. In the case of `TestHDFSContractPathHandle.java`, It somehow attempts to
import `org.junit.jupiter.api.Assertions.super` and rearrange the argument
order to `super()` call, which is kinda amusing.
```diff
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
index c65a60b18b1..a5ea66782cf 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
@@ -21,11 +21,13 @@
...
+import static org.junit.jupiter.api.Assertions.super;
+
/**
* Verify HDFS compliance with {@link org.apache.hadoop.fs.PathHandle}
* semantics.
@@ -35,15 +37,15 @@
public TestHDFSContractPathHandle(String testname, Options.HandleOpt[]
opts,
boolean serialized) {
-super(testname, opts, serialized);
+ super(opts, serialized, testname);
}
...
```
2. In the case of `TestDiskBalancerRPC.java`, it does remove JUnit 4's
`ExpectedException` variables, but it doesn't seem to rewrite the
`ExpectedException.expect` usages.
For instance,
```java
@Rule
public ExpectedException thrown = ExpectedException.none();
...
@Test
public void testSubmitPlanWithInvalidVersion() throws Exception {
...
thrown.expect(DiskBalancerException.class);
thrown.expect(new
DiskBalancerResultVerifier(Result.INVALID_PLAN_VERSION));
dataNode.submitDiskBalancerPlan(planHash, planVersion, PLAN_FILE,
plan.toJson(), false);
```
should have been rewritten into something like:
```java
final DiskBalancerException thrown =
Assertions.assertThrows(DiskBalancerException.class, () -> {
dataNode.submitDiskBalancerPlan(planHash, planVersion, PLAN_FILE,
plan.toJson(), false);
});
Assertions.assertEquals(thrown.getResult(), Result.INVALID_PLAN_VERSION);
```
... and some other interesting changes by rewrite plugin.
I have included some fixes my fork of your branch that addresses some of
those rewrite errors I mentioned above (and others):
https://github.com/smengcl/hadoop/compare/rewrite-junit5-hdfs..aajisaka:rewrite-junit5-hdfs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl commented on pull request #3304: HADOOP-14693. Test PR: Ran rewrite plugin on hadoop-hdfs module to upgrade to JUnit 5
smengcl commented on pull request #3304:
URL: https://github.com/apache/hadoop/pull/3304#issuecomment-902345346
Thanks @aajisaka for trying out the _rewrite_ plugin.
Overall the tool seems to be doing a pretty good job.
1. In the case of `TestHDFSContractPathHandle.java`, It somehow attempts to
import `org.junit.jupiter.api.Assertions.super` and rearrange the argument
order to `super()` call, which is kinda amusing.
```diff
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
index c65a60b18b1..a5ea66782cf 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestHDFSContractPathHandle.java
@@ -21,11 +21,13 @@
...
+import static org.junit.jupiter.api.Assertions.super;
+
/**
* Verify HDFS compliance with {@link org.apache.hadoop.fs.PathHandle}
* semantics.
@@ -35,15 +37,15 @@
public TestHDFSContractPathHandle(String testname, Options.HandleOpt[]
opts,
boolean serialized) {
-super(testname, opts, serialized);
+ super(opts, serialized, testname);
}
...
```
2. In the case of `TestDiskBalancerRPC.java`, it does remove JUnit 4's
`ExpectedException` variables, but it doesn't seem to rewrite the
`ExpectedException.expect` usages.
For instance,
```java
@Rule
public ExpectedException thrown = ExpectedException.none();
...
@Test
public void testSubmitPlanWithInvalidVersion() throws Exception {
...
thrown.expect(DiskBalancerException.class);
thrown.expect(new
DiskBalancerResultVerifier(Result.INVALID_PLAN_VERSION));
dataNode.submitDiskBalancerPlan(planHash, planVersion, PLAN_FILE,
plan.toJson(), false);
```
should have been rewritten into something like:
```java
final DiskBalancerException thrown =
Assertions.assertThrows(DiskBalancerException.class, () -> {
dataNode.submitDiskBalancerPlan(planHash, planVersion, PLAN_FILE,
plan.toJson(), false);
});
Assertions.assertEquals(thrown.getResult(), Result.INVALID_PLAN_VERSION);
```
... and some other interesting changes by rewrite plugin.
I have included some fixes my fork of your branch that addresses some of
those rewrite errors I mentioned above:
https://github.com/smengcl/hadoop/compare/rewrite-junit5-hdfs..aajisaka:rewrite-junit5-hdfs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3311: HDFS-16178. Make recursive rmdir in libhdfs++ cross platform
hadoop-yetus commented on pull request #3311:
URL: https://github.com/apache/hadoop/pull/3311#issuecomment-902335150
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 25m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 10s | | trunk passed |
| +1 :green_heart: | compile | 2m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 42m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 2m 34s | | the patch passed |
| +1 :green_heart: | cc | 2m 34s | | the patch passed |
| +1 :green_heart: | golang | 2m 34s | | the patch passed |
| +1 :green_heart: | javac | 2m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 86m 45s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 177m 1s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3311 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux 13a7f62566ec 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0ae3c13ceb40b6e8af2e8012e00dd9bec53ff94e |
| Default Java | Red Hat, Inc.-1.8.0_302-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/testReport/ |
| Max. process+thread count | 516 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/console |
| versions | git=2.27.0 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17857) Check real user ACLs in addition to proxied user ACLs
[
https://issues.apache.org/jira/browse/HADOOP-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17401886#comment-17401886
]
Hadoop QA commented on HADOOP-17857:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 24m
2s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
32s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
56s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
15s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 25s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 23m
40s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m
31s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 24m
45s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 24m
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 24m
8s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 24m
8s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 8s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-HADOOP-Build/227/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt{color}
| {color:orange} hadoop-common-project/hadoop-common: The patch generated 3
new + 39 unchanged - 1 fixed = 42 total (was 40) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 32s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green}{color} |
[GitHub] [hadoop] hadoop-yetus commented on pull request #3311: HDFS-16178. Make recursive rmdir in libhdfs++ cross platform
hadoop-yetus commented on pull request #3311:
URL: https://github.com/apache/hadoop/pull/3311#issuecomment-902253446
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 2s | | trunk passed |
| +1 :green_heart: | compile | 2m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 53m 13s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 15s | | the patch passed |
| +1 :green_heart: | compile | 2m 25s | | the patch passed |
| +1 :green_heart: | cc | 2m 25s | | the patch passed |
| +1 :green_heart: | golang | 2m 25s | | the patch passed |
| +1 :green_heart: | javac | 2m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 86m 4s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 161m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3311 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux b714f584b549 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0ae3c13ceb40b6e8af2e8012e00dd9bec53ff94e |
| Default Java | Red Hat, Inc.-1.8.0_302-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/testReport/ |
| Max. process+thread count | 517 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17851) Support user specified content encoding for S3A
[
https://issues.apache.org/jira/browse/HADOOP-17851?focusedWorklogId=640031=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-640031
]
ASF GitHub Bot logged work on HADOOP-17851:
---
Author: ASF GitHub Bot
Created on: 19/Aug/21 21:10
Start Date: 19/Aug/21 21:10
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3312:
URL: https://github.com/apache/hadoop/pull/3312#discussion_r692485795
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AContentEncoding.java
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.util.List;
+
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.fs.s3a.audit.S3AAuditConstants;
+import org.apache.hadoop.fs.s3a.impl.StoreContext;
+
+import static org.apache.hadoop.fs.s3a.Constants.CONTENT_ENCODING;
+import static
org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+
+/**
+ * Tests of content encoding object meta data.
+ */
+public class ITestS3AContentEncoding extends AbstractS3ATestBase {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(ITestS3ACannedACLs.class);
+
+ @Override
+ protected Configuration createConfiguration() {
+Configuration conf = super.createConfiguration();
+conf.set(CONTENT_ENCODING, "gzip");
+
+return conf;
+ }
+
+ @Test
+ public void testCreatedObjectsHaveEncoding() throws Throwable {
+S3AFileSystem fs = getFileSystem();
+Path dir = methodPath();
+fs.mkdirs(dir);
+Path path = new Path(dir, "1");
+ContractTestUtils.touch(fs, path);
+assertObjectHasEncoding(path);
+ }
+
+ /**
+ * Assert that a given object has gzip encoding specified
+ * @param path path
+ */
+ private void assertObjectHasEncoding(Path path) {
+S3AFileSystem fs = getFileSystem();
+
+StoreContext storeContext = fs.createStoreContext();
+AmazonS3 s3 = fs.getAmazonS3ClientForTesting("encoding");
Review comment:
no need for this, use getXAttr() on the path and the content encoding
header should be returns with some some header. prefix
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
##
@@ -410,6 +410,10 @@ private Constants() {
public static final String CANNED_ACL = "fs.s3a.acl.default";
public static final String DEFAULT_CANNED_ACL = "";
+ // gzip, deflate, compress, br, etc.
+ public static final String CONTENT_ENCODING = "fs.s3a.contentEncoding";
Review comment:
needs to be content.encoding ; s3a is lower case with dots only
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AContentEncoding.java
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.util.List;
+
+import com.amazonaws.services.s3.AmazonS3;
+import
[GitHub] [hadoop] steveloughran commented on a change in pull request #3312: HADOOP-17851 Support user specified content encoding for S3A
steveloughran commented on a change in pull request #3312:
URL: https://github.com/apache/hadoop/pull/3312#discussion_r692485795
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AContentEncoding.java
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.util.List;
+
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.fs.s3a.audit.S3AAuditConstants;
+import org.apache.hadoop.fs.s3a.impl.StoreContext;
+
+import static org.apache.hadoop.fs.s3a.Constants.CONTENT_ENCODING;
+import static
org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+
+/**
+ * Tests of content encoding object meta data.
+ */
+public class ITestS3AContentEncoding extends AbstractS3ATestBase {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(ITestS3ACannedACLs.class);
+
+ @Override
+ protected Configuration createConfiguration() {
+Configuration conf = super.createConfiguration();
+conf.set(CONTENT_ENCODING, "gzip");
+
+return conf;
+ }
+
+ @Test
+ public void testCreatedObjectsHaveEncoding() throws Throwable {
+S3AFileSystem fs = getFileSystem();
+Path dir = methodPath();
+fs.mkdirs(dir);
+Path path = new Path(dir, "1");
+ContractTestUtils.touch(fs, path);
+assertObjectHasEncoding(path);
+ }
+
+ /**
+ * Assert that a given object has gzip encoding specified
+ * @param path path
+ */
+ private void assertObjectHasEncoding(Path path) {
+S3AFileSystem fs = getFileSystem();
+
+StoreContext storeContext = fs.createStoreContext();
+AmazonS3 s3 = fs.getAmazonS3ClientForTesting("encoding");
Review comment:
no need for this, use getXAttr() on the path and the content encoding
header should be returns with some some header. prefix
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
##
@@ -410,6 +410,10 @@ private Constants() {
public static final String CANNED_ACL = "fs.s3a.acl.default";
public static final String DEFAULT_CANNED_ACL = "";
+ // gzip, deflate, compress, br, etc.
+ public static final String CONTENT_ENCODING = "fs.s3a.contentEncoding";
Review comment:
needs to be content.encoding ; s3a is lower case with dots only
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AContentEncoding.java
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import java.util.List;
+
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.fs.s3a.audit.S3AAuditConstants;
+import org.apache.hadoop.fs.s3a.impl.StoreContext;
+
+import static
[jira] [Updated] (HADOOP-17851) Support user specified content encoding for S3A
[ https://issues.apache.org/jira/browse/HADOOP-17851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17851: Parent: HADOOP-17566 Issue Type: Sub-task (was: Improvement) > Support user specified content encoding for S3A > --- > > Key: HADOOP-17851 > URL: https://issues.apache.org/jira/browse/HADOOP-17851 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Holden Karau >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > User-specified object content-encoding (part of the object metadata) is > important for allowing compressed files to be processed in the AWS ecosystem. > We should allow the user to specify the content encoding of the files being > written. > metadata can not be changed after a file is written without a rewrite. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17851) Support user specified content encoding for S3A
[ https://issues.apache.org/jira/browse/HADOOP-17851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17851: Component/s: (was: hadoop-thirdparty) fs/s3 > Support user specified content encoding for S3A > --- > > Key: HADOOP-17851 > URL: https://issues.apache.org/jira/browse/HADOOP-17851 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Holden Karau >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > User-specified object content-encoding (part of the object metadata) is > important for allowing compressed files to be processed in the AWS ecosystem. > We should allow the user to specify the content encoding of the files being > written. > metadata can not be changed after a file is written without a rewrite. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17855) S3A: Allow SSE configurations per object path
[
https://issues.apache.org/jira/browse/HADOOP-17855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17401859#comment-17401859
]
Steve Loughran commented on HADOOP-17855:
-
BTW, you know that when you do rename() in S3A FS, because it's a copy, things
get re-encrypted with the latest settings? So at the expense of that
duplication time, you can have something which outputs to one key but which is
post-processed into different KMS values by renaming by different FS instances
> S3A: Allow SSE configurations per object path
> -
>
> Key: HADOOP-17855
> URL: https://issues.apache.org/jira/browse/HADOOP-17855
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.1
>Reporter: Mike Dias
>Priority: Major
>
> Currently, we can map the SSE configurations at bucket level only:
> {code:java}
>
> fs.s3a.bucket.ireland-dev.server-side-encryption-algorithm
> SSE-KMS
>
>
> fs.s3a.bucket.ireland-dev.server-side-encryption.key
>
> arn:aws:kms:eu-west-1:98067faff834c:key/071a86ff-8881-4ba0-9230-95af6d01ca01
>
> {code}
> But sometimes we want to encrypt data in different paths with different keys
> within the same bucket. For example, a partitioned table might benefit from
> encrypting each partition with a different key when the partition represents
> a customer or a country.
> [S3 already can encrypt using different keys/configurations at the object
> level|https://aws.amazon.com/premiumsupport/knowledge-center/s3-encrypt-specific-folder/],
> so what we need to do on Hadoop is to provide a way to map which key to use.
> One idea could be mapping them in the XML config:
>
> {code:java}
>
> fs.s3a.server-side-encryption.paths
>
> s3://bucket/my_table/country=ireland,s3://bucket/my_table/country=uk,
> s3://bucket/my_table/country=germany
>
>
> fs.s3a.server-side-encryption.path-keys
>
> arn:aws:kms:eu-west-1:90ireland09:key/ireland-key,arn:aws:kms:eu-west-1:980uk0993c:key/uk-key,arn:aws:kms:eu-west-1:98germany089:key/germany-key
>
> {code}
> Or potentially fetch the mappings from the filesystem:
>
> {code:java}
>
> fs.s3a.server-side-encryption.mappings
> s3://bucket/configs/encryption_mappings.json
> {code}
> where encryption_mappings.json could be something like this:
>
> {code:java}
> {
>"path": "s3://bucket/customer_table/customerId=abc123",
>"algorithm": "SSE-KMS",
>"key": "arn:aws:kms:eu-west-1:933993746:key/abc123-key"
> }
> ...
> {
>"path": "s3://bucket/customer_table/customerId=xyx987",
>"algorithm": "SSE-KMS",
>"key": "arn:aws:kms:eu-west-1:933993746:key/xyx987-key"
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17855) S3A: Allow SSE configurations per object path
[
https://issues.apache.org/jira/browse/HADOOP-17855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17401858#comment-17401858
]
Steve Loughran commented on HADOOP-17855:
-
Pretty reluctant to do this -at least on my personal development schedule.
* when we do things with directories, we often create markers in parent dirs.
This complicates life as we'd have to choose which to use there too
* S3A Delegation tokens pass down all encryption settings so that you can
submit work into a shared cluster where all encryption options including your
secrets come with the job. This will need to be extended.
* all the usual stuff related to hierarchical references, duplicate conflicting
entries et cetera et cetera.
* would you support different SSE options (SSE-C vs SSE-KMS)? SSE-KMS is the
only sensible option, really.
That said: these are all tractable and I can see the rationale for it. If you
were to work on this I and others will do what we can to help nurture the
change in.
I look forward to your submission, please follow the documented test process.
(This probably complicates testing even more as you will need 2+ KMS keys. Docs
will need to be updated...)
> S3A: Allow SSE configurations per object path
> -
>
> Key: HADOOP-17855
> URL: https://issues.apache.org/jira/browse/HADOOP-17855
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mike Dias
>Priority: Major
>
> Currently, we can map the SSE configurations at bucket level only:
> {code:java}
>
> fs.s3a.bucket.ireland-dev.server-side-encryption-algorithm
> SSE-KMS
>
>
> fs.s3a.bucket.ireland-dev.server-side-encryption.key
>
> arn:aws:kms:eu-west-1:98067faff834c:key/071a86ff-8881-4ba0-9230-95af6d01ca01
>
> {code}
> But sometimes we want to encrypt data in different paths with different keys
> within the same bucket. For example, a partitioned table might benefit from
> encrypting each partition with a different key when the partition represents
> a customer or a country.
> [S3 already can encrypt using different keys/configurations at the object
> level|https://aws.amazon.com/premiumsupport/knowledge-center/s3-encrypt-specific-folder/],
> so what we need to do on Hadoop is to provide a way to map which key to use.
> One idea could be mapping them in the XML config:
>
> {code:java}
>
> fs.s3a.server-side-encryption.paths
>
> s3://bucket/my_table/country=ireland,s3://bucket/my_table/country=uk,
> s3://bucket/my_table/country=germany
>
>
> fs.s3a.server-side-encryption.path-keys
>
> arn:aws:kms:eu-west-1:90ireland09:key/ireland-key,arn:aws:kms:eu-west-1:980uk0993c:key/uk-key,arn:aws:kms:eu-west-1:98germany089:key/germany-key
>
> {code}
> Or potentially fetch the mappings from the filesystem:
>
> {code:java}
>
> fs.s3a.server-side-encryption.mappings
> s3://bucket/configs/encryption_mappings.json
> {code}
> where encryption_mappings.json could be something like this:
>
> {code:java}
> {
>"path": "s3://bucket/customer_table/customerId=abc123",
>"algorithm": "SSE-KMS",
>"key": "arn:aws:kms:eu-west-1:933993746:key/abc123-key"
> }
> ...
> {
>"path": "s3://bucket/customer_table/customerId=xyx987",
>"algorithm": "SSE-KMS",
>"key": "arn:aws:kms:eu-west-1:933993746:key/xyx987-key"
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17855) S3A: Allow SSE configurations per object path
[
https://issues.apache.org/jira/browse/HADOOP-17855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-17855:
Affects Version/s: 3.3.1
> S3A: Allow SSE configurations per object path
> -
>
> Key: HADOOP-17855
> URL: https://issues.apache.org/jira/browse/HADOOP-17855
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.1
>Reporter: Mike Dias
>Priority: Major
>
> Currently, we can map the SSE configurations at bucket level only:
> {code:java}
>
> fs.s3a.bucket.ireland-dev.server-side-encryption-algorithm
> SSE-KMS
>
>
> fs.s3a.bucket.ireland-dev.server-side-encryption.key
>
> arn:aws:kms:eu-west-1:98067faff834c:key/071a86ff-8881-4ba0-9230-95af6d01ca01
>
> {code}
> But sometimes we want to encrypt data in different paths with different keys
> within the same bucket. For example, a partitioned table might benefit from
> encrypting each partition with a different key when the partition represents
> a customer or a country.
> [S3 already can encrypt using different keys/configurations at the object
> level|https://aws.amazon.com/premiumsupport/knowledge-center/s3-encrypt-specific-folder/],
> so what we need to do on Hadoop is to provide a way to map which key to use.
> One idea could be mapping them in the XML config:
>
> {code:java}
>
> fs.s3a.server-side-encryption.paths
>
> s3://bucket/my_table/country=ireland,s3://bucket/my_table/country=uk,
> s3://bucket/my_table/country=germany
>
>
> fs.s3a.server-side-encryption.path-keys
>
> arn:aws:kms:eu-west-1:90ireland09:key/ireland-key,arn:aws:kms:eu-west-1:980uk0993c:key/uk-key,arn:aws:kms:eu-west-1:98germany089:key/germany-key
>
> {code}
> Or potentially fetch the mappings from the filesystem:
>
> {code:java}
>
> fs.s3a.server-side-encryption.mappings
> s3://bucket/configs/encryption_mappings.json
> {code}
> where encryption_mappings.json could be something like this:
>
> {code:java}
> {
>"path": "s3://bucket/customer_table/customerId=abc123",
>"algorithm": "SSE-KMS",
>"key": "arn:aws:kms:eu-west-1:933993746:key/abc123-key"
> }
> ...
> {
>"path": "s3://bucket/customer_table/customerId=xyx987",
>"algorithm": "SSE-KMS",
>"key": "arn:aws:kms:eu-west-1:933993746:key/xyx987-key"
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17855) S3A: Allow SSE configurations per object path
[
https://issues.apache.org/jira/browse/HADOOP-17855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-17855:
Parent: HADOOP-17566
Issue Type: Sub-task (was: New Feature)
> S3A: Allow SSE configurations per object path
> -
>
> Key: HADOOP-17855
> URL: https://issues.apache.org/jira/browse/HADOOP-17855
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mike Dias
>Priority: Major
>
> Currently, we can map the SSE configurations at bucket level only:
> {code:java}
>
> fs.s3a.bucket.ireland-dev.server-side-encryption-algorithm
> SSE-KMS
>
>
> fs.s3a.bucket.ireland-dev.server-side-encryption.key
>
> arn:aws:kms:eu-west-1:98067faff834c:key/071a86ff-8881-4ba0-9230-95af6d01ca01
>
> {code}
> But sometimes we want to encrypt data in different paths with different keys
> within the same bucket. For example, a partitioned table might benefit from
> encrypting each partition with a different key when the partition represents
> a customer or a country.
> [S3 already can encrypt using different keys/configurations at the object
> level|https://aws.amazon.com/premiumsupport/knowledge-center/s3-encrypt-specific-folder/],
> so what we need to do on Hadoop is to provide a way to map which key to use.
> One idea could be mapping them in the XML config:
>
> {code:java}
>
> fs.s3a.server-side-encryption.paths
>
> s3://bucket/my_table/country=ireland,s3://bucket/my_table/country=uk,
> s3://bucket/my_table/country=germany
>
>
> fs.s3a.server-side-encryption.path-keys
>
> arn:aws:kms:eu-west-1:90ireland09:key/ireland-key,arn:aws:kms:eu-west-1:980uk0993c:key/uk-key,arn:aws:kms:eu-west-1:98germany089:key/germany-key
>
> {code}
> Or potentially fetch the mappings from the filesystem:
>
> {code:java}
>
> fs.s3a.server-side-encryption.mappings
> s3://bucket/configs/encryption_mappings.json
> {code}
> where encryption_mappings.json could be something like this:
>
> {code:java}
> {
>"path": "s3://bucket/customer_table/customerId=abc123",
>"algorithm": "SSE-KMS",
>"key": "arn:aws:kms:eu-west-1:933993746:key/abc123-key"
> }
> ...
> {
>"path": "s3://bucket/customer_table/customerId=xyx987",
>"algorithm": "SSE-KMS",
>"key": "arn:aws:kms:eu-west-1:933993746:key/xyx987-key"
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17855) S3A: Allow SSE configurations per object path
[
https://issues.apache.org/jira/browse/HADOOP-17855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-17855:
Summary: S3A: Allow SSE configurations per object path (was: Allow SSE
configurations per object path)
> S3A: Allow SSE configurations per object path
> -
>
> Key: HADOOP-17855
> URL: https://issues.apache.org/jira/browse/HADOOP-17855
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
>Reporter: Mike Dias
>Priority: Major
>
> Currently, we can map the SSE configurations at bucket level only:
> {code:java}
>
> fs.s3a.bucket.ireland-dev.server-side-encryption-algorithm
> SSE-KMS
>
>
> fs.s3a.bucket.ireland-dev.server-side-encryption.key
>
> arn:aws:kms:eu-west-1:98067faff834c:key/071a86ff-8881-4ba0-9230-95af6d01ca01
>
> {code}
> But sometimes we want to encrypt data in different paths with different keys
> within the same bucket. For example, a partitioned table might benefit from
> encrypting each partition with a different key when the partition represents
> a customer or a country.
> [S3 already can encrypt using different keys/configurations at the object
> level|https://aws.amazon.com/premiumsupport/knowledge-center/s3-encrypt-specific-folder/],
> so what we need to do on Hadoop is to provide a way to map which key to use.
> One idea could be mapping them in the XML config:
>
> {code:java}
>
> fs.s3a.server-side-encryption.paths
>
> s3://bucket/my_table/country=ireland,s3://bucket/my_table/country=uk,
> s3://bucket/my_table/country=germany
>
>
> fs.s3a.server-side-encryption.path-keys
>
> arn:aws:kms:eu-west-1:90ireland09:key/ireland-key,arn:aws:kms:eu-west-1:980uk0993c:key/uk-key,arn:aws:kms:eu-west-1:98germany089:key/germany-key
>
> {code}
> Or potentially fetch the mappings from the filesystem:
>
> {code:java}
>
> fs.s3a.server-side-encryption.mappings
> s3://bucket/configs/encryption_mappings.json
> {code}
> where encryption_mappings.json could be something like this:
>
> {code:java}
> {
>"path": "s3://bucket/customer_table/customerId=abc123",
>"algorithm": "SSE-KMS",
>"key": "arn:aws:kms:eu-west-1:933993746:key/abc123-key"
> }
> ...
> {
>"path": "s3://bucket/customer_table/customerId=xyx987",
>"algorithm": "SSE-KMS",
>"key": "arn:aws:kms:eu-west-1:933993746:key/xyx987-key"
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17857) Check real user ACLs in addition to proxied user ACLs
[
https://issues.apache.org/jira/browse/HADOOP-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne reassigned HADOOP-17857:
---
Assignee: Eric Payne
> Check real user ACLs in addition to proxied user ACLs
> -
>
> Key: HADOOP-17857
> URL: https://issues.apache.org/jira/browse/HADOOP-17857
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.2.2, 2.10.1, 3.3.1
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Major
> Attachments: HADOOP-17857.001.patch
>
>
> In a secure cluster, it is possible to configure the services to allow a
> super-user to proxy to a regular user and perform actions on behalf of the
> proxied user (see [Proxy user - Superusers Acting On Behalf Of Other
> Users|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html]).
> This is useful for automating server access for multiple different users in a
> multi-tenant cluster. For example, this can be used by a super user
> submitting jobs to a YARN queue, accessing HDFS files, scheduling Oozie
> workflows, etc, which will then execute the service as the proxied user.
> Usually when these services check ACLs to determine if the user has access to
> the requested resources, the service only needs to check the ACLs for the
> proxied user. However, it is sometimes desirable to allow the proxied user to
> have access to the resources when only the real user has open ACLs.
> For instance, let's say the user {{adm}} is the only user with submit ACLs to
> the {{dataload}} queue, and the {{adm}} user wants to submit apps to the
> {{dataload}} queue on behalf of users {{headless1}} and {{headless2}}. In
> addition, we want to be able to bill {{headless1}} and {{headless2}}
> separately for the YARN resources used in the {{dataload}} queue. In order to
> do this, the apps need to run in the {{dataload}} queue as the respective
> headless users. We could open up the ACLs to the {{dataload}} queue to allow
> {{headless1}} and {{headless2}} to submit apps. But this would allow those
> users to submit any app to that queue, and not be limited to just the data
> loading apps, and we don't trust the {{headless1}} and {{headless2}} owners
> to honor that restriction.
> This JIRA proposes that we define a way to set up ACLs to restrict a
> resource's access to a super-user, but when the access happens, run it as
> the proxied user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17857) Check real user ACLs in addition to proxied user ACLs
[
https://issues.apache.org/jira/browse/HADOOP-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated HADOOP-17857:
Attachment: HADOOP-17857.001.patch
> Check real user ACLs in addition to proxied user ACLs
> -
>
> Key: HADOOP-17857
> URL: https://issues.apache.org/jira/browse/HADOOP-17857
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.2.2, 2.10.1, 3.3.1
>Reporter: Eric Payne
>Priority: Major
> Attachments: HADOOP-17857.001.patch
>
>
> In a secure cluster, it is possible to configure the services to allow a
> super-user to proxy to a regular user and perform actions on behalf of the
> proxied user (see [Proxy user - Superusers Acting On Behalf Of Other
> Users|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html]).
> This is useful for automating server access for multiple different users in a
> multi-tenant cluster. For example, this can be used by a super user
> submitting jobs to a YARN queue, accessing HDFS files, scheduling Oozie
> workflows, etc, which will then execute the service as the proxied user.
> Usually when these services check ACLs to determine if the user has access to
> the requested resources, the service only needs to check the ACLs for the
> proxied user. However, it is sometimes desirable to allow the proxied user to
> have access to the resources when only the real user has open ACLs.
> For instance, let's say the user {{adm}} is the only user with submit ACLs to
> the {{dataload}} queue, and the {{adm}} user wants to submit apps to the
> {{dataload}} queue on behalf of users {{headless1}} and {{headless2}}. In
> addition, we want to be able to bill {{headless1}} and {{headless2}}
> separately for the YARN resources used in the {{dataload}} queue. In order to
> do this, the apps need to run in the {{dataload}} queue as the respective
> headless users. We could open up the ACLs to the {{dataload}} queue to allow
> {{headless1}} and {{headless2}} to submit apps. But this would allow those
> users to submit any app to that queue, and not be limited to just the data
> loading apps, and we don't trust the {{headless1}} and {{headless2}} owners
> to honor that restriction.
> This JIRA proposes that we define a way to set up ACLs to restrict a
> resource's access to a super-user, but when the access happens, run it as
> the proxied user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17857) Check real user ACLs in addition to proxied user ACLs
[
https://issues.apache.org/jira/browse/HADOOP-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated HADOOP-17857:
Status: Patch Available (was: Open)
> Check real user ACLs in addition to proxied user ACLs
> -
>
> Key: HADOOP-17857
> URL: https://issues.apache.org/jira/browse/HADOOP-17857
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.1, 2.10.1, 3.2.2
>Reporter: Eric Payne
>Priority: Major
> Attachments: HADOOP-17857.001.patch
>
>
> In a secure cluster, it is possible to configure the services to allow a
> super-user to proxy to a regular user and perform actions on behalf of the
> proxied user (see [Proxy user - Superusers Acting On Behalf Of Other
> Users|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html]).
> This is useful for automating server access for multiple different users in a
> multi-tenant cluster. For example, this can be used by a super user
> submitting jobs to a YARN queue, accessing HDFS files, scheduling Oozie
> workflows, etc, which will then execute the service as the proxied user.
> Usually when these services check ACLs to determine if the user has access to
> the requested resources, the service only needs to check the ACLs for the
> proxied user. However, it is sometimes desirable to allow the proxied user to
> have access to the resources when only the real user has open ACLs.
> For instance, let's say the user {{adm}} is the only user with submit ACLs to
> the {{dataload}} queue, and the {{adm}} user wants to submit apps to the
> {{dataload}} queue on behalf of users {{headless1}} and {{headless2}}. In
> addition, we want to be able to bill {{headless1}} and {{headless2}}
> separately for the YARN resources used in the {{dataload}} queue. In order to
> do this, the apps need to run in the {{dataload}} queue as the respective
> headless users. We could open up the ACLs to the {{dataload}} queue to allow
> {{headless1}} and {{headless2}} to submit apps. But this would allow those
> users to submit any app to that queue, and not be limited to just the data
> loading apps, and we don't trust the {{headless1}} and {{headless2}} owners
> to honor that restriction.
> This JIRA proposes that we define a way to set up ACLs to restrict a
> resource's access to a super-user, but when the access happens, run it as
> the proxied user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17857) Check real user ACLs in addition to proxied user ACLs
[
https://issues.apache.org/jira/browse/HADOOP-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17401820#comment-17401820
]
Eric Payne commented on HADOOP-17857:
-
I suggest that we define the ACLs so that a special character tells the
AccessControlList system to check the ACLs for the real user and not those for
the proxied user.
Let's take and example for submitting jobs in the Capacity Scheduler into the
{{dataload}} queue. The use case is to only allow {{adm}} to submit to the
{{dataload}} queue, but once the apps are submitted, they run as a proxy user
(like {{headless1}}):
The following is the current syntax, and it only allows the {{adm}} user, as
itself, to submit jobs to the {{dataload}} queue.
{code:xml}
yarn.scheduler.capacity.root.dataload.acl_submit_applications
adm
{code}
With this syntax, if the {{adm}} user proxies to {{headless1}} and submits the
job, the Capacity Scheduler will reject the submission because {{headless1}}
does not have submit ACL permissions.
*PROPOSED CHANGES:*
- Add a tilde (~) to the beginning of the {{adm}} user in the value section of
the property.
In the above example, note the additon of the tilde (~):
{code:xml}
yarn.scheduler.capacity.root.dataload.acl_submit_applications
~adm
{code}
- With the tilde (~}, any proxied user submitted by the {{adm}} user will be
allowed to run in the {{dataload}} queue.
- That same proxied user will _not_ be allowed to submit by themselves if
they are not first proxied by {{adm}}.
- NOTE: with this syntax, {{adm}} will not be able to directly submit as
itself to the {{dataload}} queue. In order to both submit as {{adm}} and also
allow an {{adm}}-proxied user to submit to the {{dataload}} queue, both
{{~adm}} and {{adm}} must be specified, as follows:
{code:xml}
yarn.scheduler.capacity.root.dataload.acl_submit_applications
~adm,adm
{code}
This example could be extended to other ACL properties in other Hadoop systems.
We have been running with this change in production for over a year now, and it
works well.
> Check real user ACLs in addition to proxied user ACLs
> -
>
> Key: HADOOP-17857
> URL: https://issues.apache.org/jira/browse/HADOOP-17857
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.2.2, 2.10.1, 3.3.1
>Reporter: Eric Payne
>Priority: Major
>
> In a secure cluster, it is possible to configure the services to allow a
> super-user to proxy to a regular user and perform actions on behalf of the
> proxied user (see [Proxy user - Superusers Acting On Behalf Of Other
> Users|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html]).
> This is useful for automating server access for multiple different users in a
> multi-tenant cluster. For example, this can be used by a super user
> submitting jobs to a YARN queue, accessing HDFS files, scheduling Oozie
> workflows, etc, which will then execute the service as the proxied user.
> Usually when these services check ACLs to determine if the user has access to
> the requested resources, the service only needs to check the ACLs for the
> proxied user. However, it is sometimes desirable to allow the proxied user to
> have access to the resources when only the real user has open ACLs.
> For instance, let's say the user {{adm}} is the only user with submit ACLs to
> the {{dataload}} queue, and the {{adm}} user wants to submit apps to the
> {{dataload}} queue on behalf of users {{headless1}} and {{headless2}}. In
> addition, we want to be able to bill {{headless1}} and {{headless2}}
> separately for the YARN resources used in the {{dataload}} queue. In order to
> do this, the apps need to run in the {{dataload}} queue as the respective
> headless users. We could open up the ACLs to the {{dataload}} queue to allow
> {{headless1}} and {{headless2}} to submit apps. But this would allow those
> users to submit any app to that queue, and not be limited to just the data
> loading apps, and we don't trust the {{headless1}} and {{headless2}} owners
> to honor that restriction.
> This JIRA proposes that we define a way to set up ACLs to restrict a
> resource's access to a super-user, but when the access happens, run it as
> the proxied user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17857) Check real user ACLs in addition to proxied user ACLs
Eric Payne created HADOOP-17857:
---
Summary: Check real user ACLs in addition to proxied user ACLs
Key: HADOOP-17857
URL: https://issues.apache.org/jira/browse/HADOOP-17857
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 3.3.1, 2.10.1, 3.2.2
Reporter: Eric Payne
In a secure cluster, it is possible to configure the services to allow a
super-user to proxy to a regular user and perform actions on behalf of the
proxied user (see [Proxy user - Superusers Acting On Behalf Of Other
Users|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html]).
This is useful for automating server access for multiple different users in a
multi-tenant cluster. For example, this can be used by a super user submitting
jobs to a YARN queue, accessing HDFS files, scheduling Oozie workflows, etc,
which will then execute the service as the proxied user.
Usually when these services check ACLs to determine if the user has access to
the requested resources, the service only needs to check the ACLs for the
proxied user. However, it is sometimes desirable to allow the proxied user to
have access to the resources when only the real user has open ACLs.
For instance, let's say the user {{adm}} is the only user with submit ACLs to
the {{dataload}} queue, and the {{adm}} user wants to submit apps to the
{{dataload}} queue on behalf of users {{headless1}} and {{headless2}}. In
addition, we want to be able to bill {{headless1}} and {{headless2}} separately
for the YARN resources used in the {{dataload}} queue. In order to do this, the
apps need to run in the {{dataload}} queue as the respective headless users. We
could open up the ACLs to the {{dataload}} queue to allow {{headless1}} and
{{headless2}} to submit apps. But this would allow those users to submit any
app to that queue, and not be limited to just the data loading apps, and we
don't trust the {{headless1}} and {{headless2}} owners to honor that
restriction.
This JIRA proposes that we define a way to set up ACLs to restrict a resource's
access to a super-user, but when the access happens, run it as the proxied
user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3309: Hadoop 17853. ABFS: Enable optional store connectivity over azure specific protocol for data egress
hadoop-yetus commented on pull request #3309:
URL: https://github.com/apache/hadoop/pull/3309#issuecomment-902121514
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 11s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 41 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 5s | | trunk passed |
| +1 :green_heart: | compile | 0m 51s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 49s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 21s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 20m 40s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 0m 36s |
[/results-compile-javac-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3309/5/artifact/out/results-compile-javac-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with
JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 26 unchanged - 1
fixed = 27 total (was 27) |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 0m 33s |
[/results-compile-javac-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3309/5/artifact/out/results-compile-javac-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
|
hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new
+ 20 unchanged - 1 fixed = 21 total (was 21) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3309/5/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 133 new + 15 unchanged - 4
fixed = 148 total (was 19) |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | spotbugs | 1m 22s |
[/new-spotbugs-hadoop-tools_hadoop-azure.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3309/5/artifact/out/new-spotbugs-hadoop-tools_hadoop-azure.html)
| hadoop-tools/hadoop-azure generated 4 new + 0 unchanged - 0 fixed = 4 total
(was 0) |
| +1 :green_heart: | shadedclient | 19m 29s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 6s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 94m 6s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-tools/hadoop-azure |
| | Null passed for non-null parameter of
setStatusFromFastpathResponse(FastpathResponse) in
[jira] [Work logged] (HADOOP-17717) Update wildfly openssl to 1.1.3.Final
[ https://issues.apache.org/jira/browse/HADOOP-17717?focusedWorklogId=639960=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-639960 ] ASF GitHub Bot logged work on HADOOP-17717: --- Author: ASF GitHub Bot Created on: 19/Aug/21 17:44 Start Date: 19/Aug/21 17:44 Worklog Time Spent: 10m Work Description: jasonwzs commented on pull request #3029: URL: https://github.com/apache/hadoop/pull/3029#issuecomment-902115296 @steveloughran, is there any update on this fix? We are also waiting for the fix of this vulnerability issue. What tests need to be done in order to approve this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 639960) Time Spent: 0.5h (was: 20m) > Update wildfly openssl to 1.1.3.Final > - > > Key: HADOOP-17717 > URL: https://issues.apache.org/jira/browse/HADOOP-17717 > Project: Hadoop Common > Issue Type: Bug >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > HADOOP-17649 got stalled. IMO we can bump the version to 1.1.3.Final instead, > at least, for branch-3.3. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jasonwzs commented on pull request #3029: HADOOP-17717. Update wildfly openssl to 1.1.3.Final.
jasonwzs commented on pull request #3029: URL: https://github.com/apache/hadoop/pull/3029#issuecomment-902115296 @steveloughran, is there any update on this fix? We are also waiting for the fix of this vulnerability issue. What tests need to be done in order to approve this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on pull request #3309: Hadoop 17853. ABFS: Enable optional store connectivity over azure specific protocol for data egress
snvijaya commented on pull request #3309:
URL: https://github.com/apache/hadoop/pull/3309#issuecomment-902068996
AGGREGATED TEST RESULT
HNS-OAuth
[INFO] Results:
[INFO]
[WARNING] Tests run: 111, Failures: 0, Errors: 0, Skipped: 1
[INFO] Results:
[INFO]
[WARNING] Tests run: 626, Failures: 0, Errors: 0, Skipped: 101
[INFO] Results:
[INFO]
[WARNING] Tests run: 288, Failures: 0, Errors: 0, Skipped: 52
HNS-SharedKey
[INFO] Results:
[INFO]
[WARNING] Tests run: 111, Failures: 0, Errors: 0, Skipped: 7
[INFO] Results:
[INFO]
[WARNING] Tests run: 626, Failures: 0, Errors: 0, Skipped: 137
[INFO] Results:
[INFO]
[WARNING] Tests run: 287, Failures: 0, Errors: 0, Skipped: 68
NonHNS-SharedKey
[INFO] Results:
[INFO]
[WARNING] Tests run: 111, Failures: 0, Errors: 0, Skipped: 7
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS:181
Expecting org.apache.hadoop.security.AccessControlException with text "This
request is not authorized to perform this operation using this permission.",
403 but got : "void"
[INFO]
[ERROR] Tests run: 626, Failures: 1, Errors: 0, Skipped: 346
[INFO] Results:
[INFO]
[WARNING] Tests run: 287, Failures: 0, Errors: 0, Skipped: 68
AppendBlob-HNS-OAuth
[INFO] Results:
[INFO]
[WARNING] Tests run: 111, Failures: 0, Errors: 0, Skipped: 1
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]
ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendNoInfiniteLease:178->twoWriters:166
» IO
[INFO]
[ERROR] Tests run: 626, Failures: 0, Errors: 1, Skipped: 101
[INFO] Results:
[INFO]
[WARNING] Tests run: 288, Failures: 0, Errors: 0, Skipped: 100
Test failure observed in NonHNS-SharedKey:
ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS
is fixed as part of https://issues.apache.org/jira/browse/HADOOP-17767
Test failure observed in AppendBlob-HNS-OAuth:
ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendNoInfiniteLease
will be fixed as part of https://issues.apache.org/jira/browse/HADOOP-17590
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3314: YARN-10891. Extend QueueInfo with max-parallel-apps in CS.
hadoop-yetus commented on pull request #3314:
URL: https://github.com/apache/hadoop/pull/3314#issuecomment-902061972
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 41s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 53s | | trunk passed |
| +1 :green_heart: | compile | 12m 55s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 10m 49s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 13s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 19s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 33s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 8m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 43s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 8s | | the patch passed |
| +1 :green_heart: | compile | 12m 6s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | cc | 12m 6s | | the patch passed |
| +1 :green_heart: | javac | 12m 6s | | the patch passed |
| +1 :green_heart: | compile | 10m 58s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | cc | 10m 58s | | the patch passed |
| +1 :green_heart: | javac | 10m 58s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 54s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 3s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 16s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 5s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 8m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 9s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 15s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 19s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 102m 57s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 27m 19s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 316m 1s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3314/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat |
| uname | Linux 5365ae19b87a 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 58af85160c3b883e31d76ebb5257083995261dfc |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3314/2/testReport/ |
| Max. process+thread count | 934 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
[GitHub] [hadoop] hadoop-yetus commented on pull request #3315: HDFS-16180.FsVolumeImpl.nextBlock should consider that the block meta file has been deleted
hadoop-yetus commented on pull request #3315:
URL: https://github.com/apache/hadoop/pull/3315#issuecomment-902004529
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 57s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 13s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 53s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3315/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 13 unchanged -
0 fixed = 15 total (was 13) |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 17s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 247m 14s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3315/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 331m 31s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.TestReadStripedFileWithDecodingDeletedData |
| | hadoop.hdfs.TestSetrepIncreasing |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3315/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3315 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 399651e1a04b 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5df9846bc4b5c0b3949306816b1cfb69066f1ed9 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3315/1/testReport/ |
| Max. process+thread count | 3394 (vs. ulimit of 5500) |
|
[jira] [Work logged] (HADOOP-17156) Clear abfs readahead requests on stream close
[ https://issues.apache.org/jira/browse/HADOOP-17156?focusedWorklogId=639883=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-639883 ] ASF GitHub Bot logged work on HADOOP-17156: --- Author: ASF GitHub Bot Created on: 19/Aug/21 14:20 Start Date: 19/Aug/21 14:20 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #3285: URL: https://github.com/apache/hadoop/pull/3285#issuecomment-901956956 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 20s | | https://github.com/apache/hadoop/pull/3285 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/3285 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3285/3/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 639883) Time Spent: 1h 40m (was: 1.5h) > Clear abfs readahead requests on stream close > - > > Key: HADOOP-17156 > URL: https://issues.apache.org/jira/browse/HADOOP-17156 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Rajesh Balamohan >Assignee: Mukund Thakur >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > It would be good to close/clear pending read ahead requests on stream close(). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3285: HADOOP-17156. Purging the buffers associated with input streams during close()
hadoop-yetus commented on pull request #3285: URL: https://github.com/apache/hadoop/pull/3285#issuecomment-901956956 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 20s | | https://github.com/apache/hadoop/pull/3285 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/3285 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3285/3/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17156) Clear abfs readahead requests on stream close
[ https://issues.apache.org/jira/browse/HADOOP-17156?focusedWorklogId=639880=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-639880 ] ASF GitHub Bot logged work on HADOOP-17156: --- Author: ASF GitHub Bot Created on: 19/Aug/21 14:13 Start Date: 19/Aug/21 14:13 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3285: URL: https://github.com/apache/hadoop/pull/3285#issuecomment-901951629 parallel test runs execute in different processes; that is not the direct cause of test failure Possible * state of remote store (including paths) * things just being slow on an overloaded machines, so cleanup happening earlier * state of recycled FS being such that this test case fails -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 639880) Time Spent: 1.5h (was: 1h 20m) > Clear abfs readahead requests on stream close > - > > Key: HADOOP-17156 > URL: https://issues.apache.org/jira/browse/HADOOP-17156 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Rajesh Balamohan >Assignee: Mukund Thakur >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > It would be good to close/clear pending read ahead requests on stream close(). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3285: HADOOP-17156. Purging the buffers associated with input streams during close()
steveloughran commented on pull request #3285: URL: https://github.com/apache/hadoop/pull/3285#issuecomment-901951629 parallel test runs execute in different processes; that is not the direct cause of test failure Possible * state of remote store (including paths) * things just being slow on an overloaded machines, so cleanup happening earlier * state of recycled FS being such that this test case fails -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17156) Clear abfs readahead requests on stream close
[
https://issues.apache.org/jira/browse/HADOOP-17156?focusedWorklogId=639878=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-639878
]
ASF GitHub Bot logged work on HADOOP-17156:
---
Author: ASF GitHub Bot
Created on: 19/Aug/21 14:11
Start Date: 19/Aug/21 14:11
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3285:
URL: https://github.com/apache/hadoop/pull/3285#issuecomment-895915684
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 46s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 3s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 18s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3285/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 7 new + 0 unchanged - 0
fixed = 7 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 55s | | hadoop-azure in the patch
passed. |
| -1 :x: | asflicense | 0m 35s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3285/1/artifact/out/results-asflicense.txt)
| The patch generated 1 ASF License warnings. |
| | | 72m 17s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3285/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3285 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux ccee87c58fd4 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9
23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 475d45338a1b51cef5d139eb1be580c2f40cf137 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3285/1/testReport/ |
| Max.
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3285: HADOOP-17156. Purging the buffers associated with input streams during close()
hadoop-yetus removed a comment on pull request #3285:
URL: https://github.com/apache/hadoop/pull/3285#issuecomment-895915684
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 46s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 3s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 18s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3285/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 7 new + 0 unchanged - 0
fixed = 7 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 55s | | hadoop-azure in the patch
passed. |
| -1 :x: | asflicense | 0m 35s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3285/1/artifact/out/results-asflicense.txt)
| The patch generated 1 ASF License warnings. |
| | | 72m 17s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3285/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3285 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux ccee87c58fd4 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9
23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 475d45338a1b51cef5d139eb1be580c2f40cf137 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3285/1/testReport/ |
| Max. process+thread count | 543 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3285/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an
[jira] [Created] (HADOOP-17856) JsonSerDeser to collect IOStatistics
Steve Loughran created HADOOP-17856: --- Summary: JsonSerDeser to collect IOStatistics Key: HADOOP-17856 URL: https://issues.apache.org/jira/browse/HADOOP-17856 Project: Hadoop Common Issue Type: Sub-task Components: common Affects Versions: 3.3.1 Reporter: Steve Loughran json deserializer to build stats on reading costs, which can then be collected too to measure cost of ser/deser and of file IO from the input/output streams, if they provide it. Allows for committers to report costs better here. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17833) Improve Magic Committer Performane
[ https://issues.apache.org/jira/browse/HADOOP-17833?focusedWorklogId=639874=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-639874 ] ASF GitHub Bot logged work on HADOOP-17833: --- Author: ASF GitHub Bot Created on: 19/Aug/21 14:04 Start Date: 19/Aug/21 14:04 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #3289: URL: https://github.com/apache/hadoop/pull/3289#issuecomment-901943721 +plan to lift some of the statistic names from the manifest committer and do the same reporting as in manifest committer. will also include list costs in results. (side issue, thinking of whether the json deserializer could build stats on reading costs, which can then be collected too to measure cost of ser/deser and, by collecting stream read/write costs, those steps -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 639874) Time Spent: 1h (was: 50m) > Improve Magic Committer Performane > -- > > Key: HADOOP-17833 > URL: https://issues.apache.org/jira/browse/HADOOP-17833 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Priority: Minor > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Magic committer tasks can be slow because every file created with > overwrite=false triggers a HEAD (verify there's no file) and a LIST (that > there's no dir). And because of delayed manifestations, it may not behave as > expected. > ParquetOutputFormat is one example of a library which does this. > we could fix parquet to use overwrite=true, but (a) there may be surprises in > other uses (b) it'd still leave the list and (c) do nothing for other formats > call > Proposed: createFile() under a magic path to skip all probes for file/dir at > end of path > Only a single task attempt Will be writing to that directory and it should > know what it is doing. If there is conflicting file names and parts across > tasks that won't even get picked up at this point. Oh and none of the > committers ever check for this: you'll get the last file manifested (s3a) or > renamed (file) > If we skip the checks we will save 2 HTTP requests/file. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3289: HADOOP-17833. Improve Magic Committer performance
steveloughran commented on pull request #3289: URL: https://github.com/apache/hadoop/pull/3289#issuecomment-901943721 +plan to lift some of the statistic names from the manifest committer and do the same reporting as in manifest committer. will also include list costs in results. (side issue, thinking of whether the json deserializer could build stats on reading costs, which can then be collected too to measure cost of ser/deser and, by collecting stream read/write costs, those steps -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17833) Improve Magic Committer Performane
[
https://issues.apache.org/jira/browse/HADOOP-17833?focusedWorklogId=639873=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-639873
]
ASF GitHub Bot logged work on HADOOP-17833:
---
Author: ASF GitHub Bot
Created on: 19/Aug/21 14:00
Start Date: 19/Aug/21 14:00
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3289:
URL: https://github.com/apache/hadoop/pull/3289#issuecomment-896250187
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 3s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 18 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 1m 45s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/1/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 4m 4s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 20s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| -1 :x: | shadedclient | 7m 12s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 21s |
[/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt)
| hadoop-tools/hadoop-aws: The patch generated 12 new + 25 unchanged - 1
fixed = 37 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 24s |
[/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/1/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-aws in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | spotbugs | 1m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 0m 17s |
[/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/1/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 38m 33s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3289 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux bd273600a69a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3289: HADOOP-17833. Improve Magic Committer performance
hadoop-yetus removed a comment on pull request #3289:
URL: https://github.com/apache/hadoop/pull/3289#issuecomment-896250187
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 3s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 18 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 1m 45s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/1/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 4m 4s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 20s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| -1 :x: | shadedclient | 7m 12s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 21s |
[/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt)
| hadoop-tools/hadoop-aws: The patch generated 12 new + 25 unchanged - 1
fixed = 37 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javadoc | 0m 24s |
[/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/1/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-aws in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | spotbugs | 1m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 0m 17s |
[/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/1/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 38m 33s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3289 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux bd273600a69a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 208929ef08592d69309263588eb7fd01675c9229 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3289/1/testReport/ |
[GitHub] [hadoop] tomicooler commented on pull request #3314: YARN-10891. Extend QueueInfo with max-parallel-apps in CS.
tomicooler commented on pull request #3314: URL: https://github.com/apache/hadoop/pull/3314#issuecomment-901938204 Probably I should have just extended the CapacitySchedulerQueueInfo for the Cluster Scheduler API (REST). https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API marking as WIP -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17836) ABFS connection reset reporting excessively noisy
[ https://issues.apache.org/jira/browse/HADOOP-17836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17836. - Fix Version/s: 3.4.0 Resolution: Fixed > ABFS connection reset reporting excessively noisy > - > > Key: HADOOP-17836 > URL: https://issues.apache.org/jira/browse/HADOOP-17836 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 > Environment: long haul FTTH link to Azure cardif ~50 miles away; > laptop with wifi >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Large 10GB download from abfs failing after 50 minutes, connection reset > Assumptions > * Azure storage/routers etc get bored of long-lived HTTP connections > * ABFS client doesn't recover from socket exceptions -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16852) ABFS: Send error back to client for Read Ahead request failure
[ https://issues.apache.org/jira/browse/HADOOP-16852?focusedWorklogId=639819=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-639819 ] ASF GitHub Bot logged work on HADOOP-16852: --- Author: ASF GitHub Bot Created on: 19/Aug/21 12:41 Start Date: 19/Aug/21 12:41 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #1898: URL: https://github.com/apache/hadoop/pull/1898#issuecomment-619725991 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 27m 40s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 23m 40s | trunk passed | | +1 :green_heart: | compile | 0m 30s | trunk passed | | +1 :green_heart: | checkstyle | 0m 24s | trunk passed | | +1 :green_heart: | mvnsite | 0m 33s | trunk passed | | +1 :green_heart: | shadedclient | 16m 44s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 27s | trunk passed | | +0 :ok: | spotbugs | 0m 59s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 57s | trunk passed | | -0 :warning: | patch | 1m 15s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 0m 25s | hadoop-azure in the patch failed. | | -1 :x: | compile | 0m 25s | hadoop-azure in the patch failed. | | -1 :x: | javac | 0m 25s | hadoop-azure in the patch failed. | | -0 :warning: | checkstyle | 0m 15s | hadoop-tools/hadoop-azure: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) | | -1 :x: | mvnsite | 0m 27s | hadoop-azure in the patch failed. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 44s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 21s | the patch passed | | -1 :x: | findbugs | 0m 28s | hadoop-azure in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 29s | hadoop-azure in the patch failed. | | +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. | | | | 91m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/9/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1898 | | JIRA Issue | HADOOP-16852 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 7aea48b80f67 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 85516a8af7e | | Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/9/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/9/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/9/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/9/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt | | mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/9/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/9/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/9/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/9/testReport/ | | Max. process+thread count | 307 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output |
[jira] [Work logged] (HADOOP-16852) ABFS: Send error back to client for Read Ahead request failure
[ https://issues.apache.org/jira/browse/HADOOP-16852?focusedWorklogId=639820=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-639820 ] ASF GitHub Bot logged work on HADOOP-16852: --- Author: ASF GitHub Bot Created on: 19/Aug/21 12:41 Start Date: 19/Aug/21 12:41 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #1898: URL: https://github.com/apache/hadoop/pull/1898#issuecomment-630880793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 639820) Time Spent: 1h 10m (was: 1h) > ABFS: Send error back to client for Read Ahead request failure > -- > > Key: HADOOP-16852 > URL: https://issues.apache.org/jira/browse/HADOOP-16852 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Issue seen by a customer: > The failed requests we were seeing in the AbfsClient logging actually never > made it out over the wire. We have found that there’s an issue with ADLS > passthrough and the 8 read ahead threads that ADLSv2 spawns in > ReadBufferManager.java. We depend on thread local storage in order to get the > right JWT token and those threads do not have the right information in their > thread local storage. Thus, when they pick up a task from the read ahead > queue they fail by throwing an AzureCredentialNotFoundException exception in > AbfsRestOperation.executeHttpOperation() where it calls > client.getAccessToken(). This exception is silently swallowed by the read > ahead threads in ReadBufferWorker.run(). As a result, every read ahead > attempt results in a failed executeHttpOperation(), but still calls > AbfsClientThrottlingIntercept.updateMetrics() and contributes to throttling > (despite not making it out over the wire). After the read aheads fail, the > main task thread performs the read with the right thread local storage > information and succeeds, but first sleeps for up to 10 seconds due to the > throttling. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #1898: HADOOP-16852: Report read-ahead error back
hadoop-yetus removed a comment on pull request #1898: URL: https://github.com/apache/hadoop/pull/1898#issuecomment-630880793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #1898: HADOOP-16852: Report read-ahead error back
hadoop-yetus removed a comment on pull request #1898: URL: https://github.com/apache/hadoop/pull/1898#issuecomment-619725991 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 27m 40s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 23m 40s | trunk passed | | +1 :green_heart: | compile | 0m 30s | trunk passed | | +1 :green_heart: | checkstyle | 0m 24s | trunk passed | | +1 :green_heart: | mvnsite | 0m 33s | trunk passed | | +1 :green_heart: | shadedclient | 16m 44s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 27s | trunk passed | | +0 :ok: | spotbugs | 0m 59s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 57s | trunk passed | | -0 :warning: | patch | 1m 15s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 0m 25s | hadoop-azure in the patch failed. | | -1 :x: | compile | 0m 25s | hadoop-azure in the patch failed. | | -1 :x: | javac | 0m 25s | hadoop-azure in the patch failed. | | -0 :warning: | checkstyle | 0m 15s | hadoop-tools/hadoop-azure: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) | | -1 :x: | mvnsite | 0m 27s | hadoop-azure in the patch failed. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 44s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 21s | the patch passed | | -1 :x: | findbugs | 0m 28s | hadoop-azure in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 29s | hadoop-azure in the patch failed. | | +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. | | | | 91m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/9/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1898 | | JIRA Issue | HADOOP-16852 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 7aea48b80f67 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 85516a8af7e | | Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/9/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/9/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/9/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/9/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt | | mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/9/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/9/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/9/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/9/testReport/ | | Max. process+thread count | 307 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/9/console | | versions | git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #1898: HADOOP-16852: Report read-ahead error back
hadoop-yetus removed a comment on pull request #1898: URL: https://github.com/apache/hadoop/pull/1898#issuecomment-610182688 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 22m 15s | trunk passed | | +1 :green_heart: | compile | 0m 28s | trunk passed | | +1 :green_heart: | checkstyle | 0m 21s | trunk passed | | +1 :green_heart: | mvnsite | 0m 29s | trunk passed | | +1 :green_heart: | shadedclient | 16m 35s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 22s | trunk passed | | +0 :ok: | spotbugs | 0m 57s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 53s | trunk passed | | -0 :warning: | patch | 1m 13s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 27s | the patch passed | | +1 :green_heart: | compile | 0m 22s | the patch passed | | +1 :green_heart: | javac | 0m 22s | the patch passed | | +1 :green_heart: | checkstyle | 0m 16s | the patch passed | | +1 :green_heart: | mvnsite | 0m 25s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 36s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 19s | the patch passed | | +1 :green_heart: | findbugs | 0m 53s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 24s | hadoop-azure in the patch passed. | | +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. | | | | 64m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/8/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1898 | | JIRA Issue | HADOOP-16852 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 62d9b5860a70 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 0b855b9 | | Default Java | 1.8.0_242 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/8/testReport/ | | Max. process+thread count | 307 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/8/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16852) ABFS: Send error back to client for Read Ahead request failure
[ https://issues.apache.org/jira/browse/HADOOP-16852?focusedWorklogId=639818=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-639818 ] ASF GitHub Bot logged work on HADOOP-16852: --- Author: ASF GitHub Bot Created on: 19/Aug/21 12:40 Start Date: 19/Aug/21 12:40 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #1898: URL: https://github.com/apache/hadoop/pull/1898#issuecomment-610182688 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 22m 15s | trunk passed | | +1 :green_heart: | compile | 0m 28s | trunk passed | | +1 :green_heart: | checkstyle | 0m 21s | trunk passed | | +1 :green_heart: | mvnsite | 0m 29s | trunk passed | | +1 :green_heart: | shadedclient | 16m 35s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 22s | trunk passed | | +0 :ok: | spotbugs | 0m 57s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 53s | trunk passed | | -0 :warning: | patch | 1m 13s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 27s | the patch passed | | +1 :green_heart: | compile | 0m 22s | the patch passed | | +1 :green_heart: | javac | 0m 22s | the patch passed | | +1 :green_heart: | checkstyle | 0m 16s | the patch passed | | +1 :green_heart: | mvnsite | 0m 25s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 36s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 19s | the patch passed | | +1 :green_heart: | findbugs | 0m 53s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 24s | hadoop-azure in the patch passed. | | +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. | | | | 64m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/8/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1898 | | JIRA Issue | HADOOP-16852 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 62d9b5860a70 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 0b855b9 | | Default Java | 1.8.0_242 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/8/testReport/ | | Max. process+thread count | 307 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1898/8/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 639818) Time Spent: 50m (was: 40m) > ABFS: Send error back to client for Read Ahead request failure > -- > > Key: HADOOP-16852 > URL: https://issues.apache.org/jira/browse/HADOOP-16852 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sneha Vijayarajan >Assignee: Sneha
[jira] [Work logged] (HADOOP-16852) ABFS: Send error back to client for Read Ahead request failure
[ https://issues.apache.org/jira/browse/HADOOP-16852?focusedWorklogId=639817=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-639817 ] ASF GitHub Bot logged work on HADOOP-16852: --- Author: ASF GitHub Bot Created on: 19/Aug/21 12:40 Start Date: 19/Aug/21 12:40 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #1898: URL: https://github.com/apache/hadoop/pull/1898#issuecomment-604858193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 639817) Time Spent: 40m (was: 0.5h) > ABFS: Send error back to client for Read Ahead request failure > -- > > Key: HADOOP-16852 > URL: https://issues.apache.org/jira/browse/HADOOP-16852 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 40m > Remaining Estimate: 0h > > Issue seen by a customer: > The failed requests we were seeing in the AbfsClient logging actually never > made it out over the wire. We have found that there’s an issue with ADLS > passthrough and the 8 read ahead threads that ADLSv2 spawns in > ReadBufferManager.java. We depend on thread local storage in order to get the > right JWT token and those threads do not have the right information in their > thread local storage. Thus, when they pick up a task from the read ahead > queue they fail by throwing an AzureCredentialNotFoundException exception in > AbfsRestOperation.executeHttpOperation() where it calls > client.getAccessToken(). This exception is silently swallowed by the read > ahead threads in ReadBufferWorker.run(). As a result, every read ahead > attempt results in a failed executeHttpOperation(), but still calls > AbfsClientThrottlingIntercept.updateMetrics() and contributes to throttling > (despite not making it out over the wire). After the read aheads fail, the > main task thread performs the read with the right thread local storage > information and succeeds, but first sleeps for up to 10 seconds due to the > throttling. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #1898: HADOOP-16852: Report read-ahead error back
hadoop-yetus removed a comment on pull request #1898: URL: https://github.com/apache/hadoop/pull/1898#issuecomment-604858193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3313: HDFS-16179. Update loglevel for BlockManager#chooseExcessRedundancySt…
hadoop-yetus commented on pull request #3313:
URL: https://github.com/apache/hadoop/pull/3313#issuecomment-901835881
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 48s | | trunk passed |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 7s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 21s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 21s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 46s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 354m 15s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3313/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 451m 10s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3313/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3313 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 0cb16d9f8243 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13
01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a121470e5b175de09f0697d510e9b26612b2f6e1 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3313/1/testReport/ |
| Max. process+thread count | 2110 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3313/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus
[GitHub] [hadoop] hadoop-yetus commented on pull request #3314: YARN-10891. Extend QueueInfo with max-parallel-apps in CS.
hadoop-yetus commented on pull request #3314:
URL: https://github.com/apache/hadoop/pull/3314#issuecomment-901820154
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 40s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 16s | | trunk passed |
| +1 :green_heart: | compile | 10m 21s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 8m 38s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 48s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 59s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 50s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 31s | | the patch passed |
| +1 :green_heart: | compile | 9m 27s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | cc | 9m 27s |
[/results-compile-cc-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3314/1/artifact/out/results-compile-cc-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 5 new + 177 unchanged
- 5 fixed = 182 total (was 182) |
| +1 :green_heart: | javac | 9m 27s | | the patch passed |
| +1 :green_heart: | compile | 8m 31s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | cc | 8m 31s |
[/results-compile-cc-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3314/1/artifact/out/results-compile-cc-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
|
hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new
+ 181 unchanged - 1 fixed = 182 total (was 182) |
| +1 :green_heart: | javac | 8m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 36s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 45s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 35s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 7m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 59s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 4m 40s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 99m 41s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 27m 41s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 282m 49s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3314/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3314 |
|
[GitHub] [hadoop] Neilxzn opened a new pull request #3315: HDFS-16180.FsVolumeImpl.nextBlock should consider that the block meta file has been deleted
Neilxzn opened a new pull request #3315: URL: https://github.com/apache/hadoop/pull/3315 ### Description of PR FsVolumeImpl.nextBlock should consider that the block meta file has been deleted https://issues.apache.org/jira/browse/HDFS-16180 ### How was this patch tested? no new test. ### For code changes: FsVolumeImpl.nextBlock hanlde FileNotFoundException -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
ayushtkn commented on pull request #3302: URL: https://github.com/apache/hadoop/pull/3302#issuecomment-901709391 hmm, Not sure why the Test Result Link is also not working for most PRs: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/6/testReport/ @jojochuang are you aware of some thread/jira disabling it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
jianghuazhu commented on pull request #3302: URL: https://github.com/apache/hadoop/pull/3302#issuecomment-901702371 Some exceptions occurred here, related to UT: hadoop.cli.TestCLI But it has nothing to do with the code I submitted. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3302: HDFS-16173.Improve CopyCommands#Put#executor queue configurability.
hadoop-yetus commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-901643513
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 53m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 3s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 3s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 6s | | trunk passed |
| +1 :green_heart: | compile | 30m 31s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 23s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 1s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 34s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 23m 5s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 5s | | the patch passed |
| +1 :green_heart: | compile | 20m 13s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 20m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 59s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 49s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 38s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 50s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 20m 29s |
[/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/6/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not
generate ASF License warnings. |
| | | 263m 57s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.cli.TestCLI |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3302 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 673e639c01ad 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7abf2a2b408e129c0c0c577bc6cf7aa18e18cf54 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/6/testReport/ |
| Max. process+thread count | 2282 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
[GitHub] [hadoop] tomicooler opened a new pull request #3314: YARN-10891. Extend QueueInfo with max-parallel-apps in CS.
tomicooler opened a new pull request #3314: URL: https://github.com/apache/hadoop/pull/3314 ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
