[jira] [Commented] (HADOOP-18400) Fix file split duplicating records from a succeeding split when reading BZip2 text files
[ https://issues.apache.org/jira/browse/HADOOP-18400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606422#comment-17606422 ] ASF GitHub Bot commented on HADOOP-18400: - ashutoshcipher commented on PR #4732: URL: https://github.com/apache/hadoop/pull/4732#issuecomment-1250569098 Thanks @aajisaka for final review/merge and @saswata-dutta for your additional review. > Fix file split duplicating records from a succeeding split when reading > BZip2 text files > -- > > Key: HADOOP-18400 > URL: https://issues.apache.org/jira/browse/HADOOP-18400 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.3.3, 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > Fix data correctness issue with TextInputFormat that can occur when reading > BZip2 compressed text files. When a file split's range does not include the > start position of a BZip2 block, then it is expected to contain no records > (i.e. the split is empty). However, if it so happens that the end of this > split (exclusive) is at the start of a BZip2 block, then LineRecordReader > ends up returning all the records for that BZip2 block. This ends up > duplicating records read by a job because the next split would also end up > returning all the records for the same block (since its range would include > the start of that block). > This bug does not get triggered when the file split's range does include the > start of at least one block and ends just before the start of another block. > The reason for this has to do with when BZip2CompressionInputStream updates > its position when using the BYBLOCK READMODE. Using this read mode, the > stream's position while reading only gets updated when reading the first byte > past an end of a block marker. The bug is that if the stream, when > initialized, was adjusted to be at the end of one block, then we don't update > the position after we read the first byte of the next block. Rather, we keep > the position to be equal to the next block marker we've initialized to. If > the exclusive end position of the split is equal to stream's position, > LineRecordReader will continue to read lines until the position is updated > (an an additional record in the next block is read if needed). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ashutoshcipher commented on pull request #4732: HADOOP-18400. Fix file split duplicating records from a succeeding split when reading BZip2 text files
ashutoshcipher commented on PR #4732: URL: https://github.com/apache/hadoop/pull/4732#issuecomment-1250569098 Thanks @aajisaka for final review/merge and @saswata-dutta for your additional review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18400) Fix file split duplicating records from a succeeding split when reading BZip2 text files
[ https://issues.apache.org/jira/browse/HADOOP-18400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-18400: --- Fix Version/s: 3.4.0 3.3.9 Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk and branch-3.3. > Fix file split duplicating records from a succeeding split when reading > BZip2 text files > -- > > Key: HADOOP-18400 > URL: https://issues.apache.org/jira/browse/HADOOP-18400 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.3.3, 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > Fix data correctness issue with TextInputFormat that can occur when reading > BZip2 compressed text files. When a file split's range does not include the > start position of a BZip2 block, then it is expected to contain no records > (i.e. the split is empty). However, if it so happens that the end of this > split (exclusive) is at the start of a BZip2 block, then LineRecordReader > ends up returning all the records for that BZip2 block. This ends up > duplicating records read by a job because the next split would also end up > returning all the records for the same block (since its range would include > the start of that block). > This bug does not get triggered when the file split's range does include the > start of at least one block and ends just before the start of another block. > The reason for this has to do with when BZip2CompressionInputStream updates > its position when using the BYBLOCK READMODE. Using this read mode, the > stream's position while reading only gets updated when reading the first byte > past an end of a block marker. The bug is that if the stream, when > initialized, was adjusted to be at the end of one block, then we don't update > the position after we read the first byte of the next block. Rather, we keep > the position to be equal to the next block marker we've initialized to. If > the exclusive end position of the split is equal to stream's position, > LineRecordReader will continue to read lines until the position is updated > (an an additional record in the next block is read if needed). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18400) Fix file split duplicating records from a succeeding split when reading BZip2 text files
[ https://issues.apache.org/jira/browse/HADOOP-18400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606417#comment-17606417 ] ASF GitHub Bot commented on HADOOP-18400: - aajisaka merged PR #4732: URL: https://github.com/apache/hadoop/pull/4732 > Fix file split duplicating records from a succeeding split when reading > BZip2 text files > -- > > Key: HADOOP-18400 > URL: https://issues.apache.org/jira/browse/HADOOP-18400 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.3.3, 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Critical > Labels: pull-request-available > > Fix data correctness issue with TextInputFormat that can occur when reading > BZip2 compressed text files. When a file split's range does not include the > start position of a BZip2 block, then it is expected to contain no records > (i.e. the split is empty). However, if it so happens that the end of this > split (exclusive) is at the start of a BZip2 block, then LineRecordReader > ends up returning all the records for that BZip2 block. This ends up > duplicating records read by a job because the next split would also end up > returning all the records for the same block (since its range would include > the start of that block). > This bug does not get triggered when the file split's range does include the > start of at least one block and ends just before the start of another block. > The reason for this has to do with when BZip2CompressionInputStream updates > its position when using the BYBLOCK READMODE. Using this read mode, the > stream's position while reading only gets updated when reading the first byte > past an end of a block marker. The bug is that if the stream, when > initialized, was adjusted to be at the end of one block, then we don't update > the position after we read the first byte of the next block. Rather, we keep > the position to be equal to the next block marker we've initialized to. If > the exclusive end position of the split is equal to stream's position, > LineRecordReader will continue to read lines until the position is updated > (an an additional record in the next block is read if needed). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka merged pull request #4732: HADOOP-18400. Fix file split duplicating records from a succeeding split when reading BZip2 text files
aajisaka merged PR #4732: URL: https://github.com/apache/hadoop/pull/4732 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18457) ABFS: Support for account level throttling
Anmol Asrani created HADOOP-18457: - Summary: ABFS: Support for account level throttling Key: HADOOP-18457 URL: https://issues.apache.org/jira/browse/HADOOP-18457 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.3.4 Reporter: Anmol Asrani Fix For: 3.4.0 To add support for throttling at account level -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4874: Upgrade snakeyaml to 1.32
hadoop-yetus commented on PR #4874:
URL: https://github.com/apache/hadoop/pull/4874#issuecomment-1250485831
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ branch-3.2 Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 29s | | branch-3.2 passed |
| +1 :green_heart: | compile | 0m 22s | | branch-3.2 passed |
| +1 :green_heart: | mvnsite | 0m 28s | | branch-3.2 passed |
| +1 :green_heart: | javadoc | 0m 34s | | branch-3.2 passed |
| +1 :green_heart: | shadedclient | 49m 41s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 0m 13s | | the patch passed |
| +1 :green_heart: | javac | 0m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 14s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 18s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 81m 12s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4874/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4874 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | Linux a5326421c69f 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.2 / f0df9ae21e1ec1a4fab1d83d66d18f658eddfa6f |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~18.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4874/2/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4874/2/console |
| versions | git=2.17.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18456) NullPointerException in ObjectListingIterator's constructor
[
https://issues.apache.org/jira/browse/HADOOP-18456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606393#comment-17606393
]
Quanlong Huang commented on HADOOP-18456:
-
The code of the aggregator is recently added by HADOOP-17461. Do you have time
to take a look? [~mehakmeet]
CC [~ste...@apache.org]
> NullPointerException in ObjectListingIterator's constructor
> ---
>
> Key: HADOOP-18456
> URL: https://issues.apache.org/jira/browse/HADOOP-18456
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Reporter: Quanlong Huang
>Priority: Blocker
>
> We saw NullPointerExceptions in Impala's S3 tests: IMPALA-11592. It's thrown
> from the hadoop jar:
> {noformat}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.(Listing.java:621)
> at
> org.apache.hadoop.fs.s3a.Listing.createObjectListingIterator(Listing.java:163)
> at
> org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:144)
> at
> org.apache.hadoop.fs.s3a.Listing.getListFilesAssumingDir(Listing.java:212)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerListFiles(S3AFileSystem.java:4790)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listFiles$37(S3AFileSystem.java:4732)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2363)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2382)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.listFiles(S3AFileSystem.java:4731)
> at
> org.apache.impala.common.FileSystemUtil.listFiles(FileSystemUtil.java:754)
> ... {noformat}
> We are using a private build of the hadoop jar. Version: CDP
> 3.1.1.7.2.16.0-164
> Code snipper of where the NPE throws:
> {code:java}
> 604 @Retries.RetryRaw
> 605 ObjectListingIterator(
> 606 Path listPath,
> 607 S3ListRequest request,
> 608 AuditSpan span) throws IOException {
> 609 this.listPath = listPath;
> 610 this.maxKeys = listingOperationCallbacks.getMaxKeys();
> 611 this.request = request;
> 612 this.objectsPrev = null;
> 613 this.iostats = iostatisticsStore()
> 614 .withDurationTracking(OBJECT_LIST_REQUEST)
> 615 .withDurationTracking(OBJECT_CONTINUE_LIST_REQUEST)
> 616 .build();
> 617 this.span = span;
> 618 this.s3ListResultFuture = listingOperationCallbacks
> 619 .listObjectsAsync(request, iostats, span);
> 620 this.aggregator =
> IOStatisticsContext.getCurrentIOStatisticsContext()
> 621 .getAggregator(); // < thrown here
> 622 }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18456) NullPointerException in ObjectListingIterator's constructor
Quanlong Huang created HADOOP-18456:
---
Summary: NullPointerException in ObjectListingIterator's
constructor
Key: HADOOP-18456
URL: https://issues.apache.org/jira/browse/HADOOP-18456
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Reporter: Quanlong Huang
We saw NullPointerExceptions in Impala's S3 tests: IMPALA-11592. It's thrown
from the hadoop jar:
{noformat}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.(Listing.java:621)
at
org.apache.hadoop.fs.s3a.Listing.createObjectListingIterator(Listing.java:163)
at
org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:144)
at
org.apache.hadoop.fs.s3a.Listing.getListFilesAssumingDir(Listing.java:212)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerListFiles(S3AFileSystem.java:4790)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listFiles$37(S3AFileSystem.java:4732)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2363)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2382)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listFiles(S3AFileSystem.java:4731)
at
org.apache.impala.common.FileSystemUtil.listFiles(FileSystemUtil.java:754)
... {noformat}
We are using a private build of the hadoop jar. Version: CDP 3.1.1.7.2.16.0-164
Code snipper of where the NPE throws:
{code:java}
604 @Retries.RetryRaw
605 ObjectListingIterator(
606 Path listPath,
607 S3ListRequest request,
608 AuditSpan span) throws IOException {
609 this.listPath = listPath;
610 this.maxKeys = listingOperationCallbacks.getMaxKeys();
611 this.request = request;
612 this.objectsPrev = null;
613 this.iostats = iostatisticsStore()
614 .withDurationTracking(OBJECT_LIST_REQUEST)
615 .withDurationTracking(OBJECT_CONTINUE_LIST_REQUEST)
616 .build();
617 this.span = span;
618 this.s3ListResultFuture = listingOperationCallbacks
619 .listObjectsAsync(request, iostats, span);
620 this.aggregator = IOStatisticsContext.getCurrentIOStatisticsContext()
621 .getAggregator(); // < thrown here
622 }
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4905: YARN-11307. Fix Yarn Router Broken Link.
hadoop-yetus commented on PR #4905:
URL: https://github.com/apache/hadoop/pull/4905#issuecomment-1250458091
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 12s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 20s | | trunk passed |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 59s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4905/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router:
The patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 0m 56s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 40s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 105m 44s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4905/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4905 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 8822da1578d9 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 19b4f7f62e97a76b89da32dc719a53e4977d3bff |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4905/1/testReport/ |
| Max. process+thread count | 708 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output |
https://ci-hadoop.apache.org/jo
[jira] [Commented] (HADOOP-18443) Upgrade snakeyaml to 1.32
[ https://issues.apache.org/jira/browse/HADOOP-18443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606382#comment-17606382 ] ASF GitHub Bot commented on HADOOP-18443: - ashutoshcipher opened a new pull request, #4906: URL: https://github.com/apache/hadoop/pull/4906 ### Description of PR Upgrade snakeyaml to 1.32 to mitigate [CVE-2022-38752](https://github.com/advisories/GHSA-9w3m-gqgf-c4p9) ### For code changes: - [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Upgrade snakeyaml to 1.32 > - > > Key: HADOOP-18443 > URL: https://issues.apache.org/jira/browse/HADOOP-18443 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.3, 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > > Upgrade snakeyaml to 1.32 to mitigate CVE-2022-25857 and > [CVE-2022-38752|https://github.com/advisories/GHSA-9w3m-gqgf-c4p9] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ashutoshcipher opened a new pull request, #4906: HADOOP-18443. Upgrade snakeyaml to 1.32
ashutoshcipher opened a new pull request, #4906: URL: https://github.com/apache/hadoop/pull/4906 ### Description of PR Upgrade snakeyaml to 1.32 to mitigate [CVE-2022-38752](https://github.com/advisories/GHSA-9w3m-gqgf-c4p9) ### For code changes: - [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18443) Upgrade snakeyaml to 1.32
[ https://issues.apache.org/jira/browse/HADOOP-18443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta updated HADOOP-18443: Summary: Upgrade snakeyaml to 1.32 (was: Upgrade snakeyaml to 1.32 to mitigate CVE-2022-25857 and and CVE-2022-38752) > Upgrade snakeyaml to 1.32 > - > > Key: HADOOP-18443 > URL: https://issues.apache.org/jira/browse/HADOOP-18443 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.3, 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > > Upgrade snakeyaml to 1.32 to mitigate CVE-2022-25857 and > [CVE-2022-38752|https://github.com/advisories/GHSA-9w3m-gqgf-c4p9] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18443) Upgrade snakeyaml to 1.32 to mitigate CVE-2022-25857 and and CVE-2022-38752
[ https://issues.apache.org/jira/browse/HADOOP-18443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta updated HADOOP-18443: Summary: Upgrade snakeyaml to 1.32 to mitigate CVE-2022-25857 and and CVE-2022-38752 (was: Upgrade snakeyaml to 1.31 to mitigate CVE-2022-25857 and and CVE-2022-38752) > Upgrade snakeyaml to 1.32 to mitigate CVE-2022-25857 and and CVE-2022-38752 > --- > > Key: HADOOP-18443 > URL: https://issues.apache.org/jira/browse/HADOOP-18443 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.3, 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > > Upgrade snakeyaml to 1.32 to mitigate CVE-2022-25857 and > [CVE-2022-38752|https://github.com/advisories/GHSA-9w3m-gqgf-c4p9] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18443) Upgrade snakeyaml to 1.31 to mitigate CVE-2022-25857
[ https://issues.apache.org/jira/browse/HADOOP-18443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta updated HADOOP-18443: Description: Upgrade snakeyaml to 1.32 to mitigate CVE-2022-25857 and [CVE-2022-38752|https://github.com/advisories/GHSA-9w3m-gqgf-c4p9] (was: Upgrade snakeyaml to 1.32 to mitigate CVE-2022-25857 and [CVE-2022-38752|https://github.com/advisories/GHSA-9w3m-gqgf-c4p9] Details - [https://nvd.nist.gov/vuln/detail/CVE-2022-25857]) > Upgrade snakeyaml to 1.31 to mitigate CVE-2022-25857 > > > Key: HADOOP-18443 > URL: https://issues.apache.org/jira/browse/HADOOP-18443 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.3, 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > > Upgrade snakeyaml to 1.32 to mitigate CVE-2022-25857 and > [CVE-2022-38752|https://github.com/advisories/GHSA-9w3m-gqgf-c4p9] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18443) Upgrade snakeyaml to 1.31 to mitigate CVE-2022-25857 and and CVE-2022-38752
[ https://issues.apache.org/jira/browse/HADOOP-18443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta updated HADOOP-18443: Summary: Upgrade snakeyaml to 1.31 to mitigate CVE-2022-25857 and and CVE-2022-38752 (was: Upgrade snakeyaml to 1.31 to mitigate CVE-2022-25857) > Upgrade snakeyaml to 1.31 to mitigate CVE-2022-25857 and and CVE-2022-38752 > --- > > Key: HADOOP-18443 > URL: https://issues.apache.org/jira/browse/HADOOP-18443 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.3, 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > > Upgrade snakeyaml to 1.32 to mitigate CVE-2022-25857 and > [CVE-2022-38752|https://github.com/advisories/GHSA-9w3m-gqgf-c4p9] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18443) Upgrade snakeyaml to 1.31 to mitigate CVE-2022-25857
[ https://issues.apache.org/jira/browse/HADOOP-18443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta updated HADOOP-18443: Description: Upgrade snakeyaml to 1.32 to mitigate CVE-2022-25857 and [CVE-2022-38752|https://github.com/advisories/GHSA-9w3m-gqgf-c4p9] Details - [https://nvd.nist.gov/vuln/detail/CVE-2022-25857] was: Upgrade snakeyaml to 1.31 to mitigate CVE-2022-25857 Details - https://nvd.nist.gov/vuln/detail/CVE-2022-25857 > Upgrade snakeyaml to 1.31 to mitigate CVE-2022-25857 > > > Key: HADOOP-18443 > URL: https://issues.apache.org/jira/browse/HADOOP-18443 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.3, 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > > Upgrade snakeyaml to 1.32 to mitigate CVE-2022-25857 and > [CVE-2022-38752|https://github.com/advisories/GHSA-9w3m-gqgf-c4p9] > > Details - [https://nvd.nist.gov/vuln/detail/CVE-2022-25857] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18443) Upgrade snakeyaml to 1.31 to mitigate CVE-2022-25857
[ https://issues.apache.org/jira/browse/HADOOP-18443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606380#comment-17606380 ] Ashutosh Gupta commented on HADOOP-18443: - Thanks [~pj.fanning] . Updated > Upgrade snakeyaml to 1.31 to mitigate CVE-2022-25857 > > > Key: HADOOP-18443 > URL: https://issues.apache.org/jira/browse/HADOOP-18443 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.3, 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > > Upgrade snakeyaml to 1.31 to mitigate CVE-2022-25857 > > Details - https://nvd.nist.gov/vuln/detail/CVE-2022-25857 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4904: YARN-11238. Optimizing FederationClientInterceptor Call with Parallelism.
hadoop-yetus commented on PR #4904:
URL: https://github.com/apache/hadoop/pull/4904#issuecomment-1250418141
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 20s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 51s | | trunk passed |
| +1 :green_heart: | compile | 4m 1s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 3m 23s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 34s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 3s | | the patch passed |
| +1 :green_heart: | compile | 3m 51s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 3m 51s | | the patch passed |
| +1 :green_heart: | compile | 3m 19s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 7s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| -1 :x: | javadoc | 0m 35s |
[/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4904/2/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 1 new + 48
unchanged - 0 fixed = 49 total (was 48) |
| -1 :x: | javadoc | 0m 32s |
[/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4904/2/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 generated 1 new
+ 48 unchanged - 0 fixed = 49 total (was 48) |
| +1 :green_heart: | spotbugs | 2m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 6s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 3m 46s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 123m 58s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4904/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pul
[GitHub] [hadoop] slfan1989 opened a new pull request, #4905: YARN-11307. Fix Yarn Router Broken Link.
slfan1989 opened a new pull request, #4905: URL: https://github.com/apache/hadoop/pull/4905 JIRA: YARN-11307. Fix Yarn Router Broken Link. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4903: HDFS-16774.Improve async delete replica on datanode
hadoop-yetus commented on PR #4903:
URL: https://github.com/apache/hadoop/pull/4903#issuecomment-1250362446
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 3s | | trunk passed |
| +1 :green_heart: | compile | 1m 44s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 35s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 11s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 38s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 27s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 32s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 32s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 56s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4903/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 89 unchanged -
2 fixed = 91 total (was 91) |
| +1 :green_heart: | mvnsite | 1m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 37s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 3s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 244m 4s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 10s | | The patch does not
generate ASF License warnings. |
| | | 355m 34s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4903/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4903 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b06ccc128ba9 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 22317c8a051173edbfd74edf92de27e521ce7034 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4903/1/testReport/ |
| Max. process+thread count | 3390 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4903/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This
[GitHub] [hadoop] hadoop-yetus commented on pull request #4901: HDFS-16776 Erasure Coding: The length of targets should be checked when DN gets a reconstruction task
hadoop-yetus commented on PR #4901:
URL: https://github.com/apache/hadoop/pull/4901#issuecomment-1250344314
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 17s | | trunk passed |
| +1 :green_heart: | compile | 1m 42s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 16s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 1s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 1s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4901/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 0 unchanged -
0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 1m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 54s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 238m 55s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 7s | | The patch does not
generate ASF License warnings. |
| | | 348m 1s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4901/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4901 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux f257db2e44d1 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5b4abf4b7a804fdca0db78107d73ebe65cea719b |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4901/2/testReport/ |
| Max. process+thread count | 3649 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4901/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered
[GitHub] [hadoop] hadoop-yetus commented on pull request #4902: HDFS-16775.Improve BlockPlacementPolicyRackFaultTolerant's chooseOnce
hadoop-yetus commented on PR #4902:
URL: https://github.com/apache/hadoop/pull/4902#issuecomment-1250335997
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 13s | | trunk passed |
| +1 :green_heart: | compile | 1m 40s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 35s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 2s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| -1 :x: | spotbugs | 3m 25s |
[/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4902/1/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html)
| hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) |
| +1 :green_heart: | shadedclient | 22m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 239m 53s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 10s | | The patch does not
generate ASF License warnings. |
| | | 348m 48s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | org.apache.hadoop.net.Node is incompatible with expected argument type
DatanodeStorageInfo in
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant.chooseOnce(int,
Node, Set, long, int, List, boolean, EnumMap) At
BlockPlacementPolicyRackFaultTolerant.java:argument type DatanodeStorageInfo in
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant.chooseOnce(int,
Node, Set, long, int, List, boolean, EnumMap) At
BlockPlacementPolicyRackFaultTolerant.java:[line 226] |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4902/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4902 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 073d50524576 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9ca1469a3d1bc31c6e52d3b219f7af2aeec9459c |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1
[GitHub] [hadoop] hadoop-yetus commented on pull request #4904: YARN-11238. Optimizing FederationClientInterceptor Call with Parallelism.
hadoop-yetus commented on PR #4904:
URL: https://github.com/apache/hadoop/pull/4904#issuecomment-1250320064
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 6s | | trunk passed |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 0m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 39s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 97m 17s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4904/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4904 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux f351e7c7a779 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2dcf58f45d881d11faf96dcb5e4988e543cb744d |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4904/1/testReport/ |
| Max. process+thread count | 747 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4904/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Ser
[GitHub] [hadoop] slfan1989 opened a new pull request, #4904: YARN-11238. Optimizing FederationClientInterceptor Call with Parallelism.
slfan1989 opened a new pull request, #4904: URL: https://github.com/apache/hadoop/pull/4904 JIRA: YARN-11238. Optimizing FederationClientInterceptor Call with Parallelism. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4903: HDFS-16774.Improve async delete replica on datanode
slfan1989 commented on code in PR #4903:
URL: https://github.com/apache/hadoop/pull/4903#discussion_r973714519
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java:
##
@@ -359,6 +371,89 @@ public void run() {
IOUtils.cleanupWithLogger(null, this.volumeRef);
}
}
+
+private boolean removeReplicaFromMem() {
+ try (AutoCloseableLock lock =
fsdatasetImpl.acquireDatasetLockManager().writeLock(
+ DataNodeLockManager.LockLevel.BLOCK_POOl, block.getBlockPoolId())) {
+final ReplicaInfo info = fsdatasetImpl.volumeMap
+.get(block.getBlockPoolId(), block.getLocalBlock());
+if (info == null) {
+ ReplicaInfo infoByBlockId =
+ fsdatasetImpl.volumeMap.get(block.getBlockPoolId(),
+ block.getLocalBlock().getBlockId());
+ if (infoByBlockId == null) {
+// It is okay if the block is not found -- it
+// may be deleted earlier.
+LOG.info("Failed to delete replica " + block.getLocalBlock()
++ ": ReplicaInfo not found in removeReplicaFromMem.");
+ } else {
+LOG.error("Failed to delete replica " + block.getLocalBlock()
++ ": GenerationStamp not matched, existing replica is "
++ Block.toString(infoByBlockId) + " in removeReplicaFromMem.");
+ }
+ return false;
+}
+
+FsVolumeImpl v = (FsVolumeImpl)info.getVolume();
+if (v == null) {
+ LOG.error("Failed to delete replica " + block.getLocalBlock()
+ + ". No volume for this replica " + info + " in
removeReplicaFromMem.");
Review Comment:
Should we use {}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] haiyang1987 opened a new pull request, #4903: HDFS-16774.Improve async delete replica on datanode
haiyang1987 opened a new pull request, #4903: URL: https://github.com/apache/hadoop/pull/4903 ### Description of PR HDFS-16774.Improve async delete replica on datanode In our online cluster, a large number of ReplicaNotFoundExceptions occur when client reads the data. After tracing the root cause, it is caused by the asynchronous deletion of the replica operation has many stacked pending deletion caused ReplicationNotFoundException. Current the asynchronous delete of the replica operation process is as follows： 1.remove the replica from the ReplicaMap 2.delete the replica file on the disk [blocked in threadpool] 3.notifying namenode through IBR [blocked in threadpool] In order to avoid similar problems as much as possible, consider optimizing the execution flow： The deleting replica from ReplicaMap, deleting replica from disk and notifying namenode through IBR are processed in the same asynchronous thread. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18448) s3a endpoint per bucket configuration in pyspark is ignored
[ https://issues.apache.org/jira/browse/HADOOP-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606271#comment-17606271 ] Einav Hollander edited comment on HADOOP-18448 at 9/18/22 10:56 AM: @[Steve Loughran|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=stevel%40apache.org] is this possible to read from one endpoint + region and to write to another? this configuration is not working for me (e.g as if it is being ignored completely). what am I missing? does the endpoint must not be in a vpce endpoint format? was (Author: JIRAUSER295521): @[Steve Loughran|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=stevel%40apache.org] is this possible to read from one endpoint + region and to write to another? this configuration is not working for me (e.g as if it is being ignored completely). what am I missing? > s3a endpoint per bucket configuration in pyspark is ignored > --- > > Key: HADOOP-18448 > URL: https://issues.apache.org/jira/browse/HADOOP-18448 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.1 >Reporter: Einav Hollander >Priority: Major > > I'm using EMR emr-6.5.0 cluster in us-east-1 with ec2 instances. cluster is > running spark application using pyspark 3.2.1 > EMR is using Hadoop distribution:Amazon 3.2.1 > my spark application is reading from one bucket in us-west-2 and writing to a > bucket in us-east-1. > since I'm processing a large amount of data I'm paying a lot of money for the > network transport . in order to reduce the cost I have create a vpc interface > to s3 endpoint in us-west-2. inside the spark application I'm using aws cli > for reading the file names from us-west-2 bucket and it is working through > the s3 interface endpoint but when I use pyspark to read the data it is using > the us-east-1 s3 endpoint instead of the us-west-2 endpoint. > I tried to use per bucket configuration but it is being ignored although I > added it to the defualt configuration and to spark submit call. > I tried to set the following configuration but they are ignored: > '--conf', > "spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain", > '--conf', "spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem", > '--conf', "spark.hadoop.fs.s3a.bucket..endpoint= vpc endpoint>", > '--conf', "spark.hadoop.fs.s3a.bucket. -name>.endpoint.region=us-west-2", > '--conf', "spark.hadoop.fs.s3a.bucket..endpoint= gateway endpoint>", > '--conf', "spark.hadoop.fs.s3a.bucket. -name>.endpoint.region=us-east-1", > '--conf', "spark.hadoop.fs.s3a.path.style.access=false" -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18448) s3a endpoint per bucket configuration in pyspark is ignored
[ https://issues.apache.org/jira/browse/HADOOP-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606271#comment-17606271 ] Einav Hollander edited comment on HADOOP-18448 at 9/18/22 10:36 AM: @[Steve Loughran|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=stevel%40apache.org] is this possible to read from one endpoint + region and to write to another? this configuration is not working for me (e.g as if it is being ignored completely). what am I missing? was (Author: JIRAUSER295521): @[Steve Loughran|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=stevel%40apache.org] is this possible to read from one endpoint + region and to write to another? this configuration is not working for me . what am I missing? > s3a endpoint per bucket configuration in pyspark is ignored > --- > > Key: HADOOP-18448 > URL: https://issues.apache.org/jira/browse/HADOOP-18448 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.1 >Reporter: Einav Hollander >Priority: Major > > I'm using EMR emr-6.5.0 cluster in us-east-1 with ec2 instances. cluster is > running spark application using pyspark 3.2.1 > EMR is using Hadoop distribution:Amazon 3.2.1 > my spark application is reading from one bucket in us-west-2 and writing to a > bucket in us-east-1. > since I'm processing a large amount of data I'm paying a lot of money for the > network transport . in order to reduce the cost I have create a vpc interface > to s3 endpoint in us-west-2. inside the spark application I'm using aws cli > for reading the file names from us-west-2 bucket and it is working through > the s3 interface endpoint but when I use pyspark to read the data it is using > the us-east-1 s3 endpoint instead of the us-west-2 endpoint. > I tried to use per bucket configuration but it is being ignored although I > added it to the defualt configuration and to spark submit call. > I tried to set the following configuration but they are ignored: > '--conf', > "spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain", > '--conf', "spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem", > '--conf', "spark.hadoop.fs.s3a.bucket..endpoint= vpc endpoint>", > '--conf', "spark.hadoop.fs.s3a.bucket. -name>.endpoint.region=us-west-2", > '--conf', "spark.hadoop.fs.s3a.bucket..endpoint= gateway endpoint>", > '--conf', "spark.hadoop.fs.s3a.bucket. -name>.endpoint.region=us-east-1", > '--conf', "spark.hadoop.fs.s3a.path.style.access=false" -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-18448) s3a endpoint per bucket configuration in pyspark is ignored
[ https://issues.apache.org/jira/browse/HADOOP-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606271#comment-17606271 ] Einav Hollander edited comment on HADOOP-18448 at 9/18/22 10:34 AM: @[Steve Loughran|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=stevel%40apache.org] is this possible to read from one endpoint + region and to write to another? this configuration is not working for me . what am I missing? was (Author: JIRAUSER295521): @[Steve Loughran|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=stevel%40apache.org] is this possible to read from one endpoint + region and to write to another? > s3a endpoint per bucket configuration in pyspark is ignored > --- > > Key: HADOOP-18448 > URL: https://issues.apache.org/jira/browse/HADOOP-18448 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.1 >Reporter: Einav Hollander >Priority: Major > > I'm using EMR emr-6.5.0 cluster in us-east-1 with ec2 instances. cluster is > running spark application using pyspark 3.2.1 > EMR is using Hadoop distribution:Amazon 3.2.1 > my spark application is reading from one bucket in us-west-2 and writing to a > bucket in us-east-1. > since I'm processing a large amount of data I'm paying a lot of money for the > network transport . in order to reduce the cost I have create a vpc interface > to s3 endpoint in us-west-2. inside the spark application I'm using aws cli > for reading the file names from us-west-2 bucket and it is working through > the s3 interface endpoint but when I use pyspark to read the data it is using > the us-east-1 s3 endpoint instead of the us-west-2 endpoint. > I tried to use per bucket configuration but it is being ignored although I > added it to the defualt configuration and to spark submit call. > I tried to set the following configuration but they are ignored: > '--conf', > "spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain", > '--conf', "spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem", > '--conf', "spark.hadoop.fs.s3a.bucket..endpoint= vpc endpoint>", > '--conf', "spark.hadoop.fs.s3a.bucket. -name>.endpoint.region=us-west-2", > '--conf', "spark.hadoop.fs.s3a.bucket..endpoint= gateway endpoint>", > '--conf', "spark.hadoop.fs.s3a.bucket. -name>.endpoint.region=us-east-1", > '--conf', "spark.hadoop.fs.s3a.path.style.access=false" -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18448) s3a endpoint per bucket configuration in pyspark is ignored
[ https://issues.apache.org/jira/browse/HADOOP-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Einav Hollander updated HADOOP-18448: - Description: I'm using EMR emr-6.5.0 cluster in us-east-1 with ec2 instances. cluster is running spark application using pyspark 3.2.1 EMR is using Hadoop distribution:Amazon 3.2.1 my spark application is reading from one bucket in us-west-2 and writing to a bucket in us-east-1. since I'm processing a large amount of data I'm paying a lot of money for the network transport . in order to reduce the cost I have create a vpc interface to s3 endpoint in us-west-2. inside the spark application I'm using aws cli for reading the file names from us-west-2 bucket and it is working through the s3 interface endpoint but when I use pyspark to read the data it is using the us-east-1 s3 endpoint instead of the us-west-2 endpoint. I tried to use per bucket configuration but it is being ignored although I added it to the defualt configuration and to spark submit call. I tried to set the following configuration but they are ignored: '--conf', "spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain", '--conf', "spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem", '--conf', "spark.hadoop.fs.s3a.bucket..endpoint=", '--conf', "spark.hadoop.fs.s3a.bucket..endpoint.region=us-west-2", '--conf', "spark.hadoop.fs.s3a.bucket..endpoint=", '--conf', "spark.hadoop.fs.s3a.bucket..endpoint.region=us-east-1", '--conf', "spark.hadoop.fs.s3a.path.style.access=false" was: I'm using EMR emr-6.5.0 cluster in us-east-1 with ec2 instances. cluster is running spark application using pyspark 3.2.1 EMR is using Hadoop distribution:Amazon 3.2.1 my spark application is reading from one bucket in us-west-2 and writing to a bucket in us-east-1. since I'm processing a large amount of data I'm paying a lot of money for the network transport . in order to reduce the cost I have create a vpc interface to s3 endpoint in us-west-2. inside the spark application I'm using aws cli for reading the file names from us-west-2 bucket and it is working through the s3 interface endpoint but when I use pyspark to read the data it is using the us-east-1 s3 endpoint instead of the us-west-2 endpoint. I tried to use per bucket configuration but it is being ignored although I added it to the defualt configuration and to spark submit call. I tried to set the following configuration but they are ignored: '--conf', "spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain", '--conf', "spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem", '--conf', "spark.hadoop.fs.s3a.bucket..endpoint=", '--conf', "spark.hadoop.fs.s3a.bucket..endpoint.region=us-west-2", '--conf', "spark.hadoop.fs.s3a.bucket..endpoint=", '--conf', "spark.hadoop.fs.s3a.bucket..endpoint.region=us-east-1", '--conf', "spark.hadoop.fs.s3a.path.style.access=false", '--conf', "spark.eventLog.enabled=false", > s3a endpoint per bucket configuration in pyspark is ignored > --- > > Key: HADOOP-18448 > URL: https://issues.apache.org/jira/browse/HADOOP-18448 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.1 >Reporter: Einav Hollander >Priority: Major > > I'm using EMR emr-6.5.0 cluster in us-east-1 with ec2 instances. cluster is > running spark application using pyspark 3.2.1 > EMR is using Hadoop distribution:Amazon 3.2.1 > my spark application is reading from one bucket in us-west-2 and writing to a > bucket in us-east-1. > since I'm processing a large amount of data I'm paying a lot of money for the > network transport . in order to reduce the cost I have create a vpc interface > to s3 endpoint in us-west-2. inside the spark application I'm using aws cli > for reading the file names from us-west-2 bucket and it is working through > the s3 interface endpoint but when I use pyspark to read the data it is using > the us-east-1 s3 endpoint instead of the us-west-2 endpoint. > I tried to use per bucket configuration but it is being ignored although I > added it to the defualt configuration and to spark submit call. > I tried to set the following configuration but they are ignored: > '--conf', > "spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain", > '--conf', "spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem", > '--conf', "spark.hadoop.fs.s3a.bucket..endpoint= vpc endpoint>", > '--conf', "spark.hadoop.fs.s3a.bucket. -name>.endpoint.region=us-west-2", > '--conf', "spark.hadoop.fs.s3a.bucket..endpoint= gateway endpoint>", > '--conf', "spark.hadoop.fs.s3a.bucket. -name>.endpoint.region=us-east-1", > '--co
[jira] [Updated] (HADOOP-18448) s3a endpoint per bucket configuration in pyspark is ignored
[ https://issues.apache.org/jira/browse/HADOOP-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Einav Hollander updated HADOOP-18448: - Description: I'm using EMR emr-6.5.0 cluster in us-east-1 with ec2 instances. cluster is running spark application using pyspark 3.2.1 EMR is using Hadoop distribution:Amazon 3.2.1 my spark application is reading from one bucket in us-west-2 and writing to a bucket in us-east-1. since I'm processing a large amount of data I'm paying a lot of money for the network transport . in order to reduce the cost I have create a vpc interface to s3 endpoint in us-west-2. inside the spark application I'm using aws cli for reading the file names from us-west-2 bucket and it is working through the s3 interface endpoint but when I use pyspark to read the data it is using the us-east-1 s3 endpoint instead of the us-west-2 endpoint. I tried to use per bucket configuration but it is being ignored although I added it to the defualt configuration and to spark submit call. I tried to set the following configuration but they are ignored: '--conf', "spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain", '--conf', "spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem", '--conf', "spark.hadoop.fs.s3a.bucket..endpoint=", '--conf', "spark.hadoop.fs.s3a.bucket..endpoint.region=us-west-2", '--conf', "spark.hadoop.fs.s3a.bucket..endpoint=", '--conf', "spark.hadoop.fs.s3a.bucket..endpoint.region=us-east-1", '--conf', "spark.hadoop.fs.s3a.path.style.access=false", '--conf', "spark.eventLog.enabled=false", was: I'm using EMR emr-6.5.0 cluster in us-east-1 with ec2 instances. cluster is running spark application using pyspark 3.2.1 EMR is using Hadoop distribution:Amazon 3.2.1 my spark application is reading from one bucket in us-west-2 and writing to a bucket in us-east-1. since I'm processing a large amount of data I'm paying a lot of money for the network transport . in order to reduce the cost I have create a vpc interface to s3 endpoint in us-west-2. inside the spark application I'm using aws cli for reading the file names from us-west-2 bucket and it is working through the s3 interface endpoint but when I use pyspark to read the data it is using the us-east-1 s3 endpoint instead of the us-west-2 endpoint. I tried to use per bucket configuration but it is being ignored although I added it to the defualt configuration and to spark submit call. I tried to set the following configuration but they are ignored: '--conf', "spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain", '--conf', "spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem", '--conf', "spark.hadoop.fs.s3a.bucket..endpoint=", '--conf', "spark.hadoop.fs.s3a.bucket..endpoint.region=us-west-2", '--conf', "spark.hadoop.fs.s3a.bucket..endpoint=", '--conf', "spark.hadoop.fs.s3a.bucket..endpoint.region=us-east-1", '--conf', "spark.hadoop.fs.s3a.path.style.access=false", '--conf', "spark.executor.extraJavaOptions=-Dcom.amazonaws.services.s3.enableV4=true", '--conf', "spark.driver.extraJavaOptions=-Dcom.amazonaws.services.s3.enableV4=true", '--conf', "spark.eventLog.enabled=false", > s3a endpoint per bucket configuration in pyspark is ignored > --- > > Key: HADOOP-18448 > URL: https://issues.apache.org/jira/browse/HADOOP-18448 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.1 >Reporter: Einav Hollander >Priority: Major > > I'm using EMR emr-6.5.0 cluster in us-east-1 with ec2 instances. cluster is > running spark application using pyspark 3.2.1 > EMR is using Hadoop distribution:Amazon 3.2.1 > my spark application is reading from one bucket in us-west-2 and writing to a > bucket in us-east-1. > since I'm processing a large amount of data I'm paying a lot of money for the > network transport . in order to reduce the cost I have create a vpc interface > to s3 endpoint in us-west-2. inside the spark application I'm using aws cli > for reading the file names from us-west-2 bucket and it is working through > the s3 interface endpoint but when I use pyspark to read the data it is using > the us-east-1 s3 endpoint instead of the us-west-2 endpoint. > I tried to use per bucket configuration but it is being ignored although I > added it to the defualt configuration and to spark submit call. > I tried to set the following configuration but they are ignored: > '--conf', > "spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain", > '--conf', "spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem", > '--conf', "spark.hadoop.fs.s3a.bucket..endpoint= vpc endpoint>", > '--conf', "
[jira] [Commented] (HADOOP-18448) s3a endpoint per bucket configuration in pyspark is ignored
[ https://issues.apache.org/jira/browse/HADOOP-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606271#comment-17606271 ] Einav Hollander commented on HADOOP-18448: -- @[Steve Loughran|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=stevel%40apache.org] is this possible to read from one endpoint + region and to write to another? > s3a endpoint per bucket configuration in pyspark is ignored > --- > > Key: HADOOP-18448 > URL: https://issues.apache.org/jira/browse/HADOOP-18448 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.1 >Reporter: Einav Hollander >Priority: Major > > I'm using EMR emr-6.5.0 cluster in us-east-1 with ec2 instances. cluster is > running spark application using pyspark 3.2.1 > EMR is using Hadoop distribution:Amazon 3.2.1 > my spark application is reading from one bucket in us-west-2 and writing to a > bucket in us-east-1. > since I'm processing a large amount of data I'm paying a lot of money for the > network transport . in order to reduce the cost I have create a vpc interface > to s3 endpoint in us-west-2. inside the spark application I'm using aws cli > for reading the file names from us-west-2 bucket and it is working through > the s3 interface endpoint but when I use pyspark to read the data it is using > the us-east-1 s3 endpoint instead of the us-west-2 endpoint. > I tried to use per bucket configuration but it is being ignored although I > added it to the defualt configuration and to spark submit call. > I tried to set the following configuration but they are ignored: > '--conf', > "spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain", > '--conf', "spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem", > '--conf', "spark.hadoop.fs.s3a.bucket..endpoint= vpc endpoint>", > '--conf', "spark.hadoop.fs.s3a.bucket. -name>.endpoint.region=us-west-2", > '--conf', "spark.hadoop.fs.s3a.bucket..endpoint= gateway endpoint>", > '--conf', "spark.hadoop.fs.s3a.bucket. -name>.endpoint.region=us-east-1", > '--conf', "spark.hadoop.fs.s3a.path.style.access=false", > '--conf', > "spark.executor.extraJavaOptions=-Dcom.amazonaws.services.s3.enableV4=true", > '--conf', > "spark.driver.extraJavaOptions=-Dcom.amazonaws.services.s3.enableV4=true", > '--conf', "spark.eventLog.enabled=false", -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] haiyang1987 opened a new pull request, #4902: HDFS-16775.Improve BlockPlacementPolicyRackFaultTolerant's chooseOnce
haiyang1987 opened a new pull request, #4902: URL: https://github.com/apache/hadoop/pull/4902 ### Description of PR HDFS-16775. ### For code changes: Improve BlockPlacementPolicyRackFaultTolerant's chooseOnce -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18443) Upgrade snakeyaml to 1.31 to mitigate CVE-2022-25857
[ https://issues.apache.org/jira/browse/HADOOP-18443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606265#comment-17606265 ] PJ Fanning commented on HADOOP-18443: - v1.32 was released and added another security fix - would it be possible to change this issue and PR to use v1.32? https://github.com/advisories/GHSA-9w3m-gqgf-c4p9 > Upgrade snakeyaml to 1.31 to mitigate CVE-2022-25857 > > > Key: HADOOP-18443 > URL: https://issues.apache.org/jira/browse/HADOOP-18443 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.3.3, 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > > Upgrade snakeyaml to 1.31 to mitigate CVE-2022-25857 > > Details - https://nvd.nist.gov/vuln/detail/CVE-2022-25857 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18448) s3a endpoint per bucket configuration in pyspark is ignored
[ https://issues.apache.org/jira/browse/HADOOP-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Einav Hollander updated HADOOP-18448: - Component/s: fs/s3 (was: conf) > s3a endpoint per bucket configuration in pyspark is ignored > --- > > Key: HADOOP-18448 > URL: https://issues.apache.org/jira/browse/HADOOP-18448 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.1 >Reporter: Einav Hollander >Priority: Major > > I'm using EMR emr-6.5.0 cluster in us-east-1 with ec2 instances. cluster is > running spark application using pyspark 3.2.1 > EMR is using Hadoop distribution:Amazon 3.2.1 > my spark application is reading from one bucket in us-west-2 and writing to a > bucket in us-east-1. > since I'm processing a large amount of data I'm paying a lot of money for the > network transport . in order to reduce the cost I have create a vpc interface > to s3 endpoint in us-west-2. inside the spark application I'm using aws cli > for reading the file names from us-west-2 bucket and it is working through > the s3 interface endpoint but when I use pyspark to read the data it is using > the us-east-1 s3 endpoint instead of the us-west-2 endpoint. > I tried to use per bucket configuration but it is being ignored although I > added it to the defualt configuration and to spark submit call. > I tried to set the following configuration but they are ignored: > '--conf', > "spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain", > '--conf', "spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem", > '--conf', "spark.hadoop.fs.s3a.bucket..endpoint= vpc endpoint>", > '--conf', "spark.hadoop.fs.s3a.bucket. -name>.endpoint.region=us-west-2", > '--conf', "spark.hadoop.fs.s3a.bucket..endpoint= gateway endpoint>", > '--conf', "spark.hadoop.fs.s3a.bucket. -name>.endpoint.region=us-east-1", > '--conf', "spark.hadoop.fs.s3a.path.style.access=false", > '--conf', > "spark.executor.extraJavaOptions=-Dcom.amazonaws.services.s3.enableV4=true", > '--conf', > "spark.driver.extraJavaOptions=-Dcom.amazonaws.services.s3.enableV4=true", > '--conf', "spark.eventLog.enabled=false", -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4463: YARN-11187. Remove WhiteBox in yarn module.
hadoop-yetus commented on PR #4463:
URL: https://github.com/apache/hadoop/pull/4463#issuecomment-1250225231
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 17s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 7s | | trunk passed |
| +1 :green_heart: | compile | 1m 52s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 38s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 48s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 0s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 1m 33s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 32s | |
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 0 new + 105
unchanged - 9 fixed = 105 total (was 114) |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 25s | |
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 generated 0 new
+ 78 unchanged - 9 fixed = 78 total (was 87) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 29s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4463/16/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 43s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 16s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 23m 38s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4463/16/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 133m 29s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
|
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4463/16/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4463 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
[GitHub] [hadoop] hadoop-yetus commented on pull request #4462: MAPREDUCE-7390 Remove WhiteBox in mapreduce module.
hadoop-yetus commented on PR #4462:
URL: https://github.com/apache/hadoop/pull/4462#issuecomment-1250225071
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 2m 6s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 18s | | trunk passed |
| +1 :green_heart: | compile | 1m 3s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 51s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 56s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 51s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 14s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 40s | |
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 0 new + 100
unchanged - 1 fixed = 100 total (was 101) |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 42s | |
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 generated 0 new
+ 94 unchanged - 1 fixed = 94 total (was 95) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 37s | |
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
The patch generated 0 new + 97 unchanged - 4 fixed = 97 total (was 101) |
| +1 :green_heart: | mvnsite | 0m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 41s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 7m 17s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 105m 44s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4462/15/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4462 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4aefaf8d9041 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 26b2ee9de361d588d6eb3bc0df6af6e257b81fe2 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4462/15/testReport/ |
| Max. process+thread count | 1599 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduc
