[
https://issues.apache.org/jira/browse/HADOOP-19205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17856059#comment-17856059
]
ASF GitHub Bot commented on HADOOP-19205:
-----------------------------------------
hadoop-yetus commented on PR #6892:
URL: https://github.com/apache/hadoop/pull/6892#issuecomment-2176903980
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 7s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 47s | | trunk passed |
| +1 :green_heart: | compile | 8m 40s | | trunk passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | compile | 7m 57s | | trunk passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | checkstyle | 2m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | spotbugs | 2m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 8s | | branch has no errors
when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 48s | | the patch passed |
| +1 :green_heart: | compile | 8m 16s | | the patch passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| +1 :green_heart: | javac | 8m 16s | | the patch passed |
| +1 :green_heart: | compile | 7m 49s | | the patch passed with JDK
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| +1 :green_heart: | javac | 7m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 58s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6892/2/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 11 new + 17 unchanged - 3 fixed = 28 total (was
20) |
| +1 :green_heart: | mvnsite | 1m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 5s | | the patch passed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 |
| -1 :x: | javadoc | 0m 34s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_412-8u412-ga-1~20.04.1-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6892/2/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_412-8u412-ga-1~20.04.1-b08.txt)
| hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_412-8u412-ga-1~20.04.1-b08
with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 generated 2 new + 0
unchanged - 0 fixed = 2 total (was 0) |
| -1 :x: | spotbugs | 0m 59s |
[/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6892/2/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html)
| hadoop-tools/hadoop-aws generated 6 new + 0 unchanged - 0 fixed = 6 total
(was 0) |
| +1 :green_heart: | shadedclient | 20m 22s | | patch has no errors
when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 23m 37s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 25s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 153m 53s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-tools/hadoop-aws |
| | Possible doublecheck on
org.apache.hadoop.fs.s3a.impl.ClientManagerImpl.s3AsyncClient in
org.apache.hadoop.fs.s3a.impl.ClientManagerImpl.getOrCreateAsyncClient() At
ClientManagerImpl.java:org.apache.hadoop.fs.s3a.impl.ClientManagerImpl.getOrCreateAsyncClient()
At ClientManagerImpl.java:[lines 123-127] |
| | Possible doublecheck on
org.apache.hadoop.fs.s3a.impl.ClientManagerImpl.s3Client in
org.apache.hadoop.fs.s3a.impl.ClientManagerImpl.getOrCreateS3Client() At
ClientManagerImpl.java:org.apache.hadoop.fs.s3a.impl.ClientManagerImpl.getOrCreateS3Client()
At ClientManagerImpl.java:[lines 104-108] |
| | Possible doublecheck on
org.apache.hadoop.fs.s3a.impl.ClientManagerImpl.transferManager in
org.apache.hadoop.fs.s3a.impl.ClientManagerImpl.getOrCreateTransferManager()
At
ClientManagerImpl.java:org.apache.hadoop.fs.s3a.impl.ClientManagerImpl.getOrCreateTransferManager()
At ClientManagerImpl.java:[lines 143-146] |
| | Inconsistent synchronization of
org.apache.hadoop.fs.s3a.impl.ClientManagerImpl.s3AsyncClient; locked 40% of
time Unsynchronized access at ClientManagerImpl.java:40% of time
Unsynchronized access at ClientManagerImpl.java:[line 123] |
| | Inconsistent synchronization of
org.apache.hadoop.fs.s3a.impl.ClientManagerImpl.s3Client; locked 40% of time
Unsynchronized access at ClientManagerImpl.java:40% of time Unsynchronized
access at ClientManagerImpl.java:[line 104] |
| | Inconsistent synchronization of
org.apache.hadoop.fs.s3a.impl.ClientManagerImpl.transferManager; locked 40% of
time Unsynchronized access at ClientManagerImpl.java:40% of time
Unsynchronized access at ClientManagerImpl.java:[line 143] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6892/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6892 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux ccc570bdf976 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17
09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 83ed705a6b669397e1dd3589a4a11d631a663ee8 |
| Default Java | Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6892/2/testReport/ |
| Max. process+thread count | 1286 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws
U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6892/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> S3A initialization/close slower than with v1 SDK
> ------------------------------------------------
>
> Key: HADOOP-19205
> URL: https://issues.apache.org/jira/browse/HADOOP-19205
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Priority: Major
> Labels: pull-request-available
> Attachments: Screenshot 2024-06-14 at 17.12.59.png, Screenshot
> 2024-06-14 at 17.14.33.png
>
>
> Hive QE have observed slowdown in LLAP queries due to time to create and
> close s3a filesystems instances. A key aspect of that is they keep closing
> the fs instances (HIVE-27884), but looking at the profiles, the reason things
> seem to have regressed is
> * two s3 clients are being created (sync and async)
> * these seem to take a lot of time scanning the classpath for "global
> interceptors", which is at least an O(jars) operation; #of index entries in
> the zip files may factor too.
> Proposed:
> * create async client on demand when the transfer manager is invoked
> * look at why passwords are being scanned for if
> InstanceProfileCredentialsProvider is in use...that seems slow too
> SDK wishes
> * SDK maybe allow us to turn off that scan for interceptors?
> attaching screenshots of the profile. storediag snippet:
> {code}
> [001] fs.s3a.access.key = (unset)
> [002] fs.s3a.secret.key = (unset)
> [003] fs.s3a.session.token = (unset)
> [004] fs.s3a.server-side-encryption-algorithm = (unset)
> [005] fs.s3a.server-side-encryption.key = (unset)
> [006] fs.s3a.encryption.algorithm = (unset)
> [007] fs.s3a.encryption.key = (unset)
> [008] fs.s3a.aws.credentials.provider =
> "com.amazonaws.auth.InstanceProfileCredentialsProvider" [core-site.xml]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
