[
https://issues.apache.org/jira/browse/HADOOP-19299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17886445#comment-17886445
]
Steve Loughran commented on HADOOP-19299:
-----------------------------------------
{code}
.s3a.ITestS3AContractVectoredRead
[ERROR] testSomeRangesMergedSomeUnmerged[Buffer type :
array](org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead) Time
elapsed: 0.905 s <<< ERROR!
java.util.ConcurrentModificationException
at
java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1728)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
at
org.apache.hadoop.fs.store.audit.HttpReferrerAuditHeader.buildHttpReferrer(HttpReferrerAuditHeader.java:182)
at
org.apache.hadoop.fs.s3a.audit.impl.LoggingAuditor$LoggingAuditSpan.modifyHttpRequest(LoggingAuditor.java:388)
at
org.apache.hadoop.fs.s3a.audit.impl.ActiveAuditManagerS3A$WrappingAuditSpan.modifyHttpRequest(ActiveAuditManagerS3A.java:871)
at
org.apache.hadoop.fs.s3a.audit.impl.ActiveAuditManagerS3A.modifyHttpRequest(ActiveAuditManagerS3A.java:612)
at
software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.modifyHttpRequestAndHttpContent(ExecutionInterceptorChain.java:89)
at
software.amazon.awssdk.core.internal.handler.BaseClientHandler.runModifyHttpRequestAndHttpContentInterceptors(BaseClientHandler.java:157)
at
software.amazon.awssdk.core.internal.handler.BaseClientHandler.finalizeSdkHttpFullRequest(BaseClientHandler.java:83)
at
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:151)
at
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$0(BaseSyncClientHandler.java:66)
at
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
at
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:60)
at
software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52)
at
software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:60)
at
software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:5174)
at
software.amazon.awssdk.services.s3.S3Client.getObject(S3Client.java:9005)
at
org.apache.hadoop.fs.s3a.S3AFileSystem$InputStreamCallbacksImpl.getObject(S3AFileSystem.java:1934)
at
org.apache.hadoop.fs.s3a.S3AInputStream.lambda$getS3Object$7(S3AInputStream.java:1223)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:122)
at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:376)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:372)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:347)
at
org.apache.hadoop.fs.s3a.S3AInputStream.getS3Object(S3AInputStream.java:1220)
at
org.apache.hadoop.fs.s3a.S3AInputStream.getS3ObjectInputStream(S3AInputStream.java:1117)
at
org.apache.hadoop.fs.s3a.S3AInputStream.readCombinedRangeAndUpdateChildren(S3AInputStream.java:963)
at
org.apache.hadoop.fs.s3a.S3AInputStream.lambda$readVectored$5(S3AInputStream.java:945)
at
org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:225)
at
org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:225)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
{code}
> ConcurrentModificationException in HttpReferrerAuditHeader
> ----------------------------------------------------------
>
> Key: HADOOP-19299
> URL: https://issues.apache.org/jira/browse/HADOOP-19299
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.4.1
> Reporter: Steve Loughran
> Priority: Major
>
> Surfaced during a test run doing vector iO, where multiple parallel GETs were
> being issued within the same audit span, just when the header is built by
> enumerating the attributes.
> {code}
> queries = attributes.entrySet().stream()
> .filter(e -> !filter.contains(e.getKey()))
> .map(e -> e.getKey() + "=" + e.getValue())
> .collect(Collectors.joining("&"));
> {code}
> Hypothesis: multiple GET requests are conflicting in updating/reading the
> header.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
