steveloughran commented on PR #7882:
URL: https://github.com/apache/hadoop/pull/7882#issuecomment-3412377933
Now that 3rd party is good, I'm getting S3 express happy, mainly by test
tuning. But many, many errors with vectored reads
```
[ERROR] Errors:
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testAllRangesMergedIntoOne
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead.testAllRangesMergedIntoOne » IO
Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead.testAllRangesMergedIntoOne » IO
Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testBufferSlicing
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead.testBufferSlicing » IO Server error
accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead.testBufferSlicing » IO Server error
accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testConsecutiveRanges
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead.testConsecutiveRanges » IO Server
error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead.testConsecutiveRanges » IO Server
error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testMultipleVectoredReads
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead.testMultipleVectoredReads » IO
Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead.testMultipleVectoredReads » IO
Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testNormalReadAfterVectoredRead
[INFO] Run 1: PASS
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead.testNormalReadAfterVectoredRead »
IO Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testReadVectoredWithAALStatsCollection
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead.testReadVectoredWithAALStatsCollection
» IO Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead.testReadVectoredWithAALStatsCollection
» IO Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testSomeRandomNonOverlappingRanges
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead.testSomeRandomNonOverlappingRanges
» IO Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead.testSomeRandomNonOverlappingRanges
» IO Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadAfterNormalRead
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadAfterNormalRead »
IO Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadAfterNormalRead »
IO Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadAndReadFully
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead>AbstractContractVectoredReadTest.testVectoredReadAndReadFully:220
» IO test/vectored_file.txt: Stream is closed!
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead>AbstractContractVectoredReadTest.testVectoredReadAndReadFully:220
» IO test/vectored_file.txt: Stream is closed!
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadMultipleRanges
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead>AbstractContractVectoredReadTest.testVectoredReadMultipleRanges:206
» Execution java.io.IOException: Server error accessing
s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead>AbstractContractVectoredReadTest.testVectoredReadMultipleRanges:206
» Execution java.io.IOException: Server error accessing
s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadWholeFile
[ERROR] Run 1:
ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadWholeFile » IO
Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[ERROR] Run 2:
ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadWholeFile » IO
Server error accessing s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
[INFO]
[INFO]
```
```
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testReadVectoredWithAALStatsCollection
-- Time elapsed: 2.201 s <<< ERROR!
java.io.IOException: Server error accessing
s3://stevel--usw2-az1--x-s3/test/vectored_file.txt
at
software.amazon.s3.analyticsaccelerator.exceptions.ExceptionHandler.createIOException(ExceptionHandler.java:94)
at
software.amazon.s3.analyticsaccelerator.exceptions.ExceptionHandler.lambda$static$3(ExceptionHandler.java:44)
at
software.amazon.s3.analyticsaccelerator.exceptions.ExceptionHandler.lambda$toIOException$6(ExceptionHandler.java:74)
at java.util.Optional.map(Optional.java:215)
at
software.amazon.s3.analyticsaccelerator.exceptions.ExceptionHandler.toIOException(ExceptionHandler.java:74)
at
software.amazon.s3.analyticsaccelerator.util.ObjectClientUtil.handleException(ObjectClientUtil.java:46)
at
software.amazon.s3.analyticsaccelerator.S3SyncSdkObjectClient.lambda$getObject$4(S3SyncSdkObjectClient.java:134)
at
software.amazon.s3.analyticsaccelerator.common.telemetry.DefaultTelemetry.measureImpl(DefaultTelemetry.java:181)
at
software.amazon.s3.analyticsaccelerator.common.telemetry.DefaultTelemetry.measure(DefaultTelemetry.java:98)
at
software.amazon.s3.analyticsaccelerator.common.telemetry.Telemetry.measureCritical(Telemetry.java:121)
at
software.amazon.s3.analyticsaccelerator.S3SyncSdkObjectClient.getObject(S3SyncSdkObjectClient.java:120)
at
software.amazon.s3.analyticsaccelerator.io.physical.reader.StreamReader.fetchObjectContent(StreamReader.java:276)
at
software.amazon.s3.analyticsaccelerator.io.physical.reader.StreamReader.lambda$null$2(StreamReader.java:193)
at
software.amazon.s3.shaded.dev.failsafe.Functions.lambda$toCtxSupplier$9(Functions.java:228)
at
software.amazon.s3.shaded.dev.failsafe.Functions.lambda$get$0(Functions.java:46)
at
software.amazon.s3.shaded.dev.failsafe.internal.TimeoutExecutor.lambda$apply$1(TimeoutExecutor.java:93)
at
software.amazon.s3.shaded.dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
at
software.amazon.s3.shaded.dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
at
software.amazon.s3.shaded.dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
at
software.amazon.s3.shaded.dev.failsafe.FailsafeExecutor.run(FailsafeExecutor.java:220)
at
software.amazon.s3.analyticsaccelerator.util.retry.DefaultRetryStrategyImpl.execute(DefaultRetryStrategyImpl.java:87)
at
software.amazon.s3.analyticsaccelerator.io.physical.reader.StreamReader.lambda$null$3(StreamReader.java:170)
at
software.amazon.s3.analyticsaccelerator.common.telemetry.DefaultTelemetry.measureImpl(DefaultTelemetry.java:156)
at
software.amazon.s3.analyticsaccelerator.common.telemetry.DefaultTelemetry.measure(DefaultTelemetry.java:77)
at
software.amazon.s3.analyticsaccelerator.common.telemetry.Telemetry.measureCritical(Telemetry.java:107)
at
software.amazon.s3.analyticsaccelerator.io.physical.reader.StreamReader.lambda$processReadTask$4(StreamReader.java:157)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: At least
one of the pre-conditions you specified did not hold (Service: S3, Status Code:
412, Request ID: 0161ddf60d000199ee568ca80509a4823423357c, Extended Request ID:
qRuo1CRhYqw) (SDK Attempt Count: 1)
at
software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:113)
at
software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:61)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:73)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
at
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at
software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
at
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
at
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
at
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$0(BaseSyncClientHandler.java:66)
at
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
at
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:60)
at
software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52)
at
software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:60)
at
software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:6416)
at
software.amazon.awssdk.services.s3.DelegatingS3Client.lambda$getObject$49(DelegatingS3Client.java:5196)
at
software.amazon.awssdk.services.s3.internal.crossregion.S3CrossRegionSyncClient.invokeOperation(S3CrossRegionSyncClient.java:67)
at
software.amazon.awssdk.services.s3.DelegatingS3Client.getObject(DelegatingS3Client.java:5196)
at
software.amazon.awssdk.services.s3.S3Client.getObject(S3Client.java:11199)
at
software.amazon.s3.analyticsaccelerator.S3SyncSdkObjectClient.lambda$getObject$4(S3SyncSdkObjectClient.java:131)
... 24 more
```
plus some saying stream is closed
```
[ERROR]
org.apache.hadoop.fs.contract.s3a.ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadAndReadFully
-- Time elapsed: 2.280 s <<< ERROR!
java.io.IOException: test/vectored_file.txt: Stream is closed!
at
org.apache.hadoop.fs.s3a.impl.streams.AnalyticsStream.throwIfClosed(AnalyticsStream.java:300)
at
org.apache.hadoop.fs.s3a.impl.streams.AnalyticsStream.seek(AnalyticsStream.java:95)
at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:85)
at
org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:124)
at
org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:129)
at
org.apache.hadoop.fs.contract.AbstractContractVectoredReadTest.testVectoredReadAndReadFully(AbstractContractVectoredReadTest.java:220)
```
this doesn't happen with other stores
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
