[jira] [Commented] (HADOOP-17499) AbstractContractStreamIOStatisticsTest fails if read buffer not full
[
https://issues.apache.org/jira/browse/HADOOP-17499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17808222#comment-17808222
]
Steve Loughran commented on HADOOP-17499:
-
seen this again; should be asserting the bytes read count >= 0 and <= requested
len
> AbstractContractStreamIOStatisticsTest fails if read buffer not full
>
>
> Key: HADOOP-17499
> URL: https://issues.apache.org/jira/browse/HADOOP-17499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
>
> {code}
> [ERROR]
> testInputStreamStatisticRead(org.apache.hadoop.fs.s3a.statistics.ITestS3AContractStreamIOStatistics)
> Time elapsed: 6.252 s <<< FAILURE!
> org.junit.ComparisonFailure: [Counter named stream_read_bytes with expected
> value 129] expected:<[129]L> but was:<[3]L>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> {code}
> Test should handle all cases where bytes read > 0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19043) S3A: Regression: ITestS3AOpenCost fails on prefetch test runs
[
https://issues.apache.org/jira/browse/HADOOP-19043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807774#comment-17807774
]
Steve Loughran commented on HADOOP-19043:
-
{code}
[ERROR] Tests run: 8, Failures: 2, Errors: 2, Skipped: 1, Time elapsed: 52.665
s <<< FAILURE! - in org.apache.hadoop.fs.s3a.performance.ITestS3AOpenCost
[ERROR]
testVectorReadPastEOF(org.apache.hadoop.fs.s3a.performance.ITestS3AOpenCost)
Time elapsed: 5.406 s <<< ERROR!
java.lang.ClassCastException:
org.apache.hadoop.fs.s3a.prefetch.S3APrefetchingInputStream cannot be cast to
org.apache.hadoop.fs.s3a.S3AInputStream
at
org.apache.hadoop.fs.s3a.performance.ITestS3AOpenCost.assertS3StreamClosed(ITestS3AOpenCost.java:434)
at
org.apache.hadoop.fs.s3a.performance.ITestS3AOpenCost.lambda$testVectorReadPastEOF$14(ITestS3AOpenCost.java:412)
at
org.apache.hadoop.fs.s3a.performance.OperationCostValidator.exec(OperationCostValidator.java:167)
at
org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest.verifyMetrics(AbstractS3ACostTest.java:318)
at
org.apache.hadoop.fs.s3a.performance.ITestS3AOpenCost.testVectorReadPastEOF(ITestS3AOpenCost.java:409)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
[ERROR]
testPositionedReadableReadPastEOF(org.apache.hadoop.fs.s3a.performance.ITestS3AOpenCost)
Time elapsed: 5.929 s <<< FAILURE!
org.junit.ComparisonFailure: expected:<[16]> but was:<[-1]>
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
org.apache.hadoop.fs.s3a.performance.ITestS3AOpenCost.lambda$testPositionedReadableReadPastEOF$12(ITestS3AOpenCost.java:381)
at
org.apache.hadoop.fs.s3a.performance.OperationCostValidator.exec(OperationCostValidator.java:167)
at
org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest.verifyMetrics(AbstractS3ACostTest.java:318)
at
org.apache.hadoop.fs.s3a.performance.ITestS3AOpenCost.testPositionedReadableReadPastEOF(ITestS3AOpenCost.java:374)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
[ERROR]
testPositionedReadableReadFullyPastEOF(org.apache.hadoop.fs.s3a.performance.ITestS3AOpenCost)
Time elapsed: 7.195 s <<< ERROR!
java.lang.ClassCastExcep
[jira] [Created] (HADOOP-19043) S3A: Regression: ITestS3AOpenCost fails on prefetch test runs
Steve Loughran created HADOOP-19043:
---
Summary: S3A: Regression: ITestS3AOpenCost fails on prefetch test
runs
Key: HADOOP-19043
URL: https://issues.apache.org/jira/browse/HADOOP-19043
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3, test
Affects Versions: 3.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Getting test failures in the new ITestS3AOpenCost tests when run with
{{-Dprefetch}}
Thought I'd tested this, but clearly not
* class cast failures on asserts (fix: skip)
* bytes read different in one test: (fix: identify and address)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19042) S3A: detect and recover from SSL ConnectionReset exceptions
[
https://issues.apache.org/jira/browse/HADOOP-19042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807685#comment-17807685
]
Steve Loughran commented on HADOOP-19042:
-
{code}
Caused by: javax.net.ssl.SSLException: Connection reset
at sun.security.ssl.Alert.createSSLException(Alert.java:127)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:331)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:274)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:269)
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:138)
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1400)
at
sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1368)
at sun.security.ssl.SSLSocketImpl.access$300(SSLSocketImpl.java:73)
at
sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:962)
at
com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at
com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:197)
at
com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:176)
at
com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
at java.io.InputStream.skip(InputStream.java:224)
at
com.amazonaws.internal.SdkFilterInputStream.skip(SdkFilterInputStream.java:96)
at
com.amazonaws.internal.SdkFilterInputStream.skip(SdkFilterInputStream.java:96)
at
com.amazonaws.internal.SdkFilterInputStream.skip(SdkFilterInputStream.java:96)
at
com.amazonaws.internal.SdkFilterInputStream.skip(SdkFilterInputStream.java:96)
at
com.amazonaws.internal.SdkFilterInputStream.skip(SdkFilterInputStream.java:96)
at
com.amazonaws.util.LengthCheckInputStream.skip(LengthCheckInputStream.java:182)
at
com.amazonaws.internal.SdkFilterInputStream.skip(SdkFilterInputStream.java:96)
at
com.amazonaws.services.s3.internal.S3AbortableInputStream.skip(S3AbortableInputStream.java:155)
at
com.amazonaws.internal.SdkFilterInputStream.skip(SdkFilterInputStream.java:96)
at
org.apache.hadoop.fs.s3a.S3AInputStream.seekInStream(S3AInputStream.java:368)
at
org.apache.hadoop.fs.s3a.S3AInputStream.lambda$lazySeek$1(S3AInputStream.java:431)
at org.apache.hadoop.fs.s3a.Invoker.lambda$maybeRetry$3(Invoker.java:284)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:122)
at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:410)
at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:282)
at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:326)
at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:427)
at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:545)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.read(FileLink.java:132)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.hbase.io.util.BlockIOUtils.readWithExtraOnHeap(BlockIOUtils.java:130)
at
org.apache.hadoop.hbase.io.util.BlockIOUtils.readWithExtra(BlockIOUtils.java:163)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1486)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1717)
{code}
> S3A: detect and recover from SSL ConnectionReset exceptions
> ---
>
> Key: HADOOP-19042
> URL: https://issues.apache.org/jira/browse/HADOOP-19042
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0, 3.3.6
>Reporter: Steve Loughran
>Priority: Major
>
> s3a input stream doesn't recover from SSL exceptions, specifically
> ConnectionReset
> This is a variant of HADOOP-19027, except it's surfaced on an older release...
> # need to make sure the specific exception is handled by aborting stream and
> retrying -so map to the new HttpChannelEOFException
> # all of thisd needs to be backported
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19042) S3A: detect and recover from SSL ConnectionReset exceptions
Steve Loughran created HADOOP-19042: --- Summary: S3A: detect and recover from SSL ConnectionReset exceptions Key: HADOOP-19042 URL: https://issues.apache.org/jira/browse/HADOOP-19042 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.3.6, 3.4.0 Reporter: Steve Loughran s3a input stream doesn't recover from SSL exceptions, specifically ConnectionReset This is a variant of HADOOP-19027, except it's surfaced on an older release... # need to make sure the specific exception is handled by aborting stream and retrying -so map to the new HttpChannelEOFException # all of thisd needs to be backported -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19027) S3A: S3AInputStream doesn't recover from HTTP/channel exceptions
[
https://issues.apache.org/jira/browse/HADOOP-19027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-19027.
-
Fix Version/s: 3.5.0
Resolution: Fixed
in 3.5 though I hope to backport to 3.4.1
> S3A: S3AInputStream doesn't recover from HTTP/channel exceptions
>
>
> Key: HADOOP-19027
> URL: https://issues.apache.org/jira/browse/HADOOP-19027
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> S3AInputStream doesn't seem to recover from Http exceptions raised through
> HttpClient or through OpenSSL.
> * review the recovery code to make sure it is retrying enough, it looks
> suspiciously like it doesn't
> * detect the relevant openssl, shaded httpclient and unshaded httpclient
> exceptions, map to a standard one and treat as comms error in our retry policy
> This is not the same as the load balancer/proxy returning 443/444 which we
> map to AWSNoResponseException. We can't reuse that as it expects to be
> created from an
> {{software.amazon.awssdk.awscore.exception.AwsServiceException}} exception
> with the relevant fields...changing it could potentially be incompatible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19039) Hadoop 3.4.0 Highlight big features and improvements.
[ https://issues.apache.org/jira/browse/HADOOP-19039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807181#comment-17807181 ] Steve Loughran commented on HADOOP-19039: - + java17 support + where the logging is right now. This is the first Log4j 2 release. + what have we cut? Here is some markdown text regarding S3A changes. -- This release of Hadoop moves the S3A connector to Amazon S3 to the V2 SDK. This is a significant change which offers a number of new features including the ability to work with Amazon S3 Express One Zone Storage -the new high performance, single AZ storage class. Inevitably, there will be some surprises -there is ongoing work to address some performance regressions and resilience to network problems. * There is documentation on how to migrate to the new SDK -there is explicit support for moving V1 credential provider implementations; everything else will need rework. * There is also documentation on the third-party S3 store support; the SDK's focus on having coolers specify the AWS region rather than the S3 Service and point complicates migration. We have successfully tested it against Dell ECS S3 and Google GS Storage -validation with other "S3 compatible" stores would be very much appreciated.n * If you encounter issues, please look for HADOOP JIRA entries on the same topic -and if there are none create a new issues under [HADOOP-18886](https://issues.apache.org/jira/browse/HADOOP-18886) _S3A: AWS SDK V2 Migration: stabilization and S3Express_. We will try to Address these as soon as they have been identified. --- > Hadoop 3.4.0 Highlight big features and improvements. > - > > Key: HADOOP-19039 > URL: https://issues.apache.org/jira/browse/HADOOP-19039 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > > While preparing for the release of Hadoop-3.4.0, I've noticed the inclusion > of numerous commits in this version. Therefore, highlighting significant > features and improvements becomes crucial. I've completed the initial > version and now seek the review of more experienced partner to ensure the > finalization of the version's highlights. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19013) fs.getXattrs(path) for S3FS doesn't have x-amz-server-side-encryption-aws-kms-key-id header.
[ https://issues.apache.org/jira/browse/HADOOP-19013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807180#comment-17807180 ] Steve Loughran commented on HADOOP-19013: - well, serve it up if there's no extra HEAD request for it: but do not copy it > fs.getXattrs(path) for S3FS doesn't have > x-amz-server-side-encryption-aws-kms-key-id header. > > > Key: HADOOP-19013 > URL: https://issues.apache.org/jira/browse/HADOOP-19013 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.6 >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Major > > Once a path while uploading has been encrypted with SSE-KMS with a key id and > then later when we try to read the attributes of the same file, it doesn't > contain the key id information as an attribute. should we add it? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18717) Move CodecPool getCompressor/getDecompressor logs to DEBUG
[ https://issues.apache.org/jira/browse/HADOOP-18717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18717: Component/s: common > Move CodecPool getCompressor/getDecompressor logs to DEBUG > -- > > Key: HADOOP-18717 > URL: https://issues.apache.org/jira/browse/HADOOP-18717 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.3.6 >Reporter: Claire McGinty >Priority: Trivial > Labels: pull-request-available > > The "Got brand new compressor|decompressor" logs in CodecPool[0] can be quite > noisy when reading thousands of blocks and aren't that illuminating for the > end user. I'd like to propose moving them from log.info to log.debug if > there's no objection. > > [0] > https://github.com/apache/hadoop/blob/b737869e01fe3334b948a38fe3835e48873bf3a6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/CodecPool.java#L149-L195 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18717) Move CodecPool getCompressor/getDecompressor logs to DEBUG
[ https://issues.apache.org/jira/browse/HADOOP-18717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18717: Affects Version/s: 3.3.6 > Move CodecPool getCompressor/getDecompressor logs to DEBUG > -- > > Key: HADOOP-18717 > URL: https://issues.apache.org/jira/browse/HADOOP-18717 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: Claire McGinty >Priority: Trivial > Labels: pull-request-available > > The "Got brand new compressor|decompressor" logs in CodecPool[0] can be quite > noisy when reading thousands of blocks and aren't that illuminating for the > end user. I'd like to propose moving them from log.info to log.debug if > there's no objection. > > [0] > https://github.com/apache/hadoop/blob/b737869e01fe3334b948a38fe3835e48873bf3a6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/CodecPool.java#L149-L195 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19037) S3A: S3A: ITestS3AConfiguration failing with region problems
[
https://issues.apache.org/jira/browse/HADOOP-19037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17806094#comment-17806094
]
Steve Loughran commented on HADOOP-19037:
-
stack trace comes from creating sts client with no region setting. either it
needs to be a default or we use bucket location of test fs as driver
{code}
StsClient stsClient =
STSClientFactory.builder(config, bucket, new
AnonymousAWSCredentialsProvider(), "",
"").build();
{code}
> S3A: S3A: ITestS3AConfiguration failing with region problems
>
>
> Key: HADOOP-19037
> URL: https://issues.apache.org/jira/browse/HADOOP-19037
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Priority: Major
>
> After commented out the default region in my ~/.aws/config [default} profile,
> test ITestS3AConfiguration. testS3SpecificSignerOverride() fails
> {code}
> [ERROR]
> testS3SpecificSignerOverride(org.apache.hadoop.fs.s3a.ITestS3AConfiguration)
> Time elapsed: 0.054 s <<< ERROR!
> software.amazon.awssdk.core.exception.SdkClientException: Unable to load
> region from any of the providers in the chain
> software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@12c626f8:
>
> [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@ae63559:
> Unable to load region from system settings. Region must be specified either
> via environment variable (AWS_REGION) or system property (aws.region).,
> software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@6e6cfd4c:
> No region provided in profile: default,
> software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@139147de:
> EC2 Metadata is disabled. Unable to retrieve region information from EC2
> Metadata service.]
> {code}
> I'm worried the sdk update has rolled back to the 3.3.x region problems where
> well-configured developer setups / ec2 deployments hid problems. certainly we
> can see the code is checking these paths
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19037) S3A: S3A: ITestS3AConfiguration failing with region problems
[
https://issues.apache.org/jira/browse/HADOOP-19037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17806093#comment-17806093
]
Steve Loughran commented on HADOOP-19037:
-
{code}
[ERROR]
testS3SpecificSignerOverride(org.apache.hadoop.fs.s3a.ITestS3AConfiguration)
Time elapsed: 0.054 s <<< ERROR!
software.amazon.awssdk.core.exception.SdkClientException: Unable to load region
from any of the providers in the chain
software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@12c626f8:
[software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@ae63559:
Unable to load region from system settings. Region must be specified either via
environment variable (AWS_REGION) or system property (aws.region).,
software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@6e6cfd4c: No
region provided in profile: default,
software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@139147de:
EC2 Metadata is disabled. Unable to retrieve region information from EC2
Metadata service.]
at
software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at
software.amazon.awssdk.regions.providers.AwsRegionProviderChain.getRegion(AwsRegionProviderChain.java:70)
at
software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.resolveRegion(AwsDefaultClientBuilder.java:293)
at
software.amazon.awssdk.utils.AttributeMap$DerivedValue.primeCache(AttributeMap.java:600)
at
software.amazon.awssdk.utils.AttributeMap$DerivedValue.get(AttributeMap.java:589)
at
software.amazon.awssdk.utils.AttributeMap$Builder.resolveValue(AttributeMap.java:396)
at
software.amazon.awssdk.utils.AttributeMap$Builder.internalGet(AttributeMap.java:389)
at
software.amazon.awssdk.utils.AttributeMap$Builder.access$1300(AttributeMap.java:201)
at
software.amazon.awssdk.utils.AttributeMap$Builder$1.get(AttributeMap.java:399)
at
software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.resolveSigningRegion(AwsDefaultClientBuilder.java:260)
at
software.amazon.awssdk.utils.AttributeMap$DerivedValue.primeCache(AttributeMap.java:600)
at
software.amazon.awssdk.utils.AttributeMap$DerivedValue.get(AttributeMap.java:589)
at
software.amazon.awssdk.utils.AttributeMap$Builder.resolveValue(AttributeMap.java:396)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at
software.amazon.awssdk.utils.AttributeMap$Builder.build(AttributeMap.java:362)
at
software.amazon.awssdk.core.client.config.SdkClientConfiguration$Builder.build(SdkClientConfiguration.java:232)
at
software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.finalizeAwsConfiguration(AwsDefaultClientBuilder.java:184)
at
software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.finalizeChildConfiguration(AwsDefaultClientBuilder.java:161)
at
software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.syncClientConfiguration(SdkDefaultClientBuilder.java:188)
at
software.amazon.awssdk.services.sts.DefaultStsClientBuilder.buildClient(DefaultStsClientBuilder.java:36)
at
software.amazon.awssdk.services.sts.DefaultStsClientBuilder.buildClient(DefaultStsClientBuilder.java:25)
at
software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.build(SdkDefaultClientBuilder.java:155)
at
org.apache.hadoop.fs.s3a.ITestS3AConfiguration.testS3SpecificSignerOverride(ITestS3AConfiguration.java:569)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
{code}
> S3A: S3A: ITestS3AConfiguration failing with region problems
>
>
> Key: HADOOP-19037
> URL: https://issues.apache.org/jira/browse/HADOOP-19037
> Project: Hadoop Common
>
[jira] [Created] (HADOOP-19037) S3A: S3A: ITestS3AConfiguration failing with region problems
Steve Loughran created HADOOP-19037:
---
Summary: S3A: S3A: ITestS3AConfiguration failing with region
problems
Key: HADOOP-19037
URL: https://issues.apache.org/jira/browse/HADOOP-19037
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3, test
Affects Versions: 3.4.0
Reporter: Steve Loughran
After commented out the default region in my ~/.aws/config [default} profile,
test ITestS3AConfiguration. testS3SpecificSignerOverride() fails
{code}
[ERROR]
testS3SpecificSignerOverride(org.apache.hadoop.fs.s3a.ITestS3AConfiguration)
Time elapsed: 0.054 s <<< ERROR!
software.amazon.awssdk.core.exception.SdkClientException: Unable to load region
from any of the providers in the chain
software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@12c626f8:
[software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@ae63559:
Unable to load region from system settings. Region must be specified either via
environment variable (AWS_REGION) or system property (aws.region).,
software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@6e6cfd4c: No
region provided in profile: default,
software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@139147de:
EC2 Metadata is disabled. Unable to retrieve region information from EC2
Metadata service.]
{code}
I'm worried the sdk update has rolled back to the 3.3.x region problems where
well-configured developer setups / ec2 deployments hid problems. certainly we
can see the code is checking these paths
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19033) S3A: disable checksum validation
[ https://issues.apache.org/jira/browse/HADOOP-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-19033: Parent Issue: HADOOP-18886 (was: HADOOP-18477) > S3A: disable checksum validation > > > Key: HADOOP-19033 > URL: https://issues.apache.org/jira/browse/HADOOP-19033 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > AWS v2 sdk turns on client-side checksum validation; this kills performance > Given we are using TLS to download from AWS s3, there's implicit channel > checksumming going on on, that's along with the IPv4 TCP checksumming. > We don't need it, all it does is slow us down. > proposed: disable in DefaultS3ClientFactory > I don't want to add an option to enable it as it only complicates life (yet > another config option), but I am open to persuasion -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19033) S3A: disable checksum validation
[ https://issues.apache.org/jira/browse/HADOOP-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-19033: Affects Version/s: 3.4.0 > S3A: disable checksum validation > > > Key: HADOOP-19033 > URL: https://issues.apache.org/jira/browse/HADOOP-19033 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > AWS v2 sdk turns on client-side checksum validation; this kills performance > Given we are using TLS to download from AWS s3, there's implicit channel > checksumming going on on, that's along with the IPv4 TCP checksumming. > We don't need it, all it does is slow us down. > proposed: disable in DefaultS3ClientFactory > I don't want to add an option to enable it as it only complicates life (yet > another config option), but I am open to persuasion -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18981) Move oncrpc/portmap from hadoop-nfs to hadoop-common
[ https://issues.apache.org/jira/browse/HADOOP-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18981: Fix Version/s: 3.5.0 (was: 3.4.0) > Move oncrpc/portmap from hadoop-nfs to hadoop-common > > > Key: HADOOP-18981 > URL: https://issues.apache.org/jira/browse/HADOOP-18981 > Project: Hadoop Common > Issue Type: Improvement > Components: net >Affects Versions: 3.4.0 >Reporter: Xing Lin >Assignee: Xing Lin >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > We want to use udpserver/client for other use cases, rather than only for > NFS. One such use case is to export NameNodeHAState for NameNodes via a UDP > server. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19004) S3A: Support Authentication through HttpSigner API
[
https://issues.apache.org/jira/browse/HADOOP-19004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-19004:
Fix Version/s: 3.5.0
> S3A: Support Authentication through HttpSigner API
> ---
>
> Key: HADOOP-19004
> URL: https://issues.apache.org/jira/browse/HADOOP-19004
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Harshit Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> The latest AWS SDK changes how signing works, and for signing S3Express
> signatures the new {{software.amazon.awssdk.http.auth}} auth mechanism is
> needed
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19004) S3A: Support Authentication through HttpSigner API
[
https://issues.apache.org/jira/browse/HADOOP-19004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-19004:
Fix Version/s: (was: 3.4.0)
> S3A: Support Authentication through HttpSigner API
> ---
>
> Key: HADOOP-19004
> URL: https://issues.apache.org/jira/browse/HADOOP-19004
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Harshit Gupta
>Priority: Major
> Labels: pull-request-available
>
> The latest AWS SDK changes how signing works, and for signing S3Express
> signatures the new {{software.amazon.awssdk.http.auth}} auth mechanism is
> needed
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19004) S3A: Support Authentication through HttpSigner API
[
https://issues.apache.org/jira/browse/HADOOP-19004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-19004.
-
Fix Version/s: 3.4.0
Resolution: Fixed
> S3A: Support Authentication through HttpSigner API
> ---
>
> Key: HADOOP-19004
> URL: https://issues.apache.org/jira/browse/HADOOP-19004
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Harshit Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> The latest AWS SDK changes how signing works, and for signing S3Express
> signatures the new {{software.amazon.awssdk.http.auth}} auth mechanism is
> needed
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18981) Move oncrpc/portmap from hadoop-nfs to hadoop-common
[ https://issues.apache.org/jira/browse/HADOOP-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18981: Component/s: net > Move oncrpc/portmap from hadoop-nfs to hadoop-common > > > Key: HADOOP-18981 > URL: https://issues.apache.org/jira/browse/HADOOP-18981 > Project: Hadoop Common > Issue Type: Improvement > Components: net >Affects Versions: 3.4.0 >Reporter: Xing Lin >Assignee: Xing Lin >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > We want to use udpserver/client for other use cases, rather than only for > NFS. One such use case is to export NameNodeHAState for NameNodes via a UDP > server. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18981) Move oncrpc/portmap from hadoop-nfs to hadoop-common
[ https://issues.apache.org/jira/browse/HADOOP-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18981. - Fix Version/s: 3.4.0 Resolution: Fixed > Move oncrpc/portmap from hadoop-nfs to hadoop-common > > > Key: HADOOP-18981 > URL: https://issues.apache.org/jira/browse/HADOOP-18981 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: Xing Lin >Assignee: Xing Lin >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > We want to use udpserver/client for other use cases, rather than only for > NFS. One such use case is to export NameNodeHAState for NameNodes via a UDP > server. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19033) S3A: disable checksum validation
Steve Loughran created HADOOP-19033: --- Summary: S3A: disable checksum validation Key: HADOOP-19033 URL: https://issues.apache.org/jira/browse/HADOOP-19033 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Steve Loughran Assignee: Steve Loughran AWS v2 sdk turns on client-side checksum validation; this kills performance Given we are using TLS to download from AWS s3, there's implicit channel checksumming going on on, that's along with the IPv4 TCP checksumming. We don't need it, all it does is slow us down. proposed: disable in DefaultS3ClientFactory I don't want to add an option to enable it as it only complicates life (yet another config option), but I am open to persuasion -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19032) S3A: ITestS3AFileContextURI: MultiObjectDeleteException bulk delete of odd filenames
[
https://issues.apache.org/jira/browse/HADOOP-19032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-19032:
Summary: S3A: ITestS3AFileContextURI: MultiObjectDeleteException bulk
delete of odd filenames (was: S3A: ITestS3AFileContextURI:
MultiObjectDeleteException bulk delete of odd filenames in )
> S3A: ITestS3AFileContextURI: MultiObjectDeleteException bulk delete of odd
> filenames
>
>
> Key: HADOOP-19032
> URL: https://issues.apache.org/jira/browse/HADOOP-19032
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Priority: Major
>
> Possibly transient. note bucket is versioned.
> {code}
> org.apache.hadoop.fs.s3a.AWSS3IOException:
> Remove S3 Dir Markers on
> s3a://stevel-london/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/7/testContextURI/createTest:
> org.apache.hadoop.fs.s3a.impl.MultiObjectDeleteException:
> [S3Error(Key=Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/7/testContextURI/createTest/()&^%$#@!~_+}{> Code=InternalError, Message=We encountered an internal error. Please try
> again.)] (Service: Amazon S3, Status Code: 200, Request ID:
> null):MultiObjectDeleteException:
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19032) S3A: ITestS3AFileContextURI: MultiObjectDeleteException bulk delete of odd filenames in
[
https://issues.apache.org/jira/browse/HADOOP-19032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-19032:
Summary: S3A: ITestS3AFileContextURI: MultiObjectDeleteException bulk
delete of odd filenames in (was: MultiObjectDeleteException bulk delete of
odd filenames)
> S3A: ITestS3AFileContextURI: MultiObjectDeleteException bulk delete of odd
> filenames in
>
>
> Key: HADOOP-19032
> URL: https://issues.apache.org/jira/browse/HADOOP-19032
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Priority: Major
>
> Possibly transient. note bucket is versioned.
> {code}
> org.apache.hadoop.fs.s3a.AWSS3IOException:
> Remove S3 Dir Markers on
> s3a://stevel-london/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/7/testContextURI/createTest:
> org.apache.hadoop.fs.s3a.impl.MultiObjectDeleteException:
> [S3Error(Key=Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/7/testContextURI/createTest/()&^%$#@!~_+}{> Code=InternalError, Message=We encountered an internal error. Please try
> again.)] (Service: Amazon S3, Status Code: 200, Request ID:
> null):MultiObjectDeleteException:
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19032) MultiObjectDeleteException bulk delete of odd filenames
[
https://issues.apache.org/jira/browse/HADOOP-19032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805169#comment-17805169
]
Steve Loughran commented on HADOOP-19032:
-
{code}
[ERROR] Tests run: 17, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 48.129
s <<< FAILURE! - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
[ERROR]
testCreateDirectory(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI)
Time elapsed: 13.4 s <<< ERROR!
org.apache.hadoop.fs.s3a.AWSS3IOException:
Remove S3 Dir Markers on
s3a://stevel-london/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/7/testContextURI/createTest:
org.apache.hadoop.fs.s3a.impl.MultiObjectDeleteException:
[S3Error(Key=Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/7/testContextURI/createTest/()&^%$#@!~_+}{> MultiObjectDeleteException bulk delete of odd filenames
> ---
>
> Key: HADOOP-19032
> URL: https://issues.apache.org/jira/browse/HADOOP-19032
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Priority: Major
>
> Possibly transient. note bucket is versioned.
> {code}
> org.apache.hadoop.fs.s3a.AWSS3IOException:
> Remove S3 Dir Markers on
> s3a://stevel-london/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/7/testContextURI/createTest:
> org.apache.hadoop.fs.s3a.impl.MultiObjectDeleteException:
> [S3Error(Key=Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/7/testContextURI/createTest/()&^%$#@!~_+}{> Code=InternalError, Message=We encountered an internal error. Please try
> again.)] (Service: Amazon S3, Status Code: 200, Request ID:
> null):MultiObjectDeleteException:
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19032) MultiObjectDeleteException bulk delete of odd filenames
Steve Loughran created HADOOP-19032:
---
Summary: MultiObjectDeleteException bulk delete of odd filenames
Key: HADOOP-19032
URL: https://issues.apache.org/jira/browse/HADOOP-19032
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.4.0
Reporter: Steve Loughran
Possibly transient. note bucket is versioned.
{code}
org.apache.hadoop.fs.s3a.AWSS3IOException:
Remove S3 Dir Markers on
s3a://stevel-london/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/7/testContextURI/createTest:
org.apache.hadoop.fs.s3a.impl.MultiObjectDeleteException:
[S3Error(Key=Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/7/testContextURI/createTest/()&^%$#@!~_+}{>
[jira] [Commented] (HADOOP-19027) S3A: S3AInputStream doesn't recover from HTTP/channel exceptions
[
https://issues.apache.org/jira/browse/HADOOP-19027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804710#comment-17804710
]
Steve Loughran commented on HADOOP-19027:
-
doing the mocking tests to simulate all the failures. complex
> S3A: S3AInputStream doesn't recover from HTTP/channel exceptions
>
>
> Key: HADOOP-19027
> URL: https://issues.apache.org/jira/browse/HADOOP-19027
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> S3AInputStream doesn't seem to recover from Http exceptions raised through
> HttpClient or through OpenSSL.
> * review the recovery code to make sure it is retrying enough, it looks
> suspiciously like it doesn't
> * detect the relevant openssl, shaded httpclient and unshaded httpclient
> exceptions, map to a standard one and treat as comms error in our retry policy
> This is not the same as the load balancer/proxy returning 443/444 which we
> map to AWSNoResponseException. We can't reuse that as it expects to be
> created from an
> {{software.amazon.awssdk.awscore.exception.AwsServiceException}} exception
> with the relevant fields...changing it could potentially be incompatible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19027) S3A: S3AInputStream doesn't recover from HTTP/channel exceptions
[
https://issues.apache.org/jira/browse/HADOOP-19027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-19027:
Summary: S3A: S3AInputStream doesn't recover from HTTP/channel exceptions
(was: S3A: S3AInputStream doesn't recover from HTTP exceptions)
> S3A: S3AInputStream doesn't recover from HTTP/channel exceptions
>
>
> Key: HADOOP-19027
> URL: https://issues.apache.org/jira/browse/HADOOP-19027
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> S3AInputStream doesn't seem to recover from Http exceptions raised through
> HttpClient or through OpenSSL.
> * review the recovery code to make sure it is retrying enough, it looks
> suspiciously like it doesn't
> * detect the relevant openssl, shaded httpclient and unshaded httpclient
> exceptions, map to a standard one and treat as comms error in our retry policy
> This is not the same as the load balancer/proxy returning 443/444 which we
> map to AWSNoResponseException. We can't reuse that as it expects to be
> created from an
> {{software.amazon.awssdk.awscore.exception.AwsServiceException}} exception
> with the relevant fields...changing it could potentially be incompatible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19027) S3A: S3AInputStream doesn't recover from HTTP exceptions
[
https://issues.apache.org/jira/browse/HADOOP-19027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803659#comment-17803659
]
Steve Loughran commented on HADOOP-19027:
-
The retry logic in S3AInputStream only seems to retry on the first GET, later
ones is treats as fatal and assumes this is a file version issue.
This is from https://github.com/apache/hadoop/pull/794
{code}
// With S3Guard, the metadatastore gave us metadata for the file in
// open(), so we use a slightly different retry policy, but only on initial
// open. After that, an exception generally means the file has changed
// and there is no point retrying anymore.
Invoker invoker = context.getReadInvoker();
invoker.maybeRetry(streamStatistics.openOperations == 0,
"lazySeek", pathStr, true, ...
{code}
# We want to retry on failures, except for version change events.
# we need to map stream problems (closed, no response) to retryable
> S3A: S3AInputStream doesn't recover from HTTP exceptions
>
>
> Key: HADOOP-19027
> URL: https://issues.apache.org/jira/browse/HADOOP-19027
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> S3AInputStream doesn't seem to recover from Http exceptions raised through
> HttpClient or through OpenSSL.
> * review the recovery code to make sure it is retrying enough, it looks
> suspiciously like it doesn't
> * detect the relevant openssl, shaded httpclient and unshaded httpclient
> exceptions, map to a standard one and treat as comms error in our retry policy
> This is not the same as the load balancer/proxy returning 443/444 which we
> map to AWSNoResponseException. We can't reuse that as it expects to be
> created from an
> {{software.amazon.awssdk.awscore.exception.AwsServiceException}} exception
> with the relevant fields...changing it could potentially be incompatible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19027) S3A: S3AInputStream doesn't recover from HTTP exceptions
[
https://issues.apache.org/jira/browse/HADOOP-19027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803654#comment-17803654
]
Steve Loughran commented on HADOOP-19027:
-
full stack trace
{code}
java.lang.RuntimeException:
software.amazon.awssdk.thirdparty.org.apache.http.NoHttpResponseException:
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute
HTTP request: The target server failed to respond: The target server failed to
respond
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:351)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:280)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:84)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:70)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:70)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:40)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.io.IOException: java.lang.RuntimeException:
software.amazon.awssdk.thirdparty.org.apache.http.NoHttpResponseException:
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute
HTTP request: The target server failed to respond: The target server failed to
respond
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:80)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:437)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:297)
... 16 more
Caused by: java.io.IOException: java.lang.RuntimeException:
software.amazon.awssdk.thirdparty.org.apache.http.NoHttpResponseException:
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute
HTTP request: The target server failed to respond: The target server failed to
respond
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:381)
at
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:82)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:119)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:59)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
at
org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
... 18 more
Caused by: java.lang.RuntimeException:
software.amazon.awssdk.thirdparty.org.apache.http.NoHttpResponseException:
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute
HTTP request: The target server failed to respond: The target server failed to
respond
at
org.apache.iceberg.mr.hive.vector.HiveBatchIterator.advance(HiveBatchIterator.java:129)
at
org.apache.iceberg.mr.hive.vector.HiveBatchIterator.hasNext(HiveBatchIterator.java:137)
at
org.apache.iceberg.mr.hive.vector.HiveDeleteFilter$1$1.hasNext(HiveDeleteFilter.java:98)
at
org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.nextKeyValue(IcebergInputFormat.java:299)
at
org.apache.iceberg.mr.hive.vector.HiveIcebergVectorizedRecordReader.next(HiveI
[jira] [Commented] (HADOOP-19027) S3A: S3AInputStream doesn't recover from HTTP exceptions
[
https://issues.apache.org/jira/browse/HADOOP-19027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803649#comment-17803649
]
Steve Loughran commented on HADOOP-19027:
-
openssl, "stream is closed"
{code}
Error while running task ( failure ) :
attempt_1703842027450_0084_4_02_04_0:java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
java.lang.RuntimeException: java.io.IOException:
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute
HTTP request: WFOPENSSL0035 Stream is closed: WFOPENSSL0035 Stream is closed
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:351)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:280)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:84)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:70)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:70)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:40)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.io.IOException: java.lang.RuntimeException: java.io.IOException:
software.amazon.awssdk.core.exception.SdkClientException:
Unable to execute HTTP request: WFOPENSSL0035 Stream is closed: WFOPENSSL0035
Stream is closed
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:80)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:437)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:297)
... 16 more
Caused by: java.io.IOException: java.lang.RuntimeException:
java.io.IOException: software.amazon.awssdk.core.exception.SdkClientException:
Unable to execute HTTP request: WFOPENSSL0035 Stream is closed: WFOPENSSL0035
Stream is closed
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:381)
at
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:82)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:119)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:59)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
... 18 more
Caused by: java.lang.RuntimeException: java.io.IOException:
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute
HTTP request: WFOPENSSL0035 Stream is closed: WFOPENS
SL0035 Stream is closed
at
org.apache.iceberg.mr.hive.vector.HiveBatchIterator.advance(HiveBatchIterator.java:129){noformat}
{code}
and httpclient
software.amazon.awssdk.thirdparty.org.apache.http.NoHttpResponseException with
The target server failed to respond""
{code}
Error while running task ( failure ) :
attempt_1704316855291_0218_13_02_04_0:java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
java.lang.RuntimeException:
software.amazon.awssdk.thirdparty.org.apache.http.NoHttpResponseException:
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute
HTTP request: The target server failed to respond: The target server failed to
respond
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initi
[jira] [Created] (HADOOP-19027) S3A: S3AInputStream doesn't recover from HTTP exceptions
Steve Loughran created HADOOP-19027:
---
Summary: S3A: S3AInputStream doesn't recover from HTTP exceptions
Key: HADOOP-19027
URL: https://issues.apache.org/jira/browse/HADOOP-19027
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
S3AInputStream doesn't seem to recover from Http exceptions raised through
HttpClient or through OpenSSL.
* review the recovery code to make sure it is retrying enough, it looks
suspiciously like it doesn't
* detect the relevant openssl, shaded httpclient and unshaded httpclient
exceptions, map to a standard one and treat as comms error in our retry policy
This is not the same as the load balancer/proxy returning 443/444 which we map
to AWSNoResponseException. We can't reuse that as it expects to be created from
an {{software.amazon.awssdk.awscore.exception.AwsServiceException}} exception
with the relevant fields...changing it could potentially be incompatible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18886) S3A: AWS SDK V2 Migration: stabilization and S3Express
[ https://issues.apache.org/jira/browse/HADOOP-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18886: Description: The final stabilisation changes to the V2 SDK MIgration; those moved off the HADOOP-18073 JIRA so we can close that. also adds support to Amazon S3 Express One Zone storage was:The final stabilisation changes to the V2 SDK MIgration; those moved off the HADOOP-18073 JIRA so we can close that. > S3A: AWS SDK V2 Migration: stabilization and S3Express > -- > > Key: HADOOP-18886 > URL: https://issues.apache.org/jira/browse/HADOOP-18886 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: ahmar#1 >Priority: Major > > The final stabilisation changes to the V2 SDK MIgration; those moved off the > HADOOP-18073 JIRA so we can close that. > also adds support to Amazon S3 Express One Zone storage -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19023) ITestS3AConcurrentOps#testParallelRename intermittent timeout failure
[
https://issues.apache.org/jira/browse/HADOOP-19023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803521#comment-17803521
]
Steve Loughran commented on HADOOP-19023:
-
* make sure you've not got a site config with an aggressive timeout
* do set version/component in the issue fields...it's not picked up from the
parent
> ITestS3AConcurrentOps#testParallelRename intermittent timeout failure
> -
>
> Key: HADOOP-19023
> URL: https://issues.apache.org/jira/browse/HADOOP-19023
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Viraj Jasani
>Priority: Major
>
> Need to configure higher timeout for the test.
>
> {code:java}
> [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 256.281 s <<< FAILURE! - in
> org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps
> [ERROR]
> testParallelRename(org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps)
> Time elapsed: 72.565 s <<< ERROR!
> org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: Writing Object on
> fork-0005/test/testParallelRename-source0:
> software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client
> execution did not complete before the specified timeout configuration: 15000
> millis
> at
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:215)
> at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:124)
> at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:376)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:372)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:347)
> at
> org.apache.hadoop.fs.s3a.WriteOperationHelper.retry(WriteOperationHelper.java:214)
> at
> org.apache.hadoop.fs.s3a.WriteOperationHelper.putObject(WriteOperationHelper.java:532)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.lambda$putObject$0(S3ABlockOutputStream.java:620)
> at
> org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
> at
> org.apache.hadoop.thirdparty.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
> at
> org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
> at
> org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:225)
> at
> org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:225)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: software.amazon.awssdk.core.exception.ApiCallTimeoutException:
> Client execution did not complete before the specified timeout configuration:
> 15000 millis
> at
> software.amazon.awssdk.core.exception.ApiCallTimeoutException$BuilderImpl.build(ApiCallTimeoutException.java:97)
> at
> software.amazon.awssdk.core.exception.ApiCallTimeoutException.create(ApiCallTimeoutException.java:38)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.generateApiCallTimeoutException(ApiCallTimeoutTrackingStage.java:151)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.handleInterruptedException(ApiCallTimeoutTrackingStage.java:139)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.translatePipelineException(ApiCallTimeoutTrackingStage.java:107)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
> at
> software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
> at
> software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineS
[jira] [Updated] (HADOOP-19023) ITestS3AConcurrentOps#testParallelRename intermittent timeout failure
[
https://issues.apache.org/jira/browse/HADOOP-19023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-19023:
Component/s: fs/s3
> ITestS3AConcurrentOps#testParallelRename intermittent timeout failure
> -
>
> Key: HADOOP-19023
> URL: https://issues.apache.org/jira/browse/HADOOP-19023
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Viraj Jasani
>Priority: Major
>
> Need to configure higher timeout for the test.
>
> {code:java}
> [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 256.281 s <<< FAILURE! - in
> org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps
> [ERROR]
> testParallelRename(org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps)
> Time elapsed: 72.565 s <<< ERROR!
> org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: Writing Object on
> fork-0005/test/testParallelRename-source0:
> software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client
> execution did not complete before the specified timeout configuration: 15000
> millis
> at
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:215)
> at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:124)
> at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:376)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:372)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:347)
> at
> org.apache.hadoop.fs.s3a.WriteOperationHelper.retry(WriteOperationHelper.java:214)
> at
> org.apache.hadoop.fs.s3a.WriteOperationHelper.putObject(WriteOperationHelper.java:532)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.lambda$putObject$0(S3ABlockOutputStream.java:620)
> at
> org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
> at
> org.apache.hadoop.thirdparty.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
> at
> org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
> at
> org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:225)
> at
> org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:225)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: software.amazon.awssdk.core.exception.ApiCallTimeoutException:
> Client execution did not complete before the specified timeout configuration:
> 15000 millis
> at
> software.amazon.awssdk.core.exception.ApiCallTimeoutException$BuilderImpl.build(ApiCallTimeoutException.java:97)
> at
> software.amazon.awssdk.core.exception.ApiCallTimeoutException.create(ApiCallTimeoutException.java:38)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.generateApiCallTimeoutException(ApiCallTimeoutTrackingStage.java:151)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.handleInterruptedException(ApiCallTimeoutTrackingStage.java:139)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.translatePipelineException(ApiCallTimeoutTrackingStage.java:107)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
> at
> software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
> at
> software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExcepti
[jira] [Updated] (HADOOP-19023) ITestS3AConcurrentOps#testParallelRename intermittent timeout failure
[
https://issues.apache.org/jira/browse/HADOOP-19023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-19023:
Affects Version/s: 3.4.0
> ITestS3AConcurrentOps#testParallelRename intermittent timeout failure
> -
>
> Key: HADOOP-19023
> URL: https://issues.apache.org/jira/browse/HADOOP-19023
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Viraj Jasani
>Priority: Major
>
> Need to configure higher timeout for the test.
>
> {code:java}
> [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 256.281 s <<< FAILURE! - in
> org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps
> [ERROR]
> testParallelRename(org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps)
> Time elapsed: 72.565 s <<< ERROR!
> org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: Writing Object on
> fork-0005/test/testParallelRename-source0:
> software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client
> execution did not complete before the specified timeout configuration: 15000
> millis
> at
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:215)
> at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:124)
> at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:376)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:372)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:347)
> at
> org.apache.hadoop.fs.s3a.WriteOperationHelper.retry(WriteOperationHelper.java:214)
> at
> org.apache.hadoop.fs.s3a.WriteOperationHelper.putObject(WriteOperationHelper.java:532)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.lambda$putObject$0(S3ABlockOutputStream.java:620)
> at
> org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
> at
> org.apache.hadoop.thirdparty.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
> at
> org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
> at
> org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:225)
> at
> org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:225)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: software.amazon.awssdk.core.exception.ApiCallTimeoutException:
> Client execution did not complete before the specified timeout configuration:
> 15000 millis
> at
> software.amazon.awssdk.core.exception.ApiCallTimeoutException$BuilderImpl.build(ApiCallTimeoutException.java:97)
> at
> software.amazon.awssdk.core.exception.ApiCallTimeoutException.create(ApiCallTimeoutException.java:38)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.generateApiCallTimeoutException(ApiCallTimeoutTrackingStage.java:151)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.handleInterruptedException(ApiCallTimeoutTrackingStage.java:139)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.translatePipelineException(ApiCallTimeoutTrackingStage.java:107)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
> at
> software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
> at
> software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
> at
> software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:3
[jira] [Updated] (HADOOP-19022) ITestS3AConfiguration#testRequestTimeout failure
[
https://issues.apache.org/jira/browse/HADOOP-19022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-19022:
Component/s: fs/s3
test
> ITestS3AConfiguration#testRequestTimeout failure
>
>
> Key: HADOOP-19022
> URL: https://issues.apache.org/jira/browse/HADOOP-19022
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.4.0
>Reporter: Viraj Jasani
>Priority: Minor
>
> "fs.s3a.connection.request.timeout" should be specified in milliseconds as per
> {code:java}
> Duration apiCallTimeout = getDuration(conf, REQUEST_TIMEOUT,
> DEFAULT_REQUEST_TIMEOUT_DURATION, TimeUnit.MILLISECONDS, Duration.ZERO);
> {code}
> The test fails consistently because it sets 120 ms timeout which is less than
> 15s (min network operation duration), and hence gets reset to 15000 ms based
> on the enforcement.
>
> {code:java}
> [ERROR] testRequestTimeout(org.apache.hadoop.fs.s3a.ITestS3AConfiguration)
> Time elapsed: 0.016 s <<< FAILURE!
> java.lang.AssertionError: Configured fs.s3a.connection.request.timeout is
> different than what AWS sdk configuration uses internally expected:<12>
> but was:<15000>
> at org.junit.Assert.fail(Assert.java:89)
> at org.junit.Assert.failNotEquals(Assert.java:835)
> at org.junit.Assert.assertEquals(Assert.java:647)
> at
> org.apache.hadoop.fs.s3a.ITestS3AConfiguration.testRequestTimeout(ITestS3AConfiguration.java:444)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-19022) ITestS3AConfiguration#testRequestTimeout failure
[
https://issues.apache.org/jira/browse/HADOOP-19022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803520#comment-17803520
]
Steve Loughran edited comment on HADOOP-19022 at 1/5/24 11:51 AM:
--
should be a string now, e.g "20s".
have you explicitly set it in your site config?
was (Author: ste...@apache.org):
should be a string now, e.g "20s"
> ITestS3AConfiguration#testRequestTimeout failure
>
>
> Key: HADOOP-19022
> URL: https://issues.apache.org/jira/browse/HADOOP-19022
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.4.0
>Reporter: Viraj Jasani
>Priority: Minor
>
> "fs.s3a.connection.request.timeout" should be specified in milliseconds as per
> {code:java}
> Duration apiCallTimeout = getDuration(conf, REQUEST_TIMEOUT,
> DEFAULT_REQUEST_TIMEOUT_DURATION, TimeUnit.MILLISECONDS, Duration.ZERO);
> {code}
> The test fails consistently because it sets 120 ms timeout which is less than
> 15s (min network operation duration), and hence gets reset to 15000 ms based
> on the enforcement.
>
> {code:java}
> [ERROR] testRequestTimeout(org.apache.hadoop.fs.s3a.ITestS3AConfiguration)
> Time elapsed: 0.016 s <<< FAILURE!
> java.lang.AssertionError: Configured fs.s3a.connection.request.timeout is
> different than what AWS sdk configuration uses internally expected:<12>
> but was:<15000>
> at org.junit.Assert.fail(Assert.java:89)
> at org.junit.Assert.failNotEquals(Assert.java:835)
> at org.junit.Assert.assertEquals(Assert.java:647)
> at
> org.apache.hadoop.fs.s3a.ITestS3AConfiguration.testRequestTimeout(ITestS3AConfiguration.java:444)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19022) ITestS3AConfiguration#testRequestTimeout failure
[
https://issues.apache.org/jira/browse/HADOOP-19022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-19022:
Affects Version/s: 3.4.0
> ITestS3AConfiguration#testRequestTimeout failure
>
>
> Key: HADOOP-19022
> URL: https://issues.apache.org/jira/browse/HADOOP-19022
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Viraj Jasani
>Priority: Minor
>
> "fs.s3a.connection.request.timeout" should be specified in milliseconds as per
> {code:java}
> Duration apiCallTimeout = getDuration(conf, REQUEST_TIMEOUT,
> DEFAULT_REQUEST_TIMEOUT_DURATION, TimeUnit.MILLISECONDS, Duration.ZERO);
> {code}
> The test fails consistently because it sets 120 ms timeout which is less than
> 15s (min network operation duration), and hence gets reset to 15000 ms based
> on the enforcement.
>
> {code:java}
> [ERROR] testRequestTimeout(org.apache.hadoop.fs.s3a.ITestS3AConfiguration)
> Time elapsed: 0.016 s <<< FAILURE!
> java.lang.AssertionError: Configured fs.s3a.connection.request.timeout is
> different than what AWS sdk configuration uses internally expected:<12>
> but was:<15000>
> at org.junit.Assert.fail(Assert.java:89)
> at org.junit.Assert.failNotEquals(Assert.java:835)
> at org.junit.Assert.assertEquals(Assert.java:647)
> at
> org.apache.hadoop.fs.s3a.ITestS3AConfiguration.testRequestTimeout(ITestS3AConfiguration.java:444)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19022) ITestS3AConfiguration#testRequestTimeout failure
[
https://issues.apache.org/jira/browse/HADOOP-19022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803520#comment-17803520
]
Steve Loughran commented on HADOOP-19022:
-
should be a string now, e.g "20s"
> ITestS3AConfiguration#testRequestTimeout failure
>
>
> Key: HADOOP-19022
> URL: https://issues.apache.org/jira/browse/HADOOP-19022
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Viraj Jasani
>Priority: Minor
>
> "fs.s3a.connection.request.timeout" should be specified in milliseconds as per
> {code:java}
> Duration apiCallTimeout = getDuration(conf, REQUEST_TIMEOUT,
> DEFAULT_REQUEST_TIMEOUT_DURATION, TimeUnit.MILLISECONDS, Duration.ZERO);
> {code}
> The test fails consistently because it sets 120 ms timeout which is less than
> 15s (min network operation duration), and hence gets reset to 15000 ms based
> on the enforcement.
>
> {code:java}
> [ERROR] testRequestTimeout(org.apache.hadoop.fs.s3a.ITestS3AConfiguration)
> Time elapsed: 0.016 s <<< FAILURE!
> java.lang.AssertionError: Configured fs.s3a.connection.request.timeout is
> different than what AWS sdk configuration uses internally expected:<12>
> but was:<15000>
> at org.junit.Assert.fail(Assert.java:89)
> at org.junit.Assert.failNotEquals(Assert.java:835)
> at org.junit.Assert.assertEquals(Assert.java:647)
> at
> org.apache.hadoop.fs.s3a.ITestS3AConfiguration.testRequestTimeout(ITestS3AConfiguration.java:444)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19025) Migrate ContractTestUtils to AssertJ
[
https://issues.apache.org/jira/browse/HADOOP-19025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803519#comment-17803519
]
Steve Loughran commented on HADOOP-19025:
-
looking forward to this!
> Migrate ContractTestUtils to AssertJ
>
>
> Key: HADOOP-19025
> URL: https://issues.apache.org/jira/browse/HADOOP-19025
> Project: Hadoop Common
> Issue Type: Improvement
> Components: test
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
>
> Replace assertions from JUnit4 with equivalent functionality from AssertJ, to
> make {{ContractTestUtils}} independent of JUnit version.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19026) S3A: TestIAMInstanceCredentialsProvider.testIAMInstanceCredentialsInstantiate failure
[
https://issues.apache.org/jira/browse/HADOOP-19026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803518#comment-17803518
]
Steve Loughran commented on HADOOP-19026:
-
{code}
java.lang.AssertionError: Cause not a IOException
at
org.apache.hadoop.fs.s3a.auth.TestIAMInstanceCredentialsProvider.__CLR4_4_1ey8o37g7b(TestIAMInstanceCredentialsProvider.java:100)
at
org.apache.hadoop.fs.s3a.auth.TestIAMInstanceCredentialsProvider.testIAMInstanceCredentialsInstantiate(TestIAMInstanceCredentialsProvider.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Failed to
load credentials from IMDS.
at
software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at
software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
at
software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.refreshCredentials(InstanceProfileCredentialsProvider.java:159)
at
software.amazon.awssdk.utils.cache.CachedSupplier.lambda$jitteredPrefetchValueSupplier$8(CachedSupplier.java:300)
at
software.amazon.awssdk.utils.cache.NonBlocking.fetch(NonBlocking.java:141)
at
software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:208)
at
software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:135)
at
software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.resolveCredentials(InstanceProfileCredentialsProvider.java:141)
at
org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider.getCredentials(IAMInstanceCredentialsProvider.java:135)
at
org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider.resolveCredentials(IAMInstanceCredentialsProvider.java:98)
at
org.apache.hadoop.fs.s3a.auth.TestIAMInstanceCredentialsProvider.__CLR4_4_1ey8o37g7b(TestIAMInstanceCredentialsProvider.java:75)
... 14 more
Caused by: software.amazon.awssdk.core.exception.SdkClientException: The
requested metadata is not found
at http://169.254.169.254/latest/meta-data/iam/security-credentials/
at
software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at
software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:125)
at
software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:91)
at
software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.lambda$getSecurityCredentials$3(InstanceProfileCredentialsProvider.java:256)
at
software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4(FunctionalUtils.java:108)
at
software.amazon.awssdk.utils.FunctionalUtils.invokeSafely(FunctionalUtils.java:136)
at
software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.getSecurityCredentials(InstanceProfileCredentialsProvider.java:256)
at
software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.createEndpointProvider(InstanceProfileCredentialsProvider.java:204)
at
software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.refreshCredentials(InstanceProfileCredentialsProvider.java:150)
... 22 more
{code}
> S3A: TestIAMInstanceCredentialsProvider.testIAMInstanceCredentialsInstantiate
> failure
> -
>
> Key: HADOOP-19026
> URL: https://issues.apache.org/jira/browse/HADOOP-19026
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.4.0
>
[jira] [Created] (HADOOP-19026) S3A: TestIAMInstanceCredentialsProvider.testIAMInstanceCredentialsInstantiate failure
Steve Loughran created HADOOP-19026:
---
Summary: S3A:
TestIAMInstanceCredentialsProvider.testIAMInstanceCredentialsInstantiate failure
Key: HADOOP-19026
URL: https://issues.apache.org/jira/browse/HADOOP-19026
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3, test
Affects Versions: 3.4.0
Reporter: Steve Loughran
test failure in TestIAMInstanceCredentialsProvider; looks like the test is
running in an EC2 VM whose IAM service isn't providing credentials -and the
test isn't set up to ignore that.
{code}
Caused by: software.amazon.awssdk.core.exception.SdkClientException: The
requested metadata is not found
at http://169.254.169.254/latest/meta-data/iam/security-credentials/
at
software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at
software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:125)
at
software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:91)
at
software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.lambda$getSecurityCredentials$3(InstanceProfileCredentialsProvider.java:256)
at
software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4(FunctionalUtils.java:108)
at
software.amazon.awssdk.utils.FunctionalUtils.invokeSafely(FunctionalUtils.java:136)
at
software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.getSecurityCredentials(InstanceProfileCredentialsProvider.java:256)
at
software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.createEndpointProvider(InstanceProfileCredentialsProvider.java:204)
at
software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.refreshCredentials(InstanceProfileCredentialsProvider.java:150)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17912) ABFS: Support for Encryption Context
[ https://issues.apache.org/jira/browse/HADOOP-17912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17912: Fix Version/s: 3.3.9 > ABFS: Support for Encryption Context > > > Key: HADOOP-17912 > URL: https://issues.apache.org/jira/browse/HADOOP-17912 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Pranav Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > Time Spent: 1h > Remaining Estimate: 0h > > Support for customer-provided encryption keys at the file level, superceding > the global (account-level) key use in HADOOP-17536. > ABFS driver will support an "EncryptionContext" plugin for retrieving > encryption information, the implementation for which should be provided by > the client. The keys/context retrieved will be sent via request headers to > the server, which will store the encryption context. Subsequent REST calls to > server that access data/user metadata of the file will require fetching the > encryption context through a GetFileProperties call and retrieving the key > from the custom provider, before sending the request. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19021) [ABFS] move to jdk11 HttpClient for http2 and connection keep alive
[ https://issues.apache.org/jira/browse/HADOOP-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-19021: Summary: [ABFS] move to jdk11 HttpClient for http2 and connection keep alive (was: in hadoop-azure, use jdk11 HttpClient instead of legacy java.net.HttpURLConnection, for supporting http2 and connection keep alive) > [ABFS] move to jdk11 HttpClient for http2 and connection keep alive > --- > > Key: HADOOP-19021 > URL: https://issues.apache.org/jira/browse/HADOOP-19021 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: Arnaud Nauwynck >Priority: Critical > > As described in Jira Title: "in hadoop-azure, use jdk11 HttpClient instead of > legacy java.net.HttpURLConnection, for supporting http2 and connection keep > alive" > Few remarks: > 1/ The official Azure SDK supports either OkHttp or Netty for the Http > transport. > 2/ the actual hadoop-azure use the class java.net.HttpURLConnection, which is > slow. > It does not use Http2, does not optimize SSL Hand-shake very well, and does > not keep TCP connection alive for re-use. > 3/ JDK since version >=11 have a new class HttpClient which should be a > better replacement > 4/ it might be possible to introduce a configuration property (with defaut to > use legacy class) , and an abstract factory to create connection via either > HttpURLConnection or any other pluggeable implementation (jdk 11 HttpClient, > OkHttp, Netty, ...) > 5/ the official Azure SDK is maintained by Microsoft, so should better follow > bug fixes and improvements than custom hadoop implementation? > [https://learn.microsoft.com/en-us/java/api/overview/azure/storage-file-datalake-readme?view=azure-java-stable > |https://learn.microsoft.com/en-us/java/api/overview/azure/storage-file-datalake-readme?view=azure-java-stable] > 6/ when we use code with the official Azure SDK and Hadoop(in Spark), it is > chocking to have 2 different implementations within the same JVM... > 7/ The official Azure SDK has more features that what allows the legacy > hadoop class FileSystem to do... In particular, we can append (=upload) file > by multiple threads (upload by fragments at different offsets), then flush > when every fragments are sent. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18991) Remove commons-beanutils dependency from Hadoop 3
[ https://issues.apache.org/jira/browse/HADOOP-18991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17801756#comment-17801756 ] Steve Loughran commented on HADOOP-18991: - so leave in branch-3.3 and noted as optional; cut from trunk? > Remove commons-beanutils dependency from Hadoop 3 > - > > Key: HADOOP-18991 > URL: https://issues.apache.org/jira/browse/HADOOP-18991 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Istvan Toth >Priority: Major > > Hadoop doesn't acually use it, and it pollutes the classpath of dependent > projects. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19017) Setup pre-commit CI for Windows 10
[ https://issues.apache.org/jira/browse/HADOOP-19017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19017. - Fix Version/s: 3.4.0 Resolution: Fixed > Setup pre-commit CI for Windows 10 > -- > > Key: HADOOP-19017 > URL: https://issues.apache.org/jira/browse/HADOOP-19017 > Project: Hadoop Common > Issue Type: New Feature > Components: build >Affects Versions: 3.4.0 > Environment: Windows 10 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Critical > Labels: Jenkins, pull-request-available > Fix For: 3.4.0 > > > We need to setup a pre-commit CI for validating the Hadoop PRs against > Windows 10. > On a sidenote, we've got the nightly Jenkins CI running for Hadoop on Windows > 10 - > https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-win10-x86_64/. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18991) Remove commons-beanutils dependency from Hadoop 3
[ https://issues.apache.org/jira/browse/HADOOP-18991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17801599#comment-17801599 ] Steve Loughran commented on HADOOP-18991: - hive should add explicit dependencies here > Remove commons-beanutils dependency from Hadoop 3 > - > > Key: HADOOP-18991 > URL: https://issues.apache.org/jira/browse/HADOOP-18991 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Istvan Toth >Priority: Major > > Hadoop doesn't acually use it, and it pollutes the classpath of dependent > projects. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17347) ABFS: read/cache footer with fs.azure.footer.read.request.size
[ https://issues.apache.org/jira/browse/HADOOP-17347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17347: Summary: ABFS: read/cache footer with fs.azure.footer.read.request.size (was: ABFS: Optimise read for small files/tails of files) > ABFS: read/cache footer with fs.azure.footer.read.request.size > -- > > Key: HADOOP-17347 > URL: https://issues.apache.org/jira/browse/HADOOP-17347 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.4.0 >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 12h 50m > Remaining Estimate: 0h > > Optimize read performance for the following scenarios > # Read small files completely > Files that are of size smaller than the read buffer size can be considered > as small files. In case of such files it would be better to read the full > file into the AbfsInputStream buffer. > # Read last block if the read is for footer > If the read is for the last 8 bytes, read the full file. > This will optimize reads for parquet files. [Parquet file > format|https://www.ellicium.com/parquet-file-format-structure/] > Both these optimizations will be present under configs as follows > # fs.azure.read.smallfilescompletely > # fs.azure.read.optimizefooterread -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17912) ABFS: Support for Encryption Context
[ https://issues.apache.org/jira/browse/HADOOP-17912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17912. - Fix Version/s: 3.4.0 Resolution: Fixed > ABFS: Support for Encryption Context > > > Key: HADOOP-17912 > URL: https://issues.apache.org/jira/browse/HADOOP-17912 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sumangala Patki >Assignee: Pranav Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Support for customer-provided encryption keys at the file level, superceding > the global (account-level) key use in HADOOP-17536. > ABFS driver will support an "EncryptionContext" plugin for retrieving > encryption information, the implementation for which should be provided by > the client. The keys/context retrieved will be sent via request headers to > the server, which will store the encryption context. Subsequent REST calls to > server that access data/user metadata of the file will require fetching the > encryption context through a GetFileProperties call and retrieving the key > from the custom provider, before sending the request. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18540) Upgrade Bouncy Castle to 1.70
[
https://issues.apache.org/jira/browse/HADOOP-18540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-18540.
-
Fix Version/s: 3.4.0
Resolution: Fixed
> Upgrade Bouncy Castle to 1.70
> -
>
> Key: HADOOP-18540
> URL: https://issues.apache.org/jira/browse/HADOOP-18540
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.4.0
>Reporter: D M Murali Krishna Reddy
>Assignee: D M Murali Krishna Reddy
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Upgrade Bouncycastle to 1.70 to resolve
>
> |[[sonatype-2021-4916] CWE-327: Use of a Broken or Risky Cryptographic
> Algorithm|https://ossindex.sonatype.org/vulnerability/sonatype-2021-4916?component-type=maven&component-name=org.bouncycastle/bcprov-jdk15on]|
> |[[sonatype-2019-0673] CWE-400: Uncontrolled Resource Consumption ('Resource
> Exhaustion')|https://ossindex.sonatype.org/vulnerability/sonatype-2019-0673?component-type=maven&component-name=org.bouncycastle/bcprov-jdk15on]|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19013) fs.getXattrs(path) for S3FS doesn't have x-amz-server-side-encryption-aws-kms-key-id header.
[ https://issues.apache.org/jira/browse/HADOOP-19013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17801542#comment-17801542 ] Steve Loughran commented on HADOOP-19013: - this a standard header? or do you just think it'd be useful? Either way, I support it -just make sure COPY does the right thing and sets the new header value > fs.getXattrs(path) for S3FS doesn't have > x-amz-server-side-encryption-aws-kms-key-id header. > > > Key: HADOOP-19013 > URL: https://issues.apache.org/jira/browse/HADOOP-19013 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.6 >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Major > > Once a path while uploading has been encrypted with SSE-KMS with a key id and > then later when we try to read the attributes of the same file, it doesn't > contain the key id information as an attribute. should we add it? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-19008) S3A: Upgrade AWS SDK to 2.21.41
[ https://issues.apache.org/jira/browse/HADOOP-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19008. - Fix Version/s: 3.4.0 Resolution: Fixed > S3A: Upgrade AWS SDK to 2.21.41 > --- > > Key: HADOOP-19008 > URL: https://issues.apache.org/jira/browse/HADOOP-19008 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0, 3.3.7-aws >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > sdk 2.21.41 is out and logging now picks up the log4j.properties options. > move to this ASAP -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19008) S3A: Upgrade AWS SDK to 2.21.41
Steve Loughran created HADOOP-19008: --- Summary: S3A: Upgrade AWS SDK to 2.21.41 Key: HADOOP-19008 URL: https://issues.apache.org/jira/browse/HADOOP-19008 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.4.0, 3.3.7-aws Reporter: Steve Loughran Assignee: Steve Loughran sdk 2.21.41 is out and logging now picks up the log4j.properties options. move to this ASAP -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18999) S3A: debug logging for http traffic to S3 stores
[
https://issues.apache.org/jira/browse/HADOOP-18999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-18999.
-
Resolution: Not A Problem
no longer needed as 2.21.41 logs properly!
> S3A: debug logging for http traffic to S3 stores
>
>
> Key: HADOOP-18999
> URL: https://issues.apache.org/jira/browse/HADOOP-18999
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> AWS SDK bundle.jar logging doesn't set up right.
> {code}
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> {code}
> Cloudstore commands have a -debug option to force set this through log4j
> APIs; this does work.
> Proposed:
> * add reflection-based ability to set/query log4j log levels (+tests,
> obviously)
> * add a new log `org.apache.hadoop.fs.s3a.logging.sdk`
> * if set to DEBUG, DefaultS3ClientFactory will enable logging on the aws
> internal/shaded classes
> this allows log4j.properties to turn on logging; reflection ensures all is
> well on other log back-ends and when unshaded aws sdk jars are in use
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19007) S3A: transfer manager not wired up to s3a executor pool
[ https://issues.apache.org/jira/browse/HADOOP-19007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17794656#comment-17794656 ] Steve Loughran commented on HADOOP-19007: - [~ahmar] any specific reason why this isn't done? and what happens without it? > S3A: transfer manager not wired up to s3a executor pool > --- > > Key: HADOOP-19007 > URL: https://issues.apache.org/jira/browse/HADOOP-19007 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Priority: Major > > S3ClientFactory.createS3TransferManager() doesn't use the executor declared > in S3ClientCreationParameters.transferManagerExecutor > * method needs to take S3ClientCreationParameters > * and set the transfer manager executor -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19007) S3A: transfer manager not wired up to s3a executor pool
Steve Loughran created HADOOP-19007: --- Summary: S3A: transfer manager not wired up to s3a executor pool Key: HADOOP-19007 URL: https://issues.apache.org/jira/browse/HADOOP-19007 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran S3ClientFactory.createS3TransferManager() doesn't use the executor declared in S3ClientCreationParameters.transferManagerExecutor * method needs to take S3ClientCreationParameters * and set the transfer manager executor -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18915) Tune/extend S3A http connection and thread pool settings
[ https://issues.apache.org/jira/browse/HADOOP-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17794302#comment-17794302 ] Steve Loughran commented on HADOOP-18915: - looking at the AWS SDK source, it looks like the s3express createsession code doesn't pick up the expanded settings as the timeout is fixed at 10s. What does that mean? It means if you are working with S3Express buckets *you had better have a large http connection pool and no connectivity issues* we may want to think about to test/handle this best in our own code. We should be retrying. > Tune/extend S3A http connection and thread pool settings > > > Key: HADOOP-18915 > URL: https://issues.apache.org/jira/browse/HADOOP-18915 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.7-aws > > > Increases existing pool sizes, as with server scale and vector > IO, larger pools are needed > fs.s3a.connection.maximum 200 > fs.s3a.threads.max 96 > Adds new configuration options for v2 sdk internal timeouts, > both with default of 60s: > fs.s3a.connection.acquisition.timeout > fs.s3a.connection.idle.time > All the pool/timoeut options are covered in performance.md > Moves all timeout/duration options in the s3a FS to taking > temporal units (h, m, s, ms,...); retaining the previous default > unit (normally millisecond) > Adds a minimum duration for most of these, in order to recover from > deployments where a timeout has been set on the assumption the unit > was seconds, not millis. > Uses java.time.Duration throughout the codebase; > retaining the older numeric constants in > org.apache.hadoop.fs.s3a.Constants for backwards compatibility; > these are now deprecated. > Adds new class AWSApiCallTimeoutException to be raised on > sdk-related methods and also gateway timeouts. This is a subclass > of org.apache.hadoop.net.ConnectTimeoutException to support > existing retry logic. > + reverted default value of fs.s3a.create.performance to false; > inadvertently set to true during testing. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18540) Upgrade Bouncy Castle to 1.70
[
https://issues.apache.org/jira/browse/HADOOP-18540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17794283#comment-17794283
]
Steve Loughran commented on HADOOP-18540:
-
[~BilwaST] ok, lets get that PR rebased for a retest and then merged. in 27h go
offline until 2024 so other people will have to merge
> Upgrade Bouncy Castle to 1.70
> -
>
> Key: HADOOP-18540
> URL: https://issues.apache.org/jira/browse/HADOOP-18540
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.4.0
>Reporter: D M Murali Krishna Reddy
>Assignee: D M Murali Krishna Reddy
>Priority: Major
> Labels: pull-request-available
>
> Upgrade Bouncycastle to 1.70 to resolve
>
> |[[sonatype-2021-4916] CWE-327: Use of a Broken or Risky Cryptographic
> Algorithm|https://ossindex.sonatype.org/vulnerability/sonatype-2021-4916?component-type=maven&component-name=org.bouncycastle/bcprov-jdk15on]|
> |[[sonatype-2019-0673] CWE-400: Uncontrolled Resource Consumption ('Resource
> Exhaustion')|https://ossindex.sonatype.org/vulnerability/sonatype-2019-0673?component-type=maven&component-name=org.bouncycastle/bcprov-jdk15on]|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18925) S3A: add option "fs.s3a.copy.from.local.enabled" to enable/disable CopyFromLocalOperation
[ https://issues.apache.org/jira/browse/HADOOP-18925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18925. - Fix Version/s: 3.4.0 3.3.9 Resolution: Fixed > S3A: add option "fs.s3a.copy.from.local.enabled" to enable/disable > CopyFromLocalOperation > - > > Key: HADOOP-18925 > URL: https://issues.apache.org/jira/browse/HADOOP-18925 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.6 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > reported failure of CopyFromLocalOperation.getFinalPath() during job > submission with s3a declared as cluster fs. > add an emergency option to disable this optimised uploader and revert to the > superclass implementation -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18997) S3A: Add option fs.s3a.s3express.create.session to enable/disable CreateSession
[ https://issues.apache.org/jira/browse/HADOOP-18997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18997. - Fix Version/s: 3.4.0 Resolution: Fixed > S3A: Add option fs.s3a.s3express.create.session to enable/disable > CreateSession > --- > > Key: HADOOP-18997 > URL: https://issues.apache.org/jira/browse/HADOOP-18997 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > add a way to disable the need to use the createsession call, so as to allow > for > * simplifying our role test runs > * benchmarking the performance hit > * troubleshooting IAM permissions > this can also be disabled from the sysprop "aws.disableS3ExpressAuth" -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19004) S3A: Support Authentication through HttpSigner API
[
https://issues.apache.org/jira/browse/HADOOP-19004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-19004:
Summary: S3A: Support Authentication through HttpSigner API (was: S3A:
Move to a new HttpSigner for S3Express)
> S3A: Support Authentication through HttpSigner API
> ---
>
> Key: HADOOP-19004
> URL: https://issues.apache.org/jira/browse/HADOOP-19004
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Harshit Gupta
>Priority: Major
> Labels: pull-request-available
>
> The latest AWS SDK changes how signing works, and for signing S3Express
> signatures the new {{software.amazon.awssdk.http.auth}} auth mechanism is
> needed
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19004) S3A: Move to a new HttpSigner for S3Express
Steve Loughran created HADOOP-19004:
---
Summary: S3A: Move to a new HttpSigner for S3Express
Key: HADOOP-19004
URL: https://issues.apache.org/jira/browse/HADOOP-19004
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.4.0
Reporter: Steve Loughran
Assignee: Harshit Gupta
The latest AWS SDK changes how signing works, and for signing S3Express
signatures the new {{software.amazon.awssdk.http.auth}} auth mechanism is needed
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18955) AWS SDK v2: add path capability probe "fs.s3a.capability.aws.v2"
[ https://issues.apache.org/jira/browse/HADOOP-18955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18955. - Fix Version/s: 3.4.0 Resolution: Fixed Fixed in HADOOP-18996 > AWS SDK v2: add path capability probe "fs.s3a.capability.aws.v2" > > > Key: HADOOP-18955 > URL: https://issues.apache.org/jira/browse/HADOOP-18955 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 3.4.0 > > > Add a "hasPathCapability()" probe for s3a v2 builds to aid diagnostics > -avoids needing to look for specific s3a files on classpath. > plus bucket-info to enum all capabilities which may be present -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18961) S3A: add s3guard command "bucket" to create buckets
[ https://issues.apache.org/jira/browse/HADOOP-18961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18961. - Fix Version/s: 3.4.0 Resolution: Fixed Fixed in HADOOP-18996 > S3A: add s3guard command "bucket" to create buckets > --- > > Key: HADOOP-18961 > URL: https://issues.apache.org/jira/browse/HADOOP-18961 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.4.0 > > > cloudstore has an mkbucket command > https://github.com/steveloughran/cloudstore/blob/main/src/main/site/mkbucket.md > however, its v1 api, has problems with spans and needs rework for v2. Oh, and > it has not tests. > * move the command into hadoop-aws > * add a test or two, > * add a mapper of 409 to bad request, as "BucketAlreadyOwnedByYouException" > comes in as a 409. > Testing will be tricky...as well as not actually wanting to create new > buckets, my test a/c doesn't even have the permission to do so. > Proposed tests > # invalid args must be rejected > # trying to create the current bucket must be rejected. This happens even if > you lack the permission to create > *No actual attempts to create a new bucket* -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18997) S3A: Add option fs.s3a.s3express.create.session to enable/disable CreateSession
[ https://issues.apache.org/jira/browse/HADOOP-18997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18997: Summary: S3A: Add option fs.s3a.s3express.create.session to enable/disable CreateSession (was: S3A: add way to turn off CreateSession API) > S3A: Add option fs.s3a.s3express.create.session to enable/disable > CreateSession > --- > > Key: HADOOP-18997 > URL: https://issues.apache.org/jira/browse/HADOOP-18997 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: pull-request-available > > add a way to disable the need to use the createsession call, so as to allow > for > * simplifying our role test runs > * benchmarking the performance hit > * troubleshooting IAM permissions > this can also be disabled from the sysprop "aws.disableS3ExpressAuth" -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18708) AWS SDK V2 - Implement CSE
[ https://issues.apache.org/jira/browse/HADOOP-18708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-18708: --- Assignee: Ahmar Suhail > AWS SDK V2 - Implement CSE > -- > > Key: HADOOP-18708 > URL: https://issues.apache.org/jira/browse/HADOOP-18708 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Ahmar Suhail >Priority: Major > Labels: pull-request-available > > S3 Encryption client for SDK V2 is now available, so add client side > encryption back in. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19003) S3A Assume role tests failing against S3Express stores
[
https://issues.apache.org/jira/browse/HADOOP-19003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793310#comment-17793310
]
Steve Loughran commented on HADOOP-19003:
-
{code}
[ERROR] ITestAssumeRole.testPartialDelete:696->executePartialDelete:732 »
AccessDenied
[ERROR]
ITestAssumeRole.testPartialDeleteSingleDelete:702->executePartialDelete:732 »
AccessDenied
[ERROR] ITestAssumeRole.testReadOnlyOperations:463 » AccessDenied
s3a://stevel--usw2-a...
[ERROR] ITestAssumeRole.testRestrictedCommitActions:626 » AccessDenied
s3a://stevel--u...
[ERROR] ITestAssumeRole.testRestrictedWriteSubdir:513 » AccessDenied
s3a://stevel--usw...
[ERROR]
ITestAssumedRoleCommitOperations>ITestCommitOperations.testAbortNonexistentDir:234
» AccessDenied
[ERROR]
ITestAssumedRoleCommitOperations>ITestCommitOperations.testBaseRelativePath:282
» AccessDenied
[ERROR]
ITestAssumedRoleCommitOperations>ITestCommitOperations.testBulkCommitFiles:602
» AccessDenied
[ERROR]
ITestAssumedRoleCommitOperations>ITestCommitOperations.testCommitEmptyFile:217->ITestCommitOperations.createCommitAndVerify:346
» AccessDenied
[ERROR]
ITestAssumedRoleCommitOperations>ITestCommitOperations.testCommitSmallFile:223->ITestCommitOperations.createCommitAndVerify:346
» AccessDenied
[ERROR]
ITestAssumedRoleCommitOperations>ITestCommitOperations.testCreateAbortEmptyFile:172
» AccessDenied
[ERROR]
ITestAssumedRoleCommitOperations>ITestCommitOperations.testFailuresInAbortListing:566
» AccessDenied
[ERROR]
ITestAssumedRoleCommitOperations>ITestCommitOperations.testMarkerFileRename:304
» AccessDenied
[ERROR]
ITestAssumedRoleCommitOperations>ITestCommitOperations.testRevertCommit:544 »
AccessDenied
[ERROR]
ITestAssumedRoleCommitOperations>ITestCommitOperations.testRevertMissingCommit:556
» AccessDenied
[ERROR]
ITestAssumedRoleCommitOperations>ITestCommitOperations.testUploadEmptyFile:472
» AccessDenied
[ERROR]
ITestAssumedRoleCommitOperations>ITestCommitOperations.testUploadSmallFile:503
» AccessDenied
[ERROR]
ITestAssumedRoleCommitOperations>ITestCommitOperations.testWriteNormalStream:581
» AccessDenied
[ERROR]
ITestRestrictedReadAccess.testNoReadAccess:211->checkBasicFileOperations:283 »
AccessDenied
[ERROR]
ITestRoleDelegationInFilesystem>ITestSessionDelegationInFilesystem.testDelegatedFileSystem:406->ITestSessionDelegationInFilesystem.executeDelegatedFSOperations:454
» AccessDenied
[ERROR] ITestPartialRenamesDeletes.testCopyDirFailsToReadOnlyDir:519 »
AccessDenied s3...
[ERROR] ITestPartialRenamesDeletes.testCopyDirFailsToReadOnlyDir:519 »
AccessDenied s3...
[ERROR] ITestPartialRenamesDeletes.testPartialDirDelete:607 » AccessDenied
s3a://steve...
[ERROR] ITestPartialRenamesDeletes.testPartialDirDelete:607 » AccessDenied
s3a://steve...
[ERROR] ITestPartialRenamesDeletes.testPartialEmptyDirDelete:572 »
AccessDenied s3a://...
[ERROR] ITestPartialRenamesDeletes.testPartialEmptyDirDelete:572 »
AccessDenied s3a://...
[ERROR] ITestPartialRenamesDeletes.testRenameDirFailsInDelete:457 »
AccessDenied s3a:/...
[ERROR] ITestPartialRenamesDeletes.testRenameDirFailsInDelete:457 »
AccessDenied s3a:/...
[ERROR] ITestPartialRenamesDeletes.testRenameFileFailsNoWrite:503 »
AccessDenied s3a:/...
[ERROR] ITestPartialRenamesDeletes.testRenameFileFailsNoWrite:503 »
AccessDenied s3a:/...
[ERROR] ITestPartialRenamesDeletes.testRenameParentPathNotWriteable:393 »
AccessDenied
[ERROR] ITestPartialRenamesDeletes.testRenameParentPathNotWriteable:393 »
AccessDenied
[ERROR] ITestPartialRenamesDeletes.testRenamePermissionRequirements:756 »
AccessDenied
[ERROR] ITestPartialRenamesDeletes.testRenamePermissionRequirements:756 »
AccessDenied
[ERROR] ITestPartialRenamesDeletes.testRenameSingleFileFailsInDelete:416 »
AccessDenied
[ERROR] ITestPartialRenamesDeletes.testRenameSingleFileFailsInDelete:416 »
AccessDenied
[INFO]
{code}
> S3A Assume role tests failing against S3Express stores
> --
>
> Key: HADOOP-19003
> URL: https://issues.apache.org/jira/browse/HADOOP-19003
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Priority: Minor
>
> The test suits which assume roles with restricted permissions down paths
> still fail on S3Express, even after disabling createSession.
> This is with a role which *should* work.
> Either the role setup is wrong, or there's something special about role
> configuration for S3Express buckets
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h
[jira] [Created] (HADOOP-19003) S3A Assume role tests failing against S3Express stores
Steve Loughran created HADOOP-19003: --- Summary: S3A Assume role tests failing against S3Express stores Key: HADOOP-19003 URL: https://issues.apache.org/jira/browse/HADOOP-19003 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3, test Affects Versions: 3.4.0 Reporter: Steve Loughran The test suits which assume roles with restricted permissions down paths still fail on S3Express, even after disabling createSession. This is with a role which *should* work. Either the role setup is wrong, or there's something special about role configuration for S3Express buckets -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18997) S3A: add way to turn off CreateSession API
[ https://issues.apache.org/jira/browse/HADOOP-18997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18997: Description: add a way to disable the need to use the createsession call, so as to allow for * simplifying our role test runs * benchmarking the performance hit * troubleshooting IAM permissions this can also be disabled from the sysprop "aws.disableS3ExpressAuth" was: add a way to disable the need to use the createsession call, so as to allow for benchmarking the performance hit troubleshooting IAM permissions for our role tests we can turn this off > S3A: add way to turn off CreateSession API > -- > > Key: HADOOP-18997 > URL: https://issues.apache.org/jira/browse/HADOOP-18997 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: pull-request-available > > add a way to disable the need to use the createsession call, so as to allow > for > * simplifying our role test runs > * benchmarking the performance hit > * troubleshooting IAM permissions > this can also be disabled from the sysprop "aws.disableS3ExpressAuth" -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18968) S3A FileSystem does not correctly list empty directories after HADOOP-17199
[ https://issues.apache.org/jira/browse/HADOOP-18968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18968: Component/s: fs/s3 > S3A FileSystem does not correctly list empty directories after HADOOP-17199 > --- > > Key: HADOOP-18968 > URL: https://issues.apache.org/jira/browse/HADOOP-18968 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 3.2.4 >Reporter: Max Xie >Priority: Major > Attachments: image-2023-11-10-16-36-53-337.png > > > After HADOOP-17199, s3a filesystem dose not correctly ls empty directories. > For example. > Before HADOOP-17199 > ``` > hdfs dfs -ls s3a://testbucket/dir/empty > > // work right > ``` > > After HADOOP-17199 > ``` > hdfs dfs -ls s3a://testbucket/dir/empty > ls: `s3a://testbucket/dir/empty': No such file or directory > > // wrong ouput > ``` > > After Check the code, s3a empty folder don't have prefixes Or objects, it > whil throw FIleNotFouldException. > !image-2023-11-10-16-36-53-337.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19000) Unknown S3Express bucket raises UnknownHostException rather than NoSuchBucketException; will block for retries
[ https://issues.apache.org/jira/browse/HADOOP-19000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792835#comment-17792835 ] Steve Loughran commented on HADOOP-19000: - no, not related to HADOOP-18915, though if ApiCallTimeoutException does start containing exceptions the placement there is wrong as we should look inside for any IOE. moving it down will mean that if it is fixed in the SDK then anyone upgrading their SDK will get the better handling. > Unknown S3Express bucket raises UnknownHostException rather than > NoSuchBucketException; will block for retries > -- > > Key: HADOOP-19000 > URL: https://issues.apache.org/jira/browse/HADOOP-19000 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Priority: Minor > > When an attempt is made to work with an s3 express bucket which isn't there. > the createSession API fails with UnknownHostException > This is actually retried within the sdk, so we are lucky that HADOOP-18889 > cut the retry count down. Even so, failures are slow and not very informative. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19001) LD_LIBRARY_PATH is missing HADOOP_COMMON_LIB_NATIVE_DIR
[
https://issues.apache.org/jira/browse/HADOOP-19001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792400#comment-17792400
]
Steve Loughran commented on HADOOP-19001:
-
we've tried to lock this down for security, if you can find any explicit
discussion on what decisions were made with the new scripts, that may show the
reasoning. Or its just an accidental change. See what you can find out before
we rush to change this.
> LD_LIBRARY_PATH is missing HADOOP_COMMON_LIB_NATIVE_DIR
> ---
>
> Key: HADOOP-19001
> URL: https://issues.apache.org/jira/browse/HADOOP-19001
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.2.4
>Reporter: Zilong Zhu
>Priority: Major
>
> When we run a spark job, we find that it cannot load the native library
> successfully.
> We found a difference between hadoop2 and hadoop3.
> hadoop2-Spark-System Properties：
> |java.library.path|:/hadoop/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib|
> hadoop3-Spark-System Properties：
> |java.library.path|:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib|
> The key point is：
> hadoop2-hadoop-config.sh：
> HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH" <--267
> export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JAVA_LIBRARY_PATH <--268
>
> hadoop3-hadoop-functions.sh:
> hadoop_add_param HADOOP_OPTS java.library.path \
> "-Djava.library.path=${JAVA_LIBRARY_PATH}"
> export LD_LIBRARY_PATH <--1484
>
> At the same time, the hadoop3 will clear all non-whitelisted environment
> variables.
> I'm not sure if it was intentional. But it makes our spark job unable to find
> the native library on hadoop3.
> Maybe we should modify hadoop-functions.sh(1484) and add LD_LIBRARY_PATH to
> the default configuration item yarn.nodemanager.env-whitelist.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19000) Unknown S3Express bucket raises UnknownHostException rather than NoSuchBucketException; will block for retries
[ https://issues.apache.org/jira/browse/HADOOP-19000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-19000: Summary: Unknown S3Express bucket raises UnknownHostException rather than NoSuchBucketException; will block for retries (was: Unknown S3Express bucket raises UnknownHostException rather than NoSuckBucketException) > Unknown S3Express bucket raises UnknownHostException rather than > NoSuchBucketException; will block for retries > -- > > Key: HADOOP-19000 > URL: https://issues.apache.org/jira/browse/HADOOP-19000 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Priority: Minor > > When an attempt is made to work with an s3 express bucket which isn't there. > the createSession API fails with UnknownHostException > This is actually retried within the sdk, so we are lucky that HADOOP-18889 > cut the retry count down. Even so, failures are slow and not very informative. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-19000) Unknown S3Express bucket raises UnknownHostException rather than NoSuckBucketException
[
https://issues.apache.org/jira/browse/HADOOP-19000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792110#comment-17792110
]
Steve Loughran edited comment on HADOOP-19000 at 12/1/23 3:34 PM:
--
replicate by trying to list a bucket which isn't present. note that my test
setup has fs.s3a.attempts.maximum = 1, so I don't get to wait for failures. the
default of 5 means a wait for about 5s.
{code}
> bin/hadoop fs -ls s3a://stevel2--usw2-az1--x-s3/
2023-12-01 15:15:52,385
[s3a-transfer-stevel2--usw2-az1--x-s3-unbounded-pool2-t1] DEBUG s3a.Invoker
(Invoker.java:retryUntranslated(474)) - List stevel2--usw2-az1--x-s3:/
delimiter=/ keys=5000 requester pays=null ;
software.amazon.awssdk.core.exception.SdkClientException: Received an
UnknownHostException when attempting to interact with a service. See cause for
the exact endpoint that is failing to resolve. If this is happening on an
endpoint that previously worked, there may be a network connectivity issue or
your DNS cache could be storing endpoints for too long.,
2023-12-01 15:15:52,386
[s3a-transfer-stevel2--usw2-az1--x-s3-unbounded-pool2-t1] DEBUG
s3a.S3ARetryPolicy (S3ARetryPolicy.java:shouldRetry(305)) - Retry probe for
UnknownHostException with 0 retries and 0 failovers, idempotent=true, due to
java.net.UnknownHostException:
software.amazon.awssdk.core.exception.SdkClientException: Received an
UnknownHostException when attempting to interact with a service. See cause for
the exact endpoint that is failing to resolve. If this is happening on an
endpoint that previously worked, there may be a network connectivity issue or
your DNS cache could be storing endpoints for too long.:
stevel2--usw2-az1--x-s3.s3express-usw2-az1.us-east-2.amazonaws.com
java.net.UnknownHostException:
software.amazon.awssdk.core.exception.SdkClientException: Received an
UnknownHostException when attempting to interact with a service. See cause for
the exact endpoint that is failing to resolve. If this is happening on an
endpoint that previously worked, there may be a network connectivity issue or
your DNS cache could be storing endpoints for too long.:
stevel2--usw2-az1--x-s3.s3express-usw2-az1.us-east-2.amazonaws.com
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.fs.s3a.impl.ErrorTranslation.wrapWithInnerIOE(ErrorTranslation.java:132)
at
org.apache.hadoop.fs.s3a.impl.ErrorTranslation.maybeExtractIOException(ErrorTranslation.java:105)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:213)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:481)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:2954)
at
org.apache.hadoop.fs.s3a.S3AFileSystem$ListingOperationCallbacksImpl.lambda$listObjectsAsync$0(S3AFileSystem.java:2573)
at
org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:88)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Received
an UnknownHostException when attempting to interact with a service. See cause
for the exact endpoint that is failing to resolve. If this is happening on an
endpoint that previously worked, there may be a network connectivity issue or
your DNS cache could be storing endpoints for too long.
at
software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at
software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor.modifyException(HelpfulUnknownHostExceptionInterceptor.java:59)
at
software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.modifyException(ExecutionInterceptorChain.java:202)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.runModifyException(ExceptionReportingUtils.java:54)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.reportFailureToInterceptors(ExceptionReportingUtils.java:38)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute
[jira] [Commented] (HADOOP-19000) Unknown S3Express bucket raises UnknownHostException rather than NoSuckBucketException
[
https://issues.apache.org/jira/browse/HADOOP-19000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792112#comment-17792112
]
Steve Loughran commented on HADOOP-19000:
-
now, set the retries to 10 and instead of any UHE being raised, an
ApiCallTImeoutException is raised, which we then retry ourselves, *and the
inner exception is lost*.
API retry code should include last exception raised in its stack, so we can
extract and react to. Or have I just broken this with HADOOP-18915 placement of
retry mapping. will test that locally.
{code}
ed-b026-4dc8dfc37f68-0007 Executing op_list_status with
{software.amazon.awssdk.services.s3.model.CreateSessionRequest size=0,
mutating=true};
https://audit.example.org/hadoop/1/op_list_status/b8db02c1-bbfc-4bed-b026-4dc8dfc37f68-0007/?op=op_list_status&pr=stevel&ps=3d28ab00-f260-4fd4-bd72-f2209126ee96&cm=FsShell&id=b8db02c1-bbfc-4bed-b026-4dc8dfc37f68-0007&t0=1&fs=b8db02c1-bbfc-4bed-b026-4dc8dfc37f68&t1=16&ts=170151871
2023-12-01 15:27:53,025
[s3a-transfer-stevel2--usw2-az1--x-s3-unbounded-pool2-t1] DEBUG s3a.Invoker
(Invoker.java:retryUntranslated(474)) - List stevel2--usw2-az1--x-s3:/
delimiter=/ keys=5000 requester pays=null ;
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 1 millis,
2023-12-01 15:27:53,025
[s3a-transfer-stevel2--usw2-az1--x-s3-unbounded-pool2-t1] DEBUG
s3a.S3ARetryPolicy (S3ARetryPolicy.java:shouldRetry(305)) - Retry probe for
AWSApiCallTimeoutException with 1 retries and 0 failovers, idempotent=true, due
to org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: List
stevel2--usw2-az1--x-s3:/ delimiter=/ keys=5000 requester pays=null:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 1 millis
org.apache.hadoop.fs.s3a.AWSApiCallTimeoutException: List
stevel2--usw2-az1--x-s3:/ delimiter=/ keys=5000 requester pays=null:
software.amazon.awssdk.core.exception.ApiCallTimeoutException: Client execution
did not complete before the specified timeout configuration: 1 millis
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:189)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:481)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:2954)
at
org.apache.hadoop.fs.s3a.S3AFileSystem$ListingOperationCallbacksImpl.lambda$listObjectsAsync$0(S3AFileSystem.java:2573)
at
org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:88)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: software.amazon.awssdk.core.exception.ApiCallTimeoutException:
Client execution did not complete before the specified timeout configuration:
1 millis
at
software.amazon.awssdk.core.exception.ApiCallTimeoutException$BuilderImpl.build(ApiCallTimeoutException.java:97)
at
software.amazon.awssdk.core.exception.ApiCallTimeoutException.create(ApiCallTimeoutException.java:38)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.generateApiCallTimeoutException(ApiCallTimeoutTrackingStage.java:151)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.handleInterruptedException(ApiCallTimeoutTrackingStage.java:139)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.translatePipelineException(ApiCallTimeoutTrackingStage.java:107)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at
software.amazon.awssdk.core.internal.
[jira] [Commented] (HADOOP-19000) Unknown S3Express bucket raises UnknownHostException rather than NoSuckBucketException
[ https://issues.apache.org/jira/browse/HADOOP-19000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792110#comment-17792110 ] Steve Loughran commented on HADOOP-19000: - replicate by trying to list a bucket which isn't present. note that my test setup has fs.s3a.attempts.maximum = 1, so I don't get to wait for failures. the default of 5 means a wait for about 5s. > bin/hadoop fs -ls s3a://stevel2--usw2-az1--x-s3/ 2023-12-01 15:15:52,385 [s3a-transfer-stevel2--usw2-az1--x-s3-unbounded-pool2-t1] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(474)) - List stevel2--usw2-az1--x-s3:/ delimiter=/ keys=5000 requester pays=null ; software.amazon.awssdk.core.exception.SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long., 2023-12-01 15:15:52,386 [s3a-transfer-stevel2--usw2-az1--x-s3-unbounded-pool2-t1] DEBUG s3a.S3ARetryPolicy (S3ARetryPolicy.java:shouldRetry(305)) - Retry probe for UnknownHostException with 0 retries and 0 failovers, idempotent=true, due to java.net.UnknownHostException: software.amazon.awssdk.core.exception.SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.: stevel2--usw2-az1--x-s3.s3express-usw2-az1.us-east-2.amazonaws.com java.net.UnknownHostException: software.amazon.awssdk.core.exception.SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.: stevel2--usw2-az1--x-s3.s3express-usw2-az1.us-east-2.amazonaws.com at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.fs.s3a.impl.ErrorTranslation.wrapWithInnerIOE(ErrorTranslation.java:132) at org.apache.hadoop.fs.s3a.impl.ErrorTranslation.maybeExtractIOException(ErrorTranslation.java:105) at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:213) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:481) at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431) at org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:2954) at org.apache.hadoop.fs.s3a.S3AFileSystem$ListingOperationCallbacksImpl.lambda$listObjectsAsync$0(S3AFileSystem.java:2573) at org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:88) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: software.amazon.awssdk.core.exception.SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long. at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) at software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor.modifyException(HelpfulUnknownHostExceptionInterceptor.java:59) at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.modifyException(ExecutionInterceptorChain.java:202) at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.runModifyException(ExceptionReportingUtils.java:54) at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.reportFailureToInterceptors(ExceptionReportingUtils.java:38) at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:39) a
[jira] [Created] (HADOOP-19000) Unknown S3Express bucket raises UnknownHostException rather than NoSuckBucketException
Steve Loughran created HADOOP-19000: --- Summary: Unknown S3Express bucket raises UnknownHostException rather than NoSuckBucketException Key: HADOOP-19000 URL: https://issues.apache.org/jira/browse/HADOOP-19000 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran When an attempt is made to work with an s3 express bucket which isn't there. the createSession API fails with UnknownHostException This is actually retried within the sdk, so we are lucky that HADOOP-18889 cut the retry count down. Even so, failures are slow and not very informative. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18996) S3A to provide full support for S3 Express One Zone
[ https://issues.apache.org/jira/browse/HADOOP-18996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18996: Parent Issue: HADOOP-18886 (was: HADOOP-18477) > S3A to provide full support for S3 Express One Zone > --- > > Key: HADOOP-18996 > URL: https://issues.apache.org/jira/browse/HADOOP-18996 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > HADOOP-18995 upgrades the SDK version which allows connecting to a s3 express > one zone support. > Complete support needs to be added to address tests that fail with s3 express > one zone, additional tests, documentation etc. > * hadoop-common path capability to indicate that treewalking may encounter > missing dirs > * use this in treewalking code in shell, mapreduce FileInputFormat etc to not > fail during treewalks > * extra path capability for s3express too. > * tests for this > * anything else -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18996) S3A to provide full support for S3 Express One Zone
[ https://issues.apache.org/jira/browse/HADOOP-18996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18996: Release Note: Hadoop S3A connector has explicit awareness of and support for S3Express storage. Hadoop-common and hadoop-mapreduce treewalking code (shell commands, FileInputFormat,...) can cope with paths with incomplete uploads being visible. > S3A to provide full support for S3 Express One Zone > --- > > Key: HADOOP-18996 > URL: https://issues.apache.org/jira/browse/HADOOP-18996 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > HADOOP-18995 upgrades the SDK version which allows connecting to a s3 express > one zone support. > Complete support needs to be added to address tests that fail with s3 express > one zone, additional tests, documentation etc. > * hadoop-common path capability to indicate that treewalking may encounter > missing dirs > * use this in treewalking code in shell, mapreduce FileInputFormat etc to not > fail during treewalks > * extra path capability for s3express too. > * tests for this > * anything else -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18999) S3A: debug logging for http traffic to S3 stores
Steve Loughran created HADOOP-18999:
---
Summary: S3A: debug logging for http traffic to S3 stores
Key: HADOOP-18999
URL: https://issues.apache.org/jira/browse/HADOOP-18999
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
AWS SDK bundle.jar logging doesn't set up right.
{code}
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
{code}
Cloudstore commands have a -debug option to force set this through log4j APIs;
this does work.
Proposed:
* add reflection-based ability to set/query log4j log levels (+tests, obviously)
* add a new log `org.apache.hadoop.fs.s3a.logging.sdk`
* if set to DEBUG, DefaultS3ClientFactory will enable logging on the aws
internal/shaded classes
this allows log4j.properties to turn on logging; reflection ensures all is well
on other log back-ends and when unshaded aws sdk jars are in use
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18886) S3A: AWS SDK V2 Migration: stabilization and S3Express
[ https://issues.apache.org/jira/browse/HADOOP-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18886: Summary: S3A: AWS SDK V2 Migration: stabilization and S3Express (was: S3A: AWS SDK V2 Migration: stabilization) > S3A: AWS SDK V2 Migration: stabilization and S3Express > -- > > Key: HADOOP-18886 > URL: https://issues.apache.org/jira/browse/HADOOP-18886 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Major > > The final stabilisation changes to the V2 SDK MIgration; those moved off the > HADOOP-18073 JIRA so we can close that. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18997) S3A: add way to turn off CreateSession API
[ https://issues.apache.org/jira/browse/HADOOP-18997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-18997: --- Assignee: Steve Loughran > S3A: add way to turn off CreateSession API > -- > > Key: HADOOP-18997 > URL: https://issues.apache.org/jira/browse/HADOOP-18997 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: pull-request-available > > add a way to disable the need to use the createsession call, so as to allow > for > benchmarking the performance hit > troubleshooting IAM permissions > for our role tests we can turn this off -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18996) S3A to provide full support for S3 Express One Zone
[ https://issues.apache.org/jira/browse/HADOOP-18996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18996. - Fix Version/s: 3.4.0 Resolution: Fixed > S3A to provide full support for S3 Express One Zone > --- > > Key: HADOOP-18996 > URL: https://issues.apache.org/jira/browse/HADOOP-18996 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > HADOOP-18995 upgrades the SDK version which allows connecting to a s3 express > one zone support. > Complete support needs to be added to address tests that fail with s3 express > one zone, additional tests, documentation etc. > * hadoop-common path capability to indicate that treewalking may encounter > missing dirs > * use this in treewalking code in shell, mapreduce FileInputFormat etc to not > fail during treewalks > * extra path capability for s3express too. > * tests for this > * anything else -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18997) S3A: add way to turn off CreateSession API
Steve Loughran created HADOOP-18997: --- Summary: S3A: add way to turn off CreateSession API Key: HADOOP-18997 URL: https://issues.apache.org/jira/browse/HADOOP-18997 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran add a way to disable the need to use the createsession call, so as to allow for benchmarking the performance hit troubleshooting IAM permissions for our role tests we can turn this off -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18996) S3A to provide full support for S3 Express One Zone
[ https://issues.apache.org/jira/browse/HADOOP-18996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17791267#comment-17791267 ] Steve Loughran commented on HADOOP-18996: - PR is up; as I say in the PR, if people review it and for any change other than deleting stuff/disabling tests, I'd like the issues to be added to the v2 sdk stabilisation. Why so? people need to play with the new store before they can really review the changes > S3A to provide full support for S3 Express One Zone > --- > > Key: HADOOP-18996 > URL: https://issues.apache.org/jira/browse/HADOOP-18996 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > HADOOP-18995 upgrades the SDK version which allows connecting to a s3 express > one zone support. > Complete support needs to be added to address tests that fail with s3 express > one zone, additional tests, documentation etc. > * hadoop-common path capability to indicate that treewalking may encounter > missing dirs > * use this in treewalking code in shell, mapreduce FileInputFormat etc to not > fail during treewalks > * extra path capability for s3express too. > * tests for this > * anything else -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18996) S3A to provide full support for S3 Express One Zone
[ https://issues.apache.org/jira/browse/HADOOP-18996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18996: Description: HADOOP-18995 upgrades the SDK version which allows connecting to a s3 express one zone support. Complete support needs to be added to address tests that fail with s3 express one zone, additional tests, documentation etc. * hadoop-common path capability to indicate that treewalking may encounter missing dirs * use this in treewalking code in shell, mapreduce FileInputFormat etc to not fail during treewalks * extra path capability for s3express too. * tests for this * anything else was: HADOOP-18995 upgrades the SDK version which allows connecting to a s3 express one zone support. Complete support needs to be added to address tests that fail with s3 express one zone, additional tests, documentation etc. > S3A to provide full support for S3 Express One Zone > --- > > Key: HADOOP-18996 > URL: https://issues.apache.org/jira/browse/HADOOP-18996 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Steve Loughran >Priority: Major > > HADOOP-18995 upgrades the SDK version which allows connecting to a s3 express > one zone support. > Complete support needs to be added to address tests that fail with s3 express > one zone, additional tests, documentation etc. > * hadoop-common path capability to indicate that treewalking may encounter > missing dirs > * use this in treewalking code in shell, mapreduce FileInputFormat etc to not > fail during treewalks > * extra path capability for s3express too. > * tests for this > * anything else -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18996) S3A to provide full support for S3 Express One Zone
[ https://issues.apache.org/jira/browse/HADOOP-18996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18996: Summary: S3A to provide full support for S3 Express One Zone (was: Add necessary software support for S3 Express One Zone) > S3A to provide full support for S3 Express One Zone > --- > > Key: HADOOP-18996 > URL: https://issues.apache.org/jira/browse/HADOOP-18996 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Steve Loughran >Priority: Major > > HADOOP-18995 upgrades the SDK version which allows connecting to a s3 express > one zone support. > Complete support needs to be added to address tests that fail with s3 express > one zone, additional tests, documentation etc. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18915) Tune/extend S3A http connection and thread pool settings
[ https://issues.apache.org/jira/browse/HADOOP-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18915. - Fix Version/s: 3.4.0 Resolution: Fixed > Tune/extend S3A http connection and thread pool settings > > > Key: HADOOP-18915 > URL: https://issues.apache.org/jira/browse/HADOOP-18915 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Increases existing pool sizes, as with server scale and vector > IO, larger pools are needed > fs.s3a.connection.maximum 200 > fs.s3a.threads.max 96 > Adds new configuration options for v2 sdk internal timeouts, > both with default of 60s: > fs.s3a.connection.acquisition.timeout > fs.s3a.connection.idle.time > All the pool/timoeut options are covered in performance.md > Moves all timeout/duration options in the s3a FS to taking > temporal units (h, m, s, ms,...); retaining the previous default > unit (normally millisecond) > Adds a minimum duration for most of these, in order to recover from > deployments where a timeout has been set on the assumption the unit > was seconds, not millis. > Uses java.time.Duration throughout the codebase; > retaining the older numeric constants in > org.apache.hadoop.fs.s3a.Constants for backwards compatibility; > these are now deprecated. > Adds new class AWSApiCallTimeoutException to be raised on > sdk-related methods and also gateway timeouts. This is a subclass > of org.apache.hadoop.net.ConnectTimeoutException to support > existing retry logic. > + reverted default value of fs.s3a.create.performance to false; > inadvertently set to true during testing. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18915) Tune/extend S3A http connection and thread pool settings
[ https://issues.apache.org/jira/browse/HADOOP-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18915: Summary: Tune/extend S3A http connection and thread pool settings (was: Extend S3A http client connection timeouts) > Tune/extend S3A http connection and thread pool settings > > > Key: HADOOP-18915 > URL: https://issues.apache.org/jira/browse/HADOOP-18915 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > Increases existing pool sizes, as with server scale and vector > IO, larger pools are needed > fs.s3a.connection.maximum 200 > fs.s3a.threads.max 96 > Adds new configuration options for v2 sdk internal timeouts, > both with default of 60s: > fs.s3a.connection.acquisition.timeout > fs.s3a.connection.idle.time > All the pool/timoeut options are covered in performance.md > Moves all timeout/duration options in the s3a FS to taking > temporal units (h, m, s, ms,...); retaining the previous default > unit (normally millisecond) > Adds a minimum duration for most of these, in order to recover from > deployments where a timeout has been set on the assumption the unit > was seconds, not millis. > Uses java.time.Duration throughout the codebase; > retaining the older numeric constants in > org.apache.hadoop.fs.s3a.Constants for backwards compatibility; > these are now deprecated. > Adds new class AWSApiCallTimeoutException to be raised on > sdk-related methods and also gateway timeouts. This is a subclass > of org.apache.hadoop.net.ConnectTimeoutException to support > existing retry logic. > + reverted default value of fs.s3a.create.performance to false; > inadvertently set to true during testing. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18915) Extend S3A http client connection timeouts
[ https://issues.apache.org/jira/browse/HADOOP-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18915: Description: Increases existing pool sizes, as with server scale and vector IO, larger pools are needed fs.s3a.connection.maximum 200 fs.s3a.threads.max 96 Adds new configuration options for v2 sdk internal timeouts, both with default of 60s: fs.s3a.connection.acquisition.timeout fs.s3a.connection.idle.time All the pool/timoeut options are covered in performance.md Moves all timeout/duration options in the s3a FS to taking temporal units (h, m, s, ms,...); retaining the previous default unit (normally millisecond) Adds a minimum duration for most of these, in order to recover from deployments where a timeout has been set on the assumption the unit was seconds, not millis. Uses java.time.Duration throughout the codebase; retaining the older numeric constants in org.apache.hadoop.fs.s3a.Constants for backwards compatibility; these are now deprecated. Adds new class AWSApiCallTimeoutException to be raised on sdk-related methods and also gateway timeouts. This is a subclass of org.apache.hadoop.net.ConnectTimeoutException to support existing retry logic. + reverted default value of fs.s3a.create.performance to false; inadvertently set to true during testing. was: * Add ability to configure *all* timeouts, especially acquisition time * recognise ApiCallTimeout and map tp a retryable exception * use getDuration so suffixes can be used -so remove all ambiguity about time unit * use units in core-default.xml so warnings aren't printed > Extend S3A http client connection timeouts > -- > > Key: HADOOP-18915 > URL: https://issues.apache.org/jira/browse/HADOOP-18915 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > Increases existing pool sizes, as with server scale and vector > IO, larger pools are needed > fs.s3a.connection.maximum 200 > fs.s3a.threads.max 96 > Adds new configuration options for v2 sdk internal timeouts, > both with default of 60s: > fs.s3a.connection.acquisition.timeout > fs.s3a.connection.idle.time > All the pool/timoeut options are covered in performance.md > Moves all timeout/duration options in the s3a FS to taking > temporal units (h, m, s, ms,...); retaining the previous default > unit (normally millisecond) > Adds a minimum duration for most of these, in order to recover from > deployments where a timeout has been set on the assumption the unit > was seconds, not millis. > Uses java.time.Duration throughout the codebase; > retaining the older numeric constants in > org.apache.hadoop.fs.s3a.Constants for backwards compatibility; > these are now deprecated. > Adds new class AWSApiCallTimeoutException to be raised on > sdk-related methods and also gateway timeouts. This is a subclass > of org.apache.hadoop.net.ConnectTimeoutException to support > existing retry logic. > + reverted default value of fs.s3a.create.performance to false; > inadvertently set to true during testing. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18995) S3A: Upgrade AWS SDK version to 2.21.33 for Amazon S3 Express One Zone support
[ https://issues.apache.org/jira/browse/HADOOP-18995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18995. - Fix Version/s: 3.4.0 Resolution: Fixed > S3A: Upgrade AWS SDK version to 2.21.33 for Amazon S3 Express One Zone support > -- > > Key: HADOOP-18995 > URL: https://issues.apache.org/jira/browse/HADOOP-18995 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Ahmar Suhail >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Upgrade SDK version to 2.21.33, which adds S3 Express One Zone support. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18995) S3A: Upgrade AWS SDK version to 2.21.33 for Amazon S3 Express One Zone support
[ https://issues.apache.org/jira/browse/HADOOP-18995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18995: Summary: S3A: Upgrade AWS SDK version to 2.21.33 for Amazon S3 Express One Zone support (was: Add support for Amazon S3 Express One Zone Storage - SDK version upgrade) > S3A: Upgrade AWS SDK version to 2.21.33 for Amazon S3 Express One Zone support > -- > > Key: HADOOP-18995 > URL: https://issues.apache.org/jira/browse/HADOOP-18995 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Ahmar Suhail >Priority: Major > Labels: pull-request-available > > Upgrade SDK version to 2.21.33, which adds S3 Express One Zone support. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18996) Add necessary software support for S3 Express One Zone
[ https://issues.apache.org/jira/browse/HADOOP-18996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17791037#comment-17791037 ] Steve Loughran commented on HADOOP-18996: - i can probably help here > Add necessary software support for S3 Express One Zone > -- > > Key: HADOOP-18996 > URL: https://issues.apache.org/jira/browse/HADOOP-18996 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Steve Loughran >Priority: Major > > HADOOP-18995 upgrades the SDK version which allows connecting to a s3 express > one zone support. > Complete support needs to be added to address tests that fail with s3 express > one zone, additional tests, documentation etc. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18996) Add necessary software support for S3 Express One Zone
[ https://issues.apache.org/jira/browse/HADOOP-18996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-18996: --- Assignee: Steve Loughran > Add necessary software support for S3 Express One Zone > -- > > Key: HADOOP-18996 > URL: https://issues.apache.org/jira/browse/HADOOP-18996 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Steve Loughran >Priority: Major > > HADOOP-18995 upgrades the SDK version which allows connecting to a s3 express > one zone support. > Complete support needs to be added to address tests that fail with s3 express > one zone, additional tests, documentation etc. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18994) Support OpenSSL 3
[ https://issues.apache.org/jira/browse/HADOOP-18994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18994: Summary: Support OpenSSL 3 (was: Support OpneSSL 3) > Support OpenSSL 3 > - > > Key: HADOOP-18994 > URL: https://issues.apache.org/jira/browse/HADOOP-18994 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.3.6 >Reporter: Thippana Vamsi Kalyan >Priority: Major > > Hadoop uses native OpenSSL library provided by linux distribution. > OpenSSL 1.1.1 is deprecated: > https://www.openssl.org/blog/blog/2023/03/28/1.1.1-EOL/ > We need to ensure that Hadoop works with OpenSSL 3. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18991) Remove commons-benautils dependency from Hadoop 3
[ https://issues.apache.org/jira/browse/HADOOP-18991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18991: Issue Type: Improvement (was: Bug) > Remove commons-benautils dependency from Hadoop 3 > - > > Key: HADOOP-18991 > URL: https://issues.apache.org/jira/browse/HADOOP-18991 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Istvan Toth >Priority: Major > > Hadoop doesn't acually use it, and it pollutes the classpath of dependent > projects. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18990) S3A: retry on credential expiry
Steve Loughran created HADOOP-18990:
---
Summary: S3A: retry on credential expiry
Key: HADOOP-18990
URL: https://issues.apache.org/jira/browse/HADOOP-18990
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.4.0
Reporter: Steve Loughran
Reported in AWS SDK https://github.com/aws/aws-sdk-java-v2/issues/3408
bq. In RetryableStage execute method, the "AwsCredentails" does not attempt to
renew if it has expired. Therefore, if a method called with the existing
credential is expiring soon, the number of retry is less than intended due to
the expiration of the credential.
The stack from this report doesn't show any error detail we can use to identify
the 400 exception as something we should be retrying on. This could be due to
the logging, or it could actually hold. we've have to generate some socket
credentials, let them expire and then see how hadoop fs commands failed.
Something to do by hand as an STS test to do this is probably slow. *unless we
expire all session credentials of a given role?*. Could be good, would be
traumatic for other test runs though.
{code}
software.amazon.awssdk.services.s3.model.S3Exception: The provided token has
expired. (Service: S3, Status Code: 400, Request ID: 3YWKVBNJPNTXPJX2, Extended
Request ID:
GkR56xA0r/Ek7zqQdB2ZdP3wqMMhf49HH7hc5N2TAIu47J3HEk6yvSgVNbX7ADuHDy/Irhr2rPQ=)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18915) Extend S3A http client connection timeouts
[ https://issues.apache.org/jira/browse/HADOOP-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18915: Description: * Add ability to configure *all* timeouts, especially acquisition time * recognise ApiCallTimeout and map tp a retryable exception * use getDuration so suffixes can be used -so remove all ambiguity about time unit * use units in core-default.xml so warnings aren't printed was: In the client config builders, when [setting timeouts|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AWSClientConfig.java#L120], it uses Duration.ofSeconds(), configs all use milliseconds so this needs to be updated to Duration.ofMillis(). > Extend S3A http client connection timeouts > -- > > Key: HADOOP-18915 > URL: https://issues.apache.org/jira/browse/HADOOP-18915 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > * Add ability to configure *all* timeouts, especially acquisition time > * recognise ApiCallTimeout and map tp a retryable exception > * use getDuration so suffixes can be used -so remove all ambiguity about time > unit > * use units in core-default.xml so warnings aren't printed -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18915) Extend S3A http client connection timeouts
[ https://issues.apache.org/jira/browse/HADOOP-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18915: Summary: Extend S3A http client connection timeouts (was: S3A client builder to use getTImeDuration() for HTTP timeouts) > Extend S3A http client connection timeouts > -- > > Key: HADOOP-18915 > URL: https://issues.apache.org/jira/browse/HADOOP-18915 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > In the client config builders, when [setting > timeouts|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AWSClientConfig.java#L120], > it uses Duration.ofSeconds(), configs all use milliseconds so this needs to > be updated to Duration.ofMillis(). > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
