[
https://issues.apache.org/jira/browse/HADOOP-19066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825819#comment-17825819
]
ASF GitHub Bot commented on HADOOP-19066:
-----------------------------------------
virajjasani commented on PR #6539:
URL: https://github.com/apache/hadoop/pull/6539#issuecomment-1992509237
Something seems odd. This test overrides endpoint/region configs so setting
any endpoint/region should have made no difference:
```
@Test
public void testCentralEndpointAndNullRegionFipsWithCRUD() throws
Throwable {
describe("Access the test bucket using central endpoint and"
+ " null region and fips enabled, perform file system CRUD
operations");
final Configuration conf = getConfiguration();
final Configuration newConf = new Configuration(conf);
removeBaseAndBucketOverrides(
newConf,
ENDPOINT,
AWS_REGION,
FIPS_ENDPOINT);
newConf.set(ENDPOINT, CENTRAL_ENDPOINT);
newConf.setBoolean(FIPS_ENDPOINT, true);
newFS = new S3AFileSystem();
newFS.initialize(getFileSystem().getUri(), newConf);
assertOpsUsingNewFs();
}
```
I tested using these settings and there is no difference in behaviour
because the test overrides base and bucket configs for endpoint/region.
I tried:
1. endpoint: us-west-2, region: unset
2. endpoint: central, region: unset
3. endpoint: unset, region: unset
From the stacktrace from Jira:
```
[ERROR] Tests run: 18, Failures: 0, Errors: 1, Skipped: 1, Time elapsed:
56.26 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion
[ERROR]
testCentralEndpointAndNullRegionFipsWithCRUD(org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion)
Time elapsed: 4.821 s <<< ERROR!
java.net.UnknownHostException: getFileStatus on
s3a://stevel-london/test/testCentralEndpointAndNullRegionFipsWithCRUD/srcdir:
software.amazon.awssdk.core.exception.SdkClientException: Received an
UnknownHostException when attempting to interact with a service. See cause for
the exact endpoint that is failing to resolve. If this is happening on an
endpoint that previously worked, there may be a network connectivity issue or
your DNS cache could be storing endpoints for too long.:
software.amazon.awssdk.core.exception.SdkClientException: Received an
UnknownHostException when attempting to interact with a service. See cause for
the exact endpoint that is failing to resolve. If this is happening on an
endpoint that previously worked, there may be a network connectivity issue or
your DNS cache could be storing endpoints for too long.:
stevel-london.s3-fips.eu-west-2.amazonaws.com
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.fs.s3a.impl.ErrorTranslation.wrapWithInnerIOE(ErrorTranslation.java:182)
at
org.apache.hadoop.fs.s3a.impl.ErrorTranslation.maybeExtractIOException(ErrorTranslation.java:152)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:207)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:155)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:4066)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3922)
at
org.apache.hadoop.fs.s3a.S3AFileSystem$MkdirOperationCallbacksImpl.probePathStatus(S3AFileSystem.java:3794)
at
org.apache.hadoop.fs.s3a.impl.MkdirOperation.probePathStatusOrNull(MkdirOperation.java:173)
at
org.apache.hadoop.fs.s3a.impl.MkdirOperation.getPathStatusExpectingDir(MkdirOperation.java:194)
at
org.apache.hadoop.fs.s3a.impl.MkdirOperation.execute(MkdirOperation.java:108)
at
org.apache.hadoop.fs.s3a.impl.MkdirOperation.execute(MkdirOperation.java:57)
at
org.apache.hadoop.fs.s3a.impl.ExecutingStoreOperation.apply(ExecutingStoreOperation.java:76)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2707)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2726)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:3766)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2494)
at
org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion.assertOpsUsingNewFs(ITestS3AEndpointRegion.java:461)
at
org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion.testCentralEndpointAndNullRegionFipsWithCRUD(ITestS3AEndpointRegion.java:454)
```
Here, we set:
```
removeBaseAndBucketOverrides(
newConf,
ENDPOINT,
AWS_REGION,
FIPS_ENDPOINT);
newConf.set(ENDPOINT, CENTRAL_ENDPOINT);
newConf.setBoolean(FIPS_ENDPOINT, true);
newFS = new S3AFileSystem();
newFS.initialize(getFileSystem().getUri(), newConf);
```
How could stacktrace show different region than us-east-2 when the test
overrides endpoint to central and removes region?
> AWS SDK V2 - Enabling FIPS should be allowed with central endpoint
> ------------------------------------------------------------------
>
> Key: HADOOP-19066
> URL: https://issues.apache.org/jira/browse/HADOOP-19066
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.5.0, 3.4.1
> Reporter: Viraj Jasani
> Assignee: Viraj Jasani
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> FIPS support can be enabled by setting "fs.s3a.endpoint.fips". Since the SDK
> considers overriding endpoint and enabling fips as mutually exclusive, we
> fail fast if fs.s3a.endpoint is set with fips support (details on
> HADOOP-18975).
> Now, we no longer override SDK endpoint for central endpoint since we enable
> cross region access (details on HADOOP-19044) but we would still fail fast if
> endpoint is central and fips is enabled.
> Changes proposed:
> * S3A to fail fast only if FIPS is enabled and non-central endpoint is
> configured.
> * Tests to ensure S3 bucket is accessible with default region us-east-2 with
> cross region access (expected with central endpoint).
> * Document FIPS support with central endpoint on connecting.html.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
