Laszlo Gaal created IMPALA-13017:
------------------------------------
Summary: Add configuration option to set S3 endpoint for the
minicluster
Key: IMPALA-13017
URL: https://issues.apache.org/jira/browse/IMPALA-13017
Project: IMPALA
Issue Type: Improvement
Reporter: Laszlo Gaal
Assignee: Laszlo Gaal
As AWS S3 adds functional improvements, some of the required configuration
items become stricter, or optional items become mandatory. One example is the
endpoint configuration: while older regions support the legacy endpoint syntax
of _bucketname_.s3.amazonaws.com, new regions, where S3 buckets can be created
require the AWS v4 request signing algorithm, which requires the request URL to
contain the region as well. In the latter case the bucket URL becomes
_bucketname.region-code_.amazonaws.com, e.g.
test-bucket.us-west-2.amazonaws.com.
Bucket configuration for the Hadoop Cloud Connector (for the s3a:// protocol)
works a bit differently: the bucket name is supplied separately, e.g.
s3a://landsat-pds, and various other parameters can be supplied via optional
fields in core-site.xml
When the Impala minicluster is configured to use S3 for its default filesystem,
it should be possible to supply the endpoint to use so that S3-specific tests
can be run against endpoints that don't support connecting via the legacy
generic endpoint.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
