andygrove opened a new issue, #6957: URL: https://github.com/apache/datafusion-comet/issues/6957
### What is the problem the feature request solves? For an `s3` or `s3a` data location, the eligibility gate falls back on every explicit Hadoop `fs.s3a.*` setting other than the access key, secret key, session token, endpoint, endpoint region and path-style access (it also ignores a few read and committer settings), and on every Iceberg `FileIO` `s3.*` or `client.*` property outside its allowlist (#6139, PR #6441). Clusters commonly set such keys cluster-wide, so a `HadoopFileIO` table on S3 rarely passes the gate in production. Each of these makes a write fall back today: - `fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem`, which many Spark configurations set explicitly - connection and thread pools, timeouts and retries: `fs.s3a.connection.maximum`, `fs.s3a.threads.max`, `fs.s3a.connection.timeout`, `fs.s3a.attempts.maximum`, `fs.s3a.retry.limit` - upload buffering: `fs.s3a.fast.upload.buffer`, `fs.s3a.multipart.size`, `fs.s3a.multipart.threshold`, `fs.s3a.buffer.dir`, `fs.s3a.block.size` - `fs.s3a.aws.credentials.provider` - `fs.s3a.encryption.algorithm` and `fs.s3a.encryption.key`, and the older `fs.s3a.server-side-encryption-algorithm` and `fs.s3a.server-side-encryption.key` - on `S3FileIO`: `s3.multipart.part-size-bytes`, `s3.multipart.num-threads`, `s3.multipart.threshold`, `s3.staging-dir`, `s3.retry.*`, `s3.delete-enabled`, `s3.delete.batch-size`, `s3.preload-client-enabled`, `client.credentials-provider`, `s3.acl`, `s3.write.storage-class` ### Describe the potential solution Three parts, each keeping the gate an allowlist: 1. **Settings that cannot change the written object, where it is written, or who writes it.** Admit them, as `IgnoredHadoopS3Keys` already does for the committer and read settings: the native writer has its own connection pool, retries and upload buffering. Audit the candidates against Hadoop's `org.apache.hadoop.fs.s3a.Constants` and Iceberg's `S3FileIOProperties` and `AwsClientProperties` for each pinned version. Admit `fs.s3a.impl` only when it names `S3AFileSystem`. Keep failing closed for anything that can change the request: endpoint and region selection, SSL, proxies, signing, requester pays, ACLs, storage class, checksums, access points. 2. **Credentials.** Map the well-known provider classes named in `fs.s3a.aws.credentials.provider` and `client.credentials-provider` to native credential sources, as the native Parquet scan already does in `native/core/src/parquet/objectstore/s3.rs` (instance profile, container, environment, web identity, profile, simple, temporary and anonymous). An unrecognized provider class still falls back. 3. **Server-side encryption, ACL, storage class, requester pays and `dsse-kms`.** OpenDAL's S3 backend supports all of these (`server_side_encryption*`, `default_acl`, `default_storage_class`, `enable_request_payer`), but iceberg-rust's S3 property parser maps only the `s3.sse.type` values `none`, `s3`, `kms` and `custom`. Translate the S3A encryption settings to `s3.sse.*`, and for the rest either extend iceberg-rust's parser upstream or set the options in Comet's own S3 storage factory. Each admitted or translated setting needs a `CometIcebergWriteDetectionSuite` case, and the translated ones a write through both writers against MinIO. ### Additional context Part of #5643 and the native write epic #5649. S3 object tags, access points, transfer acceleration, dual-stack and cross-region endpoints, remote signing and custom S3A signers stay a fallback: Comet's native S3 client has no equivalent for them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
