andygrove opened a new issue, #6929: URL: https://github.com/apache/datafusion-comet/issues/6929
### What is the problem the feature request solves? #6355 adds native Parquet writes to S3 (#6007). It uploads through `object_store`, and it falls back to Spark's writer whenever the S3A magic committer is in use: when `fs.s3a.committer.name` or `fs.s3a.bucket.<bucket>.committer.name` is `magic`, or when the path is under a `__magic*` directory. The fallback is needed for correctness. S3A treats a file created under `__magic_job-<id>/` (`__magic/` on older Hadoop releases) specially: it uploads the data to the file's final key as a multipart upload that it leaves incomplete, and records a pending commit that task commit collects and job commit completes. `object_store` bypasses S3A, so a native write to the path `newTaskTempFile` returns would create an ordinary object inside the magic directory instead. The committer never commits it and job cleanup deletes it, so the job would succeed without its data. That leaves native S3 writes unavailable wherever the magic committer is configured, and Spark 4.1 configures it by default. When spark-hadoop-cloud is on the classpath, `SparkContext.enableMagicCommitterIfNeeded` (SPARK-47618) sets `spark.hadoop.fs.s3a.committer.name=magic`, `spark.sql.sources.commitProtocolClass=PathOutputCommitProtocol` and `spark.sql.parquet.output.committer.class=BindingParquetOutputCommitter` unless they are already set. Earlier versions (SPARK-35383) do this only when a bucket sets `spark.hadoop.fs.s3a.bucket.<bucket>.committer.magic.enabled=true`. spark-hadoop-cloud is how Spark SQL gets the S3A committers, so on Spark 4.1 deployments that include it, native S3 writes fall back unless the user turns the magic committer off or chooses another committer. ### Describe the potential solution The magic committer needs S3A itself to create each file. Two ways to get there: - **Write through S3A.** The native writer produces the Parquet bytes and the executor writes them with Hadoop's `FileSystem.create`, so `S3AFileSystem` does the upload and the magic-path handling. Comet's libhdfs route (`fs.comet.libhdfs.schemes`) already reaches Hadoop FileSystems through JNI and is one candidate transport, although #6355 currently declines S3 writes routed through it. This would also apply the S3A write settings #6355 falls back on (server-side encryption, ACLs, storage class, content encoding and `fs.s3a.create.header.*`), which raises a broader question: should all S3A writes take this route, trading `object_store`'s upload path for S3A compatibility? That needs benchmarks. - **Reproduce the magic committer's protocol natively.** Start the multipart upload to the final key and write the `.pending` file S3A would write. This ties Comet to Hadoop-internal formats, and it does not work when `fs.s3a.committer.magic.track.commits.in.memory.enabled` is set, because task commit then reads pending commits from S3A's in-memory tracker rather than from `.pending` files. The first seems more robust. ### Additional context - This would be Spark 4.0+ only. On Spark 3.x, `CometNativeWriteExec` builds its Hadoop contexts with `Job.getInstance()`, without the session's Hadoop settings or an output path, so its commit protocol resolves to a `FileOutputCommitter` with no output path and files go straight into the destination (#3015). No S3A committer is consulted there. - Testing: Comet's test classpath has only spark-hadoop-cloud's `tests` jar, so `PathOutputCommitProtocol` and `BindingParquetOutputCommitter` cannot load and Spark 4.1's default never applies in tests. Covering this needs the main jar in test scope, which turns on the 4.1 magic defaults for every test session unless the shared test configuration opts out, and a MinIO test that commits a write through `PathOutputCommitProtocol` with the magic committer. Depends on #6355. Part of #1625. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
