comphead commented on PR #7603: URL: https://github.com/apache/datafusion-comet/pull/7603#issuecomment-6883287524
Thanks @andygrove. Both nits are addressed in d4f38d3949, and the second one turned out to be a real gap. **The audit claim.** Agreed. The test only enumerates the static `String` fields of `ParquetOutputFormat` and `ZstandardCodec`. Its byte comparison does catch iceberg-java starting to read one of the inert keys, because each one gets a value that would change the file. A key declared on another class, or read only by Iceberg's own code, is invisible to it. The contributor guide now says that instead of "cannot pass unaudited". The paragraph listing the Iceberg pins asks a bump to re-read how `ParquetWriter` and `Parquet.WriteBuilder` read their Hadoop `Configuration`, and the comment on `UnsupportedParquetMrKeys` matches. **Multipart and timeouts.** I checked these against the pinned iceberg-rust (`de0a6ac`), OpenDAL 0.58.2 and hadoop-aws 3.4.2. - `multipart.uploads.enabled`: ignoring it was wrong. With `false`, `S3ABlockOutputStream` keeps the whole file in one block and sends one PUT. iceberg-rust opens a file with `op.writer(path)` and no options, so OpenDAL buffers 5 MiB chunks (`write_multi_min_size`), and its `MultipartWriter` calls `initiate_part` as soon as a second chunk arrives. Neither OpenDAL's `S3Config` nor iceberg-rust's S3 properties can turn that off. Against a store without multipart uploads, a native write of any file over 5 MiB would fail where iceberg-java succeeds. So `false` now falls back, globally or for the data bucket. `true` stays admitted, following how `impl` is handled. The existing S3A test in `CometIcebergWriteDetectionSuite` covers both values in both spellings. - Timeouts: the native writer never reads the S3A values. iceberg-rust wraps the operator in OpenDAL's `TimeoutLayer`, set from `opendal.io-timeout-ms`, which Comet takes from `spark.comet.iceberg.ioTimeout` (30 s by default), and in a `RetryLayer`. One timed call can carry a whole row group, because parquet's `AsyncArrowWriter` hands each flushed row group to the writer in one write. A slow endpoint therefore needs `spark.comet.iceberg.ioTimeout` raised, whatever the S3A values are, and the S3A defaults (60 s per request in 3.4) are more generous than ours. - `connection.expect.continue`: neither OpenDAL nor reqwest sends `Expect: 100-continue`, so the native writer already behaves like `false`, which is the value stores that mishandle the header need. The user guide now says plainly that the native writer applies none of the ignored S3A values. It also names `spark.comet.iceberg.ioTimeout` as the setting to raise, and explains the multipart fallback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
