comphead commented on PR #7603:
URL: 
https://github.com/apache/datafusion-comet/pull/7603#issuecomment-6883287524

   Thanks @andygrove. Both nits are addressed in d4f38d3949, and the second one 
turned out to be a real gap.
   
   **The audit claim.** Agreed. The test only enumerates the static `String` 
fields of `ParquetOutputFormat` and `ZstandardCodec`. Its byte comparison does 
catch iceberg-java starting to read one of the inert keys, because each one 
gets a value that would change the file. A key declared on another class, or 
read only by Iceberg's own code, is invisible to it. The contributor guide now 
says that instead of "cannot pass unaudited". The paragraph listing the Iceberg 
pins asks a bump to re-read how `ParquetWriter` and `Parquet.WriteBuilder` read 
their Hadoop `Configuration`, and the comment on `UnsupportedParquetMrKeys` 
matches.
   
   **Multipart and timeouts.** I checked these against the pinned iceberg-rust 
(`de0a6ac`), OpenDAL 0.58.2 and hadoop-aws 3.4.2.
   
   - `multipart.uploads.enabled`: ignoring it was wrong. With `false`, 
`S3ABlockOutputStream` keeps the whole file in one block and sends one PUT. 
iceberg-rust opens a file with `op.writer(path)` and no options, so OpenDAL 
buffers 5 MiB chunks (`write_multi_min_size`), and its `MultipartWriter` calls 
`initiate_part` as soon as a second chunk arrives. Neither OpenDAL's `S3Config` 
nor iceberg-rust's S3 properties can turn that off. Against a store without 
multipart uploads, a native write of any file over 5 MiB would fail where 
iceberg-java succeeds. So `false` now falls back, globally or for the data 
bucket. `true` stays admitted, following how `impl` is handled. The existing 
S3A test in `CometIcebergWriteDetectionSuite` covers both values in both 
spellings.
   - Timeouts: the native writer never reads the S3A values. iceberg-rust wraps 
the operator in OpenDAL's `TimeoutLayer`, set from `opendal.io-timeout-ms`, 
which Comet takes from `spark.comet.iceberg.ioTimeout` (30 s by default), and 
in a `RetryLayer`. One timed call can carry a whole row group, because 
parquet's `AsyncArrowWriter` hands each flushed row group to the writer in one 
write. A slow endpoint therefore needs `spark.comet.iceberg.ioTimeout` raised, 
whatever the S3A values are, and the S3A defaults (60 s per request in 3.4) are 
more generous than ours.
   - `connection.expect.continue`: neither OpenDAL nor reqwest sends `Expect: 
100-continue`, so the native writer already behaves like `false`, which is the 
value stores that mishandle the header need.
   
   The user guide now says plainly that the native writer applies none of the 
ignored S3A values. It also names `spark.comet.iceberg.ioTimeout` as the 
setting to raise, and explains the multipart fallback.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to