xudong963 commented on PR #27027: URL: https://github.com/apache/datafusion/pull/27027#issuecomment-6620366689
Good call, thanks. I re-checked and there are real users of `written()` after all: InfluxDB 3 (`parquet_file/src/serialize.rs`), `datafusion_iceberg` (builds its `DataFile`s from it) and LakeSoul (`file_sink_writer.rs`). All of them construct the `ParquetSink` themselves and write one or a few files per sink, so the retention is harmless there. I've switched the PR to what you suggested: `written()` and its default are unchanged, there is a new `with_retain_written_metadata(false)` opt-out, and `ParquetFormat::create_writer_physical_plan` uses it for the `COPY` / `INSERT` sinks, where nothing reads the metadata. The `JoinSet` buffering is fixed either way: the writer task now stores or drops the metadata itself and only returns the row and byte counts. A per-file callback for #23472 can come separately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
