qzyu999 commented on code in PR #23656:
URL: https://github.com/apache/datafusion/pull/23656#discussion_r3679406487
##########
datafusion/datasource/src/sink.rs:
##########
@@ -40,6 +41,32 @@ use async_trait::async_trait;
use datafusion_physical_plan::execution_plan::{EvaluationType, SchedulingType};
use futures::StreamExt;
+/// Metadata about a single file produced by a [`DataSink`] write operation.
+///
+/// This struct is format-agnostic. The [`Self::format_metadata`] field carries
+/// serialized format-specific metadata (e.g., a Parquet file footer serialized
+/// via Thrift Compact Protocol).
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct FileWriteMetadata {
+ /// Object-store path where the file was written.
+ pub path: String,
+ /// Number of rows written to this specific file.
+ pub row_count: u64,
+ /// Sum of compressed row group sizes in bytes.
+ ///
+ /// Note: this may differ slightly from the actual on-disk file size as it
+ /// excludes the Parquet footer, page indexes, and other metadata overhead.
+ pub byte_size: u64,
+ /// Format-specific metadata serialized as bytes.
+ ///
+ /// For Parquet files this contains the Thrift-serialized `FileMetaData`
+ /// (the same bytes found in the Parquet footer), enabling consumers to
+ /// reconstruct column statistics without re-reading the file.
+ ///
+ /// For formats that do not produce file-level metadata this is `None`.
+ pub format_metadata: Option<Bytes>,
Review Comment:
Hi @timsaucer, my thoughts are that the primary consumers
(Iceberg/Delta/Hudi) need Thrift-serialized Parquet `FileMetaData` as-is to
extract column statistics (min/max bounds are binary, not strings). A
`HashMap<String, String>` would force lossy round-tripping and require each
consumer to re-parse. With opaque bytes, the sink implementation controls
serialization and the consumer deserializes with the appropriate codec leading
to zero data loss and no extra allocations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
