leaves12138 commented on code in PR #1027:
URL: https://github.com/apache/paimon-rust/pull/1027#discussion_r4177014320
##########
crates/paimon/src/arrow/format/mod.rs:
##########
@@ -470,111 +470,179 @@ fn supported_write_formats() -> Vec<&'static str> {
]
}
-/// Create a format writer that streams directly to storage.
+/// Reusable factory for one rolling writer, mirroring Java
FormatWriterFactory.
+/// Output and compression belong to each file; schema/options belong to the
factory.
+#[async_trait]
+pub(crate) trait FormatWriterFactory: Send + Sync {
+ async fn create_writer(
+ &self,
+ output: &OutputFile,
+ compression: &str,
+ ) -> crate::Result<Box<dyn FormatFileWriter>>;
+
+ /// Some plan factories need the previous close callback before the next
file.
+ fn needs_completed_file_stats(&self) -> bool {
+ false
+ }
+}
+
+/// Build a factory once per rolling writer. Statistics describe logical value
+/// fields independently of a shredding plan's physical schema (notably for PK
files).
+pub(crate) fn create_format_writer_factory(
+ file_format: &str,
+ schema: SchemaRef,
+ zstd_level: i32,
+ file_io: Option<FileIO>,
+ write_fields: Option<&[DataField]>,
+ format_options: Option<&HashMap<String, String>>,
+ stats_fields: Option<&[DataField]>,
+) -> crate::Result<Arc<dyn FormatWriterFactory>> {
+ let file_format = file_format.rsplit('.').next().unwrap_or_default();
+ if file_format.eq_ignore_ascii_case("parquet") {
Review Comment:
This factory only wraps Parquet, while `spec::map_shredding::validate` still
explicitly accepts `orc` and Java's `OrcFileFormat` advertises
`MAP_SHARED_SHREDDING` as supported. With `file.format=orc` and MAP
shared-shredding enabled, Rust falls through to `PlainFormatWriterFactory`, so
it writes the logical MAP layout without the shredding conversion/footer
metadata (and the ORC read path has no corresponding MAP assembly). This
silently violates the configured layout and is inconsistent with Java. Please
either implement the ORC shredding writer/read path or reject ORC for MAP
shared-shredding in validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
