wangzhigang1999 opened a new pull request, #9948: URL: https://github.com/apache/paimon/pull/9948
### Purpose Closes #9947. `DataVectorWriter` and `DedicatedFormatWriter` duplicate normal-file output, buffering, rolling checks, and prepare/close handling. Consolidate the shared lifecycle and state in `CompositeDataWriter`, keeping column splitting, consistency checks, child-writer management, and blob/video-specific rules in the concrete writers. Share physical format dispatch through `DataWriter`. Prepare and validate all sidecar metadata before publishing it, then transfer abort cleanup responsibility through `_release_prepared_files()`. Preserve metadata ordering, lower-level writer retry behavior, externally owned Blob files, and the cleanup from #9917. No public API change is intended. ### Tests - Base and candidate each passed 421 related tests on Linux / Python 3.12.8 / PyArrow 19.0.1, with no skips. Lance and Vortex dependencies were installed. Reuse existing tests, extending the metadata-order case to check a vector preparation failure followed by retry; no new test files. - Eight real-file failure probes passed across both versions and both composite writers: no partial metadata publication, retry without rewriting prepared files, and removal of owned files on abort. The existing one-shot `BatchTableWrite.prepare_commit()` contract is unchanged. - Five workloads derived from the complete 60,000-row MNIST training split passed exact readback, logical metadata comparison, and cross-version reads. Seven alternating runs per workload showed median write-to-close times within 0.9% of the base; Blob + Vector had a +1.17% median paired change, while Lance showed wider variation. These measurements use a warm local filesystem and do not establish OSS or concurrent-writer performance. - Separate perf and cProfile runs found the main costs in Parquet encoding and per-row Blob processing. Shared lifecycle methods were not a major hotspot. Flake8 and whitespace checks passed. Validation baseline: `d0d2dec0ae4c568fd1b1f92d5b77d8376d286196`. Benchmark and profiling scripts are outside this patch. No performance improvement is claimed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
