JingsongLi commented on code in PR #7598:
URL: https://github.com/apache/paimon/pull/7598#discussion_r3443076339
##########
paimon-core/src/main/java/org/apache/paimon/append/DedicatedFormatRollingFileWriter.java:
##########
@@ -394,16 +384,50 @@ private void handleWriteException(Throwable e) {
}
/**
- * Writes a bundle of records by iterating through each row.
+ * Writes a bundle of records while preserving bundle semantics for
dedicated child writers.
*
* @param bundle The bundle of records to write
* @throws IOException if writing fails
*/
@Override
public void writeBundle(BundleRecords bundle) throws IOException {
- // TODO: support bundle projection
- for (InternalRow row : bundle) {
- write(row);
+ if (bundle.rowCount() == 0) {
+ return;
+ }
+
+ if (externalStorageBlobWriter != null) {
+ for (InternalRow row : bundle) {
+ write(row);
+ }
+ return;
+ }
+
+ // Dedicated fan-out reuses the same logical bundle for
normal/blob/vector writers, but
+ // only explicitly replayable bundles can be forwarded safely without
copying.
+ ReplayableBundleRecords replayableBundle =
+ bundle instanceof ReplayableBundleRecords
+ ? (ReplayableBundleRecords) bundle
+ : MaterializedBundleRecords.from(bundle, writeSchema);
+
+ try {
+ openWritersIfNeeded();
+ if (blobWriter != null) {
+ blobWriter.writeBundle(replayableBundle);
Review Comment:
The new fan-out path writes the whole bundle into each rolling child writer
before any target-size check runs. `MultipleBlobFileWriter.writeBundle`
delegates to `RollingFileWriterImpl.writeBundle`, and that only calls
`reachTargetSize(true)` after `currentWriter.writeBundle(bundle)` has consumed
the entire batch. This regresses the old dedicated `writeBundle` path, which
iterated rows through `write(row)` and let blob/vector/main writers roll within
the bundle; a large bundle with blobs can now produce a single blob file far
beyond `blobTargetFileSize` (and similarly oversized main/vector files). Could
we preserve the fast path only for writers that can split/check inside the
bundle, or add chunking / a fallback so target-file-size rolling still happens
during large bundle writes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
