JingsongLi commented on PR #9100:
URL: https://github.com/apache/paimon/pull/9100#issuecomment-5218243238
Thanks for working on this. I wonder whether
`BundleRecords.isDirectWriteBundle()` is needed at all.
As I understand it, this flag does not actually guarantee a zero-copy or
native write. It only controls whether `RowDataFileWriter` may bypass its
per-record path and forward the bundle to `SingleFileWriter`. Once the
per-record consumers have been ruled out (auxiliary writers, row tracking, and
per-record statistics), it should be safe to forward any `BundleRecords`:
```java
long rowCount = bundle.rowCount();
if (auxiliaryFileWriters.isEmpty()
&& sequenceNumberTracker.supportsRowCountUpdate()
&& !requiresPerRecordStats()) {
super.writeBundle(bundle);
sequenceNumberTracker.updateByRowCount(rowCount);
return;
}
for (InternalRow row : bundle) {
write(row);
}
```
The existing abstractions already provide the remaining dispatch:
- `SingleFileWriter.writeBundle` falls back to row iteration when the format
writer is not a `BundleFormatWriter`.
- A `BundleFormatWriter` can inspect the bundle type and decide whether to
consume it natively, convert or copy it, or fall back to `addElement` row by
row.
- If the underlying native write is asynchronous, the format writer can copy
the borrowed data or acquire independent ownership before returning.
In other words, whether to copy or use a native batch path looks like a
format-writer implementation detail. The extra boolean duplicates that dispatch
and is slightly misleading: `true` still does not mean that the write is direct
or zero-copy.
Could we remove `isDirectWriteBundle()` and instead strengthen the contract
of `BundleFormatWriter.writeBundle()` to require semantic equivalence with
row-by-row writes and prohibit retaining borrowed buffers after the method
returns unless they have been copied or independently retained? This would keep
`BundleRecords` as a data abstraction and leave the write strategy entirely to
the format writer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
