SteNicholas commented on code in PR #471:
URL: https://github.com/apache/paimon-cpp/pull/471#discussion_r4388639117
##########
src/paimon/format/vortex/vortex_file_batch_reader.cpp:
##########
@@ -448,24 +408,42 @@ Result<std::unique_ptr<::ArrowSchema>>
VortexFileBatchReader::GetFileSchema() co
Status VortexFileBatchReader::SetReadSchema(
::ArrowSchema* read_schema, const std::shared_ptr<Predicate>& predicate,
const std::optional<RoaringBitmap32>& selection_bitmap) {
+ if (closed_) {
+ return Status::Invalid("Vortex reader is closed");
+ }
if (read_schema == nullptr) {
return Status::Invalid("Vortex read schema is nullptr");
}
- // TODO(zhouyc-ali): push the row selection into the Vortex scan
- (void)selection_bitmap;
- // No pushdown: the whole file is scanned and reading restarts from row 0,
so drop the
- // single-use scan; the next read re-creates it.
+ // Keep selected physical row ids to map output positions after selection
pushdown.
+ std::vector<uint64_t> selection_row_ids;
+ if (selection_bitmap) {
+
selection_row_ids.reserve(static_cast<size_t>(selection_bitmap->Cardinality()));
Review Comment:
**Preserve compressed selections for dense bitmap scans**. This expands
every selected row into a retained uint64_t, and vx_data_source_scan then makes
another full copy through Buffer::copy_from. This also affects ordinary
deletion-vector reads: AbstractSplitRead flips the deletion bitmap into the
surviving-row selection, so a file with just one deleted row selects almost the
entire file. With 10 million rows and one deletion, I reproduced a 10,352-byte
serialized bitmap expanding into a 79,999,992-byte C++ vector; the native scan
adds another equally sized index buffer before the first batch. This makes
memory scale with the entire file's surviving row count and can exhaust memory
for large files or concurrent readers. Please preserve the compressed
bitmap/ranges across the FFI, or use exclusion indices for dense selections,
while retaining accurate physical row mapping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
