SteNicholas commented on code in PR #471:
URL: https://github.com/apache/paimon-cpp/pull/471#discussion_r4388639117
##########
src/paimon/format/vortex/vortex_file_batch_reader.cpp:
##########
@@ -448,24 +408,42 @@ Result<std::unique_ptr<::ArrowSchema>>
VortexFileBatchReader::GetFileSchema() co
Status VortexFileBatchReader::SetReadSchema(
::ArrowSchema* read_schema, const std::shared_ptr<Predicate>& predicate,
const std::optional<RoaringBitmap32>& selection_bitmap) {
+ if (closed_) {
+ return Status::Invalid("Vortex reader is closed");
+ }
if (read_schema == nullptr) {
return Status::Invalid("Vortex read schema is nullptr");
}
- // TODO(zhouyc-ali): push the row selection into the Vortex scan
- (void)selection_bitmap;
- // No pushdown: the whole file is scanned and reading restarts from row 0,
so drop the
- // single-use scan; the next read re-creates it.
+ // Keep selected physical row ids to map output positions after selection
pushdown.
+ std::vector<uint64_t> selection_row_ids;
+ if (selection_bitmap) {
+
selection_row_ids.reserve(static_cast<size_t>(selection_bitmap->Cardinality()));
Review Comment:
[P2] Preserve compressed selections for dense bitmap scans
This expands every selected row into a retained uint64_t, and
vx_data_source_scan then makes another full copy through Buffer::copy_from.
This also affects ordinary deletion-vector reads: AbstractSplitRead flips the
deletion bitmap into the surviving-row selection, so a file with just one
deleted row selects almost the entire file. With 10 million rows and one
deletion, I reproduced a 10,352-byte serialized bitmap expanding into a
79,999,992-byte C++ vector; the native scan adds another equally sized index
buffer before the first batch. This makes memory scale with the entire file's
surviving row count and can exhaust memory for large files or concurrent
readers. Please preserve the compressed bitmap/ranges across the FFI, or use
exclusion indices for dense selections, while retaining accurate physical row
mapping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
