zhouyc-ali commented on code in PR #471:
URL: https://github.com/apache/paimon-cpp/pull/471#discussion_r4465886733
##########
docs/source/user_guide/read.rst:
##########
@@ -41,6 +41,26 @@ cross-language task scheduling and interaction (e.g., Java
and C++), substantial
across the two language ecosystems.
+Vortex Read Options
+-------------------
+
+For Vortex files, pass these options through ``ReadContextBuilder::AddOption``:
+
+- ``vortex.read.membership-pushdown`` accepts ``auto`` (default), ``force`` or
``off``.
+ ``auto`` limits each IN/NOT IN expression to 64 comparison and boolean nodes
after
+ deduplication and integer-range compression. ``force`` bypasses this budget
while
+ retaining type and value safety checks. ``off`` disables only IN/NOT IN
pushdown.
+- ``vortex.read.io-coalesce`` accepts ``object-storage`` (default) or
``local``.
+ The default retains Vortex's object-storage coalescing policy. ``local``
uses a
+ 64 KiB maximum gap and a 4 MiB coalescing limit for low-latency storage.
+ Choose the mode explicitly; it is not selected from the file path.
+
+Native pushdown may return candidate rows requiring further filtering. For
exact
+results, retain the full residual predicate: enable
``EnablePredicateFilter(true)``
+with ``TableRead``, or apply equivalent filtering in the caller. ``force``
does not
+bypass this requirement.
+
Review Comment:
Moved configuration keys, defaults and option structs into `vortex_defs.h`.
The 1 MiB/16 MiB I/O defaults match Java/JNI's object-store path. I/O limits
and the IN/NOT IN limit are now numeric options.
Moved Vortex options to `api/file_format.rst` and removed all Vortex
additions from `read.rst`.
##########
src/paimon/format/vortex/vortex_file_batch_reader.h:
##########
@@ -50,7 +52,9 @@ class VortexFileBatchReader : public FileBatchReader {
public:
static Result<std::unique_ptr<VortexFileBatchReader>> Create(
const std::shared_ptr<InputStream>& input, int32_t batch_size,
- const std::shared_ptr<arrow::MemoryPool>& arrow_pool);
+ const std::shared_ptr<arrow::MemoryPool>& arrow_pool,
+ const VortexPredicateOptions& predicate_options = {},
+ VortexIoCoalesceMode coalesce_mode =
VortexIoCoalesceMode::OBJECT_STORAGE);
Review Comment:
Removed default arguments and placed `arrow_pool` last in the reader
signatures and call sites.
##########
src/paimon/format/vortex/vortex_file_batch_reader.h:
##########
@@ -118,8 +118,21 @@ class VortexFileBatchReader : public FileBatchReader {
uint64_t previous_first_row_ = std::numeric_limits<uint64_t>::max();
uint64_t previous_batch_row_count_ = 0;
+ // Keep the smaller side of the bitmap. Dense selections store excluded
physical row ids,
+ // avoiding an O(file rows) index list both here and in the native scan's
copy.
+ bool has_selection_ = false;
+ bool selection_excludes_ = false;
+ uint64_t selected_row_count_ = 0;
+ std::vector<uint64_t> selection_indices_;
+
+ // A pushed filter returns physical row ids in a hidden _row_idx column.
Retain the last
+ // batch's column because emitted offsets no longer identify physical rows
after filtering.
+ bool has_filter_ = false;
+ std::shared_ptr<arrow::Array> previous_batch_row_idx_;
+
std::shared_ptr<arrow::MemoryPool> arrow_pool_;
- std::shared_ptr<PredicateFilter> predicate_filter_; // from
SetReadSchema; not pushed down
+ VortexPredicateOptions predicate_options_;
+ std::shared_ptr<PredicateFilter> predicate_filter_;
Review Comment:
Encapsulated selection and physical row tracking in `SelectionState` and
`RowPositionState`, including validation and mapping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]