zhouyc-ali commented on code in PR #471:
URL: https://github.com/apache/paimon-cpp/pull/471#discussion_r4465886733


##########
docs/source/user_guide/read.rst:
##########
@@ -41,6 +41,26 @@ cross-language task scheduling and interaction (e.g., Java 
and C++), substantial
 across the two language ecosystems.
 
 
+Vortex Read Options
+-------------------
+
+For Vortex files, pass these options through ``ReadContextBuilder::AddOption``:
+
+- ``vortex.read.membership-pushdown`` accepts ``auto`` (default), ``force`` or 
``off``.
+  ``auto`` limits each IN/NOT IN expression to 64 comparison and boolean nodes 
after
+  deduplication and integer-range compression. ``force`` bypasses this budget 
while
+  retaining type and value safety checks. ``off`` disables only IN/NOT IN 
pushdown.
+- ``vortex.read.io-coalesce`` accepts ``object-storage`` (default) or 
``local``.
+  The default retains Vortex's object-storage coalescing policy. ``local`` 
uses a
+  64 KiB maximum gap and a 4 MiB coalescing limit for low-latency storage.
+  Choose the mode explicitly; it is not selected from the file path.
+
+Native pushdown may return candidate rows requiring further filtering. For 
exact
+results, retain the full residual predicate: enable 
``EnablePredicateFilter(true)``
+with ``TableRead``, or apply equivalent filtering in the caller. ``force`` 
does not
+bypass this requirement.
+

Review Comment:
   Moved configuration keys, defaults and option structs into `vortex_defs.h`.
   
   The 1 MiB/16 MiB I/O defaults match Java/JNI's object-store path. I/O limits 
and the IN/NOT IN limit are now numeric options.
   
   Moved Vortex options to `api/file_format.rst` and removed all Vortex 
additions from `read.rst`.



##########
src/paimon/format/vortex/vortex_file_batch_reader.h:
##########
@@ -50,7 +52,9 @@ class VortexFileBatchReader : public FileBatchReader {
  public:
     static Result<std::unique_ptr<VortexFileBatchReader>> Create(
         const std::shared_ptr<InputStream>& input, int32_t batch_size,
-        const std::shared_ptr<arrow::MemoryPool>& arrow_pool);
+        const std::shared_ptr<arrow::MemoryPool>& arrow_pool,
+        const VortexPredicateOptions& predicate_options = {},
+        VortexIoCoalesceMode coalesce_mode = 
VortexIoCoalesceMode::OBJECT_STORAGE);

Review Comment:
   Removed default arguments and placed `arrow_pool` last in the reader 
signatures and call sites.



##########
src/paimon/format/vortex/vortex_file_batch_reader.h:
##########
@@ -118,8 +118,21 @@ class VortexFileBatchReader : public FileBatchReader {
     uint64_t previous_first_row_ = std::numeric_limits<uint64_t>::max();
     uint64_t previous_batch_row_count_ = 0;
 
+    // Keep the smaller side of the bitmap. Dense selections store excluded 
physical row ids,
+    // avoiding an O(file rows) index list both here and in the native scan's 
copy.
+    bool has_selection_ = false;
+    bool selection_excludes_ = false;
+    uint64_t selected_row_count_ = 0;
+    std::vector<uint64_t> selection_indices_;
+
+    // A pushed filter returns physical row ids in a hidden _row_idx column. 
Retain the last
+    // batch's column because emitted offsets no longer identify physical rows 
after filtering.
+    bool has_filter_ = false;
+    std::shared_ptr<arrow::Array> previous_batch_row_idx_;
+
     std::shared_ptr<arrow::MemoryPool> arrow_pool_;
-    std::shared_ptr<PredicateFilter> predicate_filter_;  // from 
SetReadSchema; not pushed down
+    VortexPredicateOptions predicate_options_;
+    std::shared_ptr<PredicateFilter> predicate_filter_;

Review Comment:
   Encapsulated selection and physical row tracking in `SelectionState` and 
`RowPositionState`, including validation and mapping.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to