zjw1111 commented on code in PR #559:
URL: https://github.com/apache/paimon-cpp/pull/559#discussion_r4626607166


##########
src/paimon/core/table/source/index_full_text_search_split_impl.h:
##########
@@ -54,12 +54,14 @@ class IndexFullTextSearchSplitImpl : public 
IndexFullTextSearchSplit {
     ///                          least one and all within the shard range.
     /// @param scalar_index_files The global index files that intersect the 
shard range and can
     ///                          pre-filter its rows, each accepted by 
`CheckScalarIndexFile()`.
+    /// @param schema_id         The schema used when planning, absent from 
versions 1 and 2.
     /// @return The split, or `Invalid` if an argument breaks these rules.
     static Result<std::shared_ptr<IndexFullTextSearchSplitImpl>> Create(
         int64_t snapshot_id, const std::string& column_name, int64_t 
row_range_start,
         int64_t row_range_end, std::vector<std::shared_ptr<IndexFileMeta>> 
full_text_index_files,
         std::vector<Range> search_row_ranges,
-        std::vector<std::shared_ptr<IndexFileMeta>> scalar_index_files = {});
+        std::vector<std::shared_ptr<IndexFileMeta>> scalar_index_files = {},
+        std::optional<int64_t> schema_id = std::nullopt);

Review Comment:
   Could you remove both default arguments from `Create()` and pass 
`scalar_index_files` and `schema_id` explicitly at each call site?



##########
cmake_modules/lucene.diff:
##########
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Lucene++ 3.0.9 uses pointer member access on reference parameters in these 
templates.
+
+diff --git a/include/lucene++/Collection.h b/include/lucene++/Collection.h
+--- a/include/lucene++/Collection.h
++++ b/include/lucene++/Collection.h
+@@ -167,7 +167,7 @@
+     }
+
+     void swap(this_type& other) {
+-        container.swap(other->container);
++        container.swap(other.container);

Review Comment:
   Is there a production call path that instantiates `Collection::swap()` or 
`Set::swap()`? I only found calls in `TestContainerSwap`. Could you clarify why 
this patch is needed for the current full-text search path?



##########
src/paimon/core/table/source/raw_full_text_search_split_impl.h:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "paimon/result.h"
+#include "paimon/table/source/full_text_search_split.h"
+#include "paimon/utils/range.h"
+
+namespace paimon {
+/// Carries the index type so a raw split can be read independently of its 
plan's index splits.
+class RawFullTextSearchSplitImpl : public RawFullTextSearchSplit {
+ public:
+    /// "PFTRAWS1" in ASCII.
+    static constexpr int64_t MAGIC = 0x5046545241575331L;
+    /// Version 2 adds the text field ID. Version 3 adds the planning schema 
ID.
+    static constexpr int32_t VERSION = 3;

Review Comment:
   Could you start the raw split at V1 and require both the text field ID and 
the planning schema ID in that first version? This PR introduces the raw split, 
so the compatibility branches for the earlier, unreleased layouts can be 
removed.



##########
src/paimon/core/table/source/data_evolution_full_text_read.cpp:
##########
@@ -416,20 +564,42 @@ Result<std::shared_ptr<ScoredGlobalIndexResult>> 
DataEvolutionFullTextRead::Read
         });
     }
     std::vector<SearchResult> results;
+    std::vector<std::future<SearchResult>> futures;
     if (searches.size() <= 1) {
         for (const auto& search : searches) {
             results.push_back(search());
         }
     } else {
-        std::vector<std::future<SearchResult>> futures;
         futures.reserve(searches.size());
         for (const auto& search : searches) {
             futures.push_back(Via(executor_.get(), search));
         }
+    }
+    std::optional<SearchResult> raw_result;
+    if (!raw_ranges.empty()) {
+        RawFullTextReadImpl raw_read(root_path_, read_schema, options_, 
partition_filter_, filter_,
+                                     text_field_, query_, limit_, executor_, 
pool_);
+        raw_result = raw_read.Search(*snapshot_id, raw_index_type, raw_rows);

Review Comment:
   `Read()` submits shard tasks that capture `this` before calling 
`raw_read.Search()`. If manifest task submission throws during the raw scan, 
`Read()` exits before `CollectAll()`. `ExecuteLocal()` can then destroy the 
reader while a shard task still uses it. Could we drain the submitted futures 
on every exit path, as `FileStoreScan` does with `ScopeGuard`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to