SteNicholas commented on code in PR #559:
URL: https://github.com/apache/paimon-cpp/pull/559#discussion_r4640152422


##########
src/paimon/core/table/source/raw_full_text_search_split_impl.h:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "paimon/result.h"
+#include "paimon/table/source/full_text_search_split.h"
+#include "paimon/utils/range.h"
+
+namespace paimon {
+/// Carries the index type so a raw split can be read independently of its 
plan's index splits.
+class RawFullTextSearchSplitImpl : public RawFullTextSearchSplit {
+ public:
+    /// "PFTRAWS1" in ASCII.
+    static constexpr int64_t MAGIC = 0x5046545241575331L;
+    /// Version 2 adds the text field ID. Version 3 adds the planning schema 
ID.
+    static constexpr int32_t VERSION = 3;

Review Comment:
   Raw splits now start at V1 and require both the text field ID and the 
planning schema ID. I removed the draft layouts. The round-trip test checks the 
exact V1 bytes, and the decoder rejects other raw versions. [Round-trip 
test](https://github.com/apache/paimon-cpp/blob/91a569c54b76cbea14969785f88a7a9253d4ee09/src/paimon/core/table/source/full_text_search_split_test.cpp#L480-L505)



##########
src/paimon/core/table/source/index_full_text_search_split_impl.h:
##########
@@ -54,12 +54,14 @@ class IndexFullTextSearchSplitImpl : public 
IndexFullTextSearchSplit {
     ///                          least one and all within the shard range.
     /// @param scalar_index_files The global index files that intersect the 
shard range and can
     ///                          pre-filter its rows, each accepted by 
`CheckScalarIndexFile()`.
+    /// @param schema_id         The schema used when planning, absent from 
versions 1 and 2.
     /// @return The split, or `Invalid` if an argument breaks these rules.
     static Result<std::shared_ptr<IndexFullTextSearchSplitImpl>> Create(
         int64_t snapshot_id, const std::string& column_name, int64_t 
row_range_start,
         int64_t row_range_end, std::vector<std::shared_ptr<IndexFileMeta>> 
full_text_index_files,
         std::vector<Range> search_row_ranges,
-        std::vector<std::shared_ptr<IndexFileMeta>> scalar_index_files = {});
+        std::vector<std::shared_ptr<IndexFileMeta>> scalar_index_files = {},
+        std::optional<int64_t> schema_id = std::nullopt);

Review Comment:
   Removed both defaults from `IndexFullTextSearchSplitImpl::Create()` and 
updated every caller to pass `scalar_index_files` and `schema_id` explicitly. 
The planner passes its schema ID, and the decoder passes `std::nullopt` for 
stored V1 and V2 splits. [Factory 
signature](https://github.com/apache/paimon-cpp/blob/91a569c54b76cbea14969785f88a7a9253d4ee09/src/paimon/core/table/source/index_full_text_search_split_impl.h#L59-L64)



##########
cmake_modules/lucene.diff:
##########
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Lucene++ 3.0.9 uses pointer member access on reference parameters in these 
templates.
+
+diff --git a/include/lucene++/Collection.h b/include/lucene++/Collection.h
+--- a/include/lucene++/Collection.h
++++ b/include/lucene++/Collection.h
+@@ -167,7 +167,7 @@
+     }
+
+     void swap(this_type& other) {
+-        container.swap(other->container);
++        container.swap(other.container);

Review Comment:
   The production build fails before either swap needs to be instantiated. I 
compiled Lucene++ 3.0.9’s production precompiled header, `LuceneInc.h`, with 
Apple Clang. It rejected `other->container` in `Collection.h:170` and 
`other->setContainer` in `Set.h:111` because `other` is a reference. Changing 
both accesses to `.` makes the same header compile. I documented that build 
requirement in the patch. [Lucene++ 
patch](https://github.com/apache/paimon-cpp/blob/91a569c54b76cbea14969785f88a7a9253d4ee09/cmake_modules/lucene.diff#L19-L42)



##########
src/paimon/core/table/source/data_evolution_full_text_read.cpp:
##########
@@ -416,20 +564,42 @@ Result<std::shared_ptr<ScoredGlobalIndexResult>> 
DataEvolutionFullTextRead::Read
         });
     }
     std::vector<SearchResult> results;
+    std::vector<std::future<SearchResult>> futures;
     if (searches.size() <= 1) {
         for (const auto& search : searches) {
             results.push_back(search());
         }
     } else {
-        std::vector<std::future<SearchResult>> futures;
         futures.reserve(searches.size());
         for (const auto& search : searches) {
             futures.push_back(Via(executor_.get(), search));
         }
+    }
+    std::optional<SearchResult> raw_result;
+    if (!raw_ranges.empty()) {
+        RawFullTextReadImpl raw_read(root_path_, read_schema, options_, 
partition_filter_, filter_,
+                                     text_field_, query_, limit_, executor_, 
pool_);
+        raw_result = raw_read.Search(*snapshot_id, raw_index_type, raw_rows);

Review Comment:
   A `ScopeGuard` now waits on every outstanding shard future before the reader 
can be destroyed. It is installed before the first submission. The regression 
test rejects the second shard submission and, separately, the first 
raw-manifest submission, then checks that `Read()` waits for the accepted tasks 
in both cases. 
[Guard](https://github.com/apache/paimon-cpp/blob/91a569c54b76cbea14969785f88a7a9253d4ee09/src/paimon/core/table/source/data_evolution_full_text_read.cpp#L615-L643)
 · [Regression 
test](https://github.com/apache/paimon-cpp/blob/91a569c54b76cbea14969785f88a7a9253d4ee09/src/paimon/core/table/source/data_evolution_full_text_read_test.cpp#L517-L568)



##########
cmake_modules/ThirdpartyToolchain.cmake:
##########
@@ -882,10 +882,13 @@ macro(build_lucene)
         "-DCMAKE_INSTALL_PREFIX=${LUCENE_PREFIX}")
 
     set(LUCENE_LIB "${LUCENE_PREFIX}/lib/liblucene++.a")
+    set(LUCENE_PATCH_FILE "${CMAKE_CURRENT_LIST_DIR}/lucene.diff")
     externalproject_add(lucene_ep
                         ${EP_COMMON_OPTIONS}
                         URL ${LUCENE_SOURCE_URL}
                         URL_HASH 
"SHA256=${PAIMON_LUCENE_BUILD_SHA256_CHECKSUM}"
+                        PATCH_COMMAND bash -c

Review Comment:
   Both `lucene_ep` and `full_text_ep` now use the helpers from #570. Each gets 
its patch command from `paimon_patch_command` and registers its patch, install 
directory, and version inputs with `paimon_register_patched_project`. 
[Lucene++](https://github.com/apache/paimon-cpp/blob/91a569c54b76cbea14969785f88a7a9253d4ee09/cmake_modules/ThirdpartyToolchain.cmake#L890-L923)
 · [Paimon Full 
Text](https://github.com/apache/paimon-cpp/blob/91a569c54b76cbea14969785f88a7a9253d4ee09/cmake_modules/ThirdpartyToolchain.cmake#L1508-L1536)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to