SteNicholas commented on code in PR #559: URL: https://github.com/apache/paimon-cpp/pull/559#discussion_r4640152422
########## src/paimon/core/table/source/raw_full_text_search_split_impl.h: ########## @@ -0,0 +1,94 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +#pragma once + +#include <cstdint> +#include <memory> +#include <optional> +#include <string> +#include <utility> +#include <vector> + +#include "paimon/result.h" +#include "paimon/table/source/full_text_search_split.h" +#include "paimon/utils/range.h" + +namespace paimon { +/// Carries the index type so a raw split can be read independently of its plan's index splits. +class RawFullTextSearchSplitImpl : public RawFullTextSearchSplit { + public: + /// "PFTRAWS1" in ASCII. + static constexpr int64_t MAGIC = 0x5046545241575331L; + /// Version 2 adds the text field ID. Version 3 adds the planning schema ID. + static constexpr int32_t VERSION = 3; Review Comment: Raw splits now start at V1 and require both the text field ID and the planning schema ID. I removed the draft layouts. The round-trip test checks the exact V1 bytes, and the decoder rejects other raw versions. [Round-trip test](https://github.com/apache/paimon-cpp/blob/91a569c54b76cbea14969785f88a7a9253d4ee09/src/paimon/core/table/source/full_text_search_split_test.cpp#L480-L505) ########## src/paimon/core/table/source/index_full_text_search_split_impl.h: ########## @@ -54,12 +54,14 @@ class IndexFullTextSearchSplitImpl : public IndexFullTextSearchSplit { /// least one and all within the shard range. /// @param scalar_index_files The global index files that intersect the shard range and can /// pre-filter its rows, each accepted by `CheckScalarIndexFile()`. + /// @param schema_id The schema used when planning, absent from versions 1 and 2. /// @return The split, or `Invalid` if an argument breaks these rules. static Result<std::shared_ptr<IndexFullTextSearchSplitImpl>> Create( int64_t snapshot_id, const std::string& column_name, int64_t row_range_start, int64_t row_range_end, std::vector<std::shared_ptr<IndexFileMeta>> full_text_index_files, std::vector<Range> search_row_ranges, - std::vector<std::shared_ptr<IndexFileMeta>> scalar_index_files = {}); + std::vector<std::shared_ptr<IndexFileMeta>> scalar_index_files = {}, + std::optional<int64_t> schema_id = std::nullopt); Review Comment: Removed both defaults from `IndexFullTextSearchSplitImpl::Create()` and updated every caller to pass `scalar_index_files` and `schema_id` explicitly. The planner passes its schema ID, and the decoder passes `std::nullopt` for stored V1 and V2 splits. [Factory signature](https://github.com/apache/paimon-cpp/blob/91a569c54b76cbea14969785f88a7a9253d4ee09/src/paimon/core/table/source/index_full_text_search_split_impl.h#L59-L64) ########## cmake_modules/lucene.diff: ########## @@ -0,0 +1,43 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Lucene++ 3.0.9 uses pointer member access on reference parameters in these templates. + +diff --git a/include/lucene++/Collection.h b/include/lucene++/Collection.h +--- a/include/lucene++/Collection.h ++++ b/include/lucene++/Collection.h +@@ -167,7 +167,7 @@ + } + + void swap(this_type& other) { +- container.swap(other->container); ++ container.swap(other.container); Review Comment: The production build fails before either swap needs to be instantiated. I compiled Lucene++ 3.0.9’s production precompiled header, `LuceneInc.h`, with Apple Clang. It rejected `other->container` in `Collection.h:170` and `other->setContainer` in `Set.h:111` because `other` is a reference. Changing both accesses to `.` makes the same header compile. I documented that build requirement in the patch. [Lucene++ patch](https://github.com/apache/paimon-cpp/blob/91a569c54b76cbea14969785f88a7a9253d4ee09/cmake_modules/lucene.diff#L19-L42) ########## src/paimon/core/table/source/data_evolution_full_text_read.cpp: ########## @@ -416,20 +564,42 @@ Result<std::shared_ptr<ScoredGlobalIndexResult>> DataEvolutionFullTextRead::Read }); } std::vector<SearchResult> results; + std::vector<std::future<SearchResult>> futures; if (searches.size() <= 1) { for (const auto& search : searches) { results.push_back(search()); } } else { - std::vector<std::future<SearchResult>> futures; futures.reserve(searches.size()); for (const auto& search : searches) { futures.push_back(Via(executor_.get(), search)); } + } + std::optional<SearchResult> raw_result; + if (!raw_ranges.empty()) { + RawFullTextReadImpl raw_read(root_path_, read_schema, options_, partition_filter_, filter_, + text_field_, query_, limit_, executor_, pool_); + raw_result = raw_read.Search(*snapshot_id, raw_index_type, raw_rows); Review Comment: A `ScopeGuard` now waits on every outstanding shard future before the reader can be destroyed. It is installed before the first submission. The regression test rejects the second shard submission and, separately, the first raw-manifest submission, then checks that `Read()` waits for the accepted tasks in both cases. [Guard](https://github.com/apache/paimon-cpp/blob/91a569c54b76cbea14969785f88a7a9253d4ee09/src/paimon/core/table/source/data_evolution_full_text_read.cpp#L615-L643) · [Regression test](https://github.com/apache/paimon-cpp/blob/91a569c54b76cbea14969785f88a7a9253d4ee09/src/paimon/core/table/source/data_evolution_full_text_read_test.cpp#L517-L568) ########## cmake_modules/ThirdpartyToolchain.cmake: ########## @@ -882,10 +882,13 @@ macro(build_lucene) "-DCMAKE_INSTALL_PREFIX=${LUCENE_PREFIX}") set(LUCENE_LIB "${LUCENE_PREFIX}/lib/liblucene++.a") + set(LUCENE_PATCH_FILE "${CMAKE_CURRENT_LIST_DIR}/lucene.diff") externalproject_add(lucene_ep ${EP_COMMON_OPTIONS} URL ${LUCENE_SOURCE_URL} URL_HASH "SHA256=${PAIMON_LUCENE_BUILD_SHA256_CHECKSUM}" + PATCH_COMMAND bash -c Review Comment: Both `lucene_ep` and `full_text_ep` now use the helpers from #570. Each gets its patch command from `paimon_patch_command` and registers its patch, install directory, and version inputs with `paimon_register_patched_project`. [Lucene++](https://github.com/apache/paimon-cpp/blob/91a569c54b76cbea14969785f88a7a9253d4ee09/cmake_modules/ThirdpartyToolchain.cmake#L890-L923) · [Paimon Full Text](https://github.com/apache/paimon-cpp/blob/91a569c54b76cbea14969785f88a7a9253d4ee09/cmake_modules/ThirdpartyToolchain.cmake#L1508-L1536) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
