This is an automated email from the ASF dual-hosted git repository.

Gabriel39 pushed a commit to branch orc
in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git


The following commit(s) were added to refs/heads/orc by this push:
     new 4ec25901b12 [fix](orc) Share stripe statistics between range readers 
(#435)
4ec25901b12 is described below

commit 4ec25901b1266235f56ebe5c948277d6f3c2c616
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Wed Nov 18 11:13:10 2026 +0800

    [fix](orc) Share stripe statistics between range readers (#435)
    
    * [fix](orc) Share stripe statistics between range readers
    
    ### Background
    
    Independent ORC readers use separate streams and predicate appliers. Stripe 
statistics live in a file-level Metadata protobuf, separate from the serialized 
file tail.
    
    ### What problem does this PR solve?
    
    Issue Number: None
    Related PR: apache/doris#70664 (physical stripe readers)
    PR Type: Bugfix
    
    Problem Summary:
    
    Physical stripe readers repeat complete Metadata parsing and all-file SARG 
evaluation. Eight one-stripe readers plus their planning reader can parse 
Metadata nine times and evaluate 72 stripe statistics. Share lazy immutable 
Metadata and add bounded stripe evaluation while preserving reader-local 
predicates and row-group filtering.
    
    ### Details
    
    - **Frequency:** Found by code review; triggered by independent readers 
with SARG over the same multi-stripe file.
    - **Root cause:** Serialized tails omit Metadata, and getNeedReadStripes 
evaluates the whole file regardless of the reader's range.
    - **Why existing tests missed it:** Predicate tests check pruning results 
without counting repeated setup across independent readers.
    - **Fix:** Add an opaque thread-safe lazy statistics handle, an interval 
overload and operation counters. A successful load publishes immutable 
Metadata; failed loads remain retryable. The handle retains no streams, 
allocators, metrics or SARG appliers.
    - **Other paths and branches:** The existing full-file API delegates to the 
interval implementation. No format or default changes; native consumers must 
rebuild for the extended interface.
    
    ### Testing
    
    Local Doris ASAN BE build passed with this SDK candidate. Standard BE unit 
tests, including the new concurrent cold/warm-handle test, are running; results 
will be updated before merge. The test checks one parse after parent 
destruction, bounded evaluation, independent predicates, row-group pruning and 
absolute row positions.
    
    ### Release note
    
    None
    
    ### Check List (For Author)
    
    - Behavior changed: No
    - Does this need documentation: No; SDK interface contracts are documented 
in the header.
    - Pre-submission status: Partially verified: unit tests are running.
    
    ### AI Disclosure
    
    - Tool: Codex
    - Generated: SDK implementation, tests and description.
    - Human review: not stated
    - Prompts: Fix F-01 from the review of apache/doris#70664 and update that 
PR.
    
    * [fix](orc) Preserve optional reader metrics
    
    ### Background
    
    Standalone ORC readers can leave ReaderOptions metrics unset. Statistics 
loading and predicate pruning support those callers.
    
    ### What problem does this PR solve?
    
    Issue Number: None
    Related PR: apache/doris#70664
    PR Type: Bugfix
    
    Problem Summary:
    
    The new statistics counters dereferenced the optional metrics pointer. Keep 
instrumentation conditional so default ReaderOptions continues to support 
statistics reads and SARG pruning.
    
    ### Details
    
    - **Frequency:** Always when the new counters run without configured 
metrics.
    - **Root cause:** The metrics pointer defaults to nullptr, but the added 
counters assumed it was configured.
    - **Why existing tests missed it:** The new counter tests supplied metrics 
explicitly; running the existing standalone-reader cases exposed the missing 
guards.
    - **Fix:** Follow SargsApplier's existing optional-metrics pattern for both 
Metadata parsing and stripe evaluation.
    - **Other paths and branches:** Both full-file and interval APIs use the 
guarded counters. No format or config changes.
    
    ### Testing
    
    All 238 selected Doris ORC and SDK reader/predicate tests passed locally 
under ASAN, including default ReaderOptions, concurrent cold/warm statistics 
sharing, and exact stripe ownership.
    
    ### Release note
    
    None
    
    ### Check List (For Author)
    
    - Behavior changed: No
    - Does this need documentation: No
    - Pre-submission status: Ready
    
    ### AI Disclosure
    
    - Tool: Codex
    - Generated: Metrics guards and this message.
    - Human review: not stated
    
    * [chore](orc) Declare thread linkage for statistics loading
    
    ### Background
    
    The shared statistics loader uses std::call_once. ORC already discovers the 
platform thread dependency during configuration.
    
    ### What problem does this PR solve?
    
    Issue Number: None
    Related PR: apache/doris#70664
    PR Type: Build/CI/Chore
    
    Problem Summary:
    
    Declare Threads::Threads on the ORC library so standalone native consumers 
inherit the loader's thread linkage.
    
    ### Details
    
    - **Affected:** ORC library link interface.
    - **Developer and release impact:** Standalone consumers receive the 
required platform thread flags; Doris already links threads.
    - **License and third-party:** No new dependency; use the existing 
discovered system thread target.
    - **Rollback:** Remove the target dependency together with the shared 
loader.
    
    ### Testing
    
    Reconfigured through the standard Doris BE unit script and passed all 238 
selected ORC reader/predicate tests under ASAN.
    
    ### Release note
    
    None
    
    ### Check List (For Author)
    
    - Behavior changed: No
    - Does this need documentation: No
    - Pre-submission status: Ready
    
    ### AI Disclosure
    
    - Tool: Codex
    - Generated: Link dependency and this message.
    - Human review: not stated
    
    * [fix](orc) Keep shared statistics retryable after load failures
    
    ### Background
    
    Independent readers share lazily parsed immutable stripe statistics.
    
    ### What problem does this PR solve?
    
    Issue Number: None
    Related PR: apache/doris#70664 (downstream physical stripe readers)
    PR Type: Bugfix
    
    Problem Summary:
    A throwing call_once initializer relies on runtime exception-reset behavior.
    Replace it with a mutex and an explicit loaded state so failed loaders 
cannot
    publish success and other readers can retry with their own streams.
    
    ### Details
    
    - **Frequency:** Reported in SDK review on an affected runtime combination; 
not reproduced on the local macOS runtime or in a running Doris BE.
    - **Root cause:** pthread_once-backed call_once implementations can fail to 
reset initialization state after an exception.
    - **Why existing tests missed it:** Successful cold/warm sharing tests did 
not inject Metadata load failures.
    - **Fix:** Protect shared state with a mutex, parse locally, and mark 
loaded only after success. Keep successful empty Metadata distinct from an 
unloaded handle.
    - **Other paths and branches:** All Metadata accessors use this loader. 
This corrects code introduced in the same unmerged SDK PR; no public API or 
file-format changes in this followup.
    
    ### Testing
    
    All 239 selected Doris/SDK reader and predicate tests passed under ASAN,
    including 196 Doris ORC tests. The new test injects both a read exception 
and an
    invalid protobuf tag after exporting a cold handle and destroying its 
parent.
    It checks retry through a different reader and cache reuse by the failed 
reader
    and a later sibling after both loaders are destroyed.
    
    The same new test passes on the unchanged call_once implementation locally;
    this is a conforming-runtime control, not a reproduced failing baseline.
    
    ### Release note
    
    None
    
    ### Check List (For Author)
    
    - Behavior changed: No
    - Does this need documentation: No; existing API contracts remain unchanged.
    - Pre-submission status: Partially verified: the affected runtime 
combination is unavailable locally.
    
    ### AI Disclosure
    
    - Tool: Codex
    - Generated: implementation, tests and description.
    - Human review: not stated
    - Prompts: Fix the review finding on apache/doris-thirdparty#435 and update 
that PR.
    
    ---------
    
    Co-authored-by: morningman <[email protected]>
---
 c++/include/orc/Reader.hh         |  23 ++++-
 c++/src/CMakeLists.txt            |   1 +
 c++/src/Options.hh                |  11 ++
 c++/src/Reader.cc                 |  87 +++++++++++-----
 c++/src/Reader.hh                 |  25 ++++-
 c++/test/TestPredicatePushdown.cc | 207 ++++++++++++++++++++++++++++++++++++++
 6 files changed, 324 insertions(+), 30 deletions(-)

diff --git a/c++/include/orc/Reader.hh b/c++/include/orc/Reader.hh
index 1ab60b1d2c5..e45f1ce450e 100644
--- a/c++/include/orc/Reader.hh
+++ b/c++/include/orc/Reader.hh
@@ -41,11 +41,16 @@ namespace orc {
   struct ReaderOptionsPrivate;
   struct RowReaderOptionsPrivate;
   class InputStream;
+  class SharedStripeStatistics;
   /**
    * Expose the reader metrics including the latency and
    * number of calls of the decompression/decoding/IO modules.
    */
   struct ReaderMetrics {
+    // Metadata protobuf parse attempts and getNeedReadStripes evaluations 
only.
+    // RowReader's range-local stripe/row-group PPD retains its existing 
metrics.
+    std::atomic<uint64_t> MetadataParseCount{0};
+    std::atomic<uint64_t> StripeStatisticsEvaluationCount{0};
     std::atomic<uint64_t> ReaderCall{0};
     // ReaderInclusiveLatencyUs contains the latency of
     // the decompression/decoding/IO modules.
@@ -96,6 +101,14 @@ namespace orc {
      */
     ReaderOptions& setSerializedFileTail(const std::string& serialization);
 
+    /**
+     * Share lazily loaded, immutable stripe statistics between independent 
readers.
+     * The handle must come from the same immutable file version as this 
reader's tail.
+     * Streams, metrics and predicate appliers remain reader-local.
+     */
+    ReaderOptions& 
setSharedStripeStatistics(std::shared_ptr<SharedStripeStatistics> statistics);
+    std::shared_ptr<SharedStripeStatistics> getSharedStripeStatistics() const;
+
     /**
      * Set the memory allocator.
      */
@@ -661,7 +674,15 @@ namespace orc {
 
     virtual std::vector<int> getNeedReadStripes(const RowReaderOptions& opts) 
= 0;
 
-    virtual void getSelectedColumns(const std::list<std::string>& names, 
std::vector<bool>& selectedColumns) = 0;
+    /** Return decisions for [firstStripe, firstStripe + stripeCount), indexed 
from zero. */
+    virtual std::vector<int> getNeedReadStripes(const RowReaderOptions& opts, 
uint64_t firstStripe,
+                                                uint64_t stripeCount) = 0;
+
+    /** Export a thread-safe lazy statistics handle without reading Metadata. 
*/
+    virtual std::shared_ptr<SharedStripeStatistics> 
getSharedStripeStatistics() const = 0;
+
+    virtual void getSelectedColumns(const std::list<std::string>& names,
+                                    std::vector<bool>& selectedColumns) = 0;
   };
 
   /**
diff --git a/c++/src/CMakeLists.txt b/c++/src/CMakeLists.txt
index b9904160b83..9aab4255b13 100644
--- a/c++/src/CMakeLists.txt
+++ b/c++/src/CMakeLists.txt
@@ -200,6 +200,7 @@ endif(BUILD_LIBHDFSPP)
 add_library (orc STATIC ${SOURCE_FILES})
 
 target_link_libraries (orc
+  Threads::Threads
   orc::protobuf
   orc::zlib
   orc::snappy
diff --git a/c++/src/Options.hh b/c++/src/Options.hh
index 014d0ef0e3b..70819dbd3ce 100644
--- a/c++/src/Options.hh
+++ b/c++/src/Options.hh
@@ -50,6 +50,7 @@ namespace orc {
     MemoryPool* memoryPool;
     std::string serializedTail;
     ReaderMetrics* metrics;
+    std::shared_ptr<SharedStripeStatistics> stripeStatistics;
 
     ReaderOptionsPrivate() {
       tailLocation = std::numeric_limits<uint64_t>::max();
@@ -84,6 +85,16 @@ namespace orc {
     // PASS
   }
 
+  ReaderOptions& ReaderOptions::setSharedStripeStatistics(
+      std::shared_ptr<SharedStripeStatistics> statistics) {
+    privateBits->stripeStatistics = std::move(statistics);
+    return *this;
+  }
+
+  std::shared_ptr<SharedStripeStatistics> 
ReaderOptions::getSharedStripeStatistics() const {
+    return privateBits->stripeStatistics;
+  }
+
   ReaderOptions& ReaderOptions::setMemoryPool(MemoryPool& pool) {
     privateBits->memoryPool = &pool;
     return *this;
diff --git a/c++/src/Reader.cc b/c++/src/Reader.cc
index af704655f35..1f7cb3716c2 100644
--- a/c++/src/Reader.cc
+++ b/c++/src/Reader.cc
@@ -741,6 +741,12 @@ namespace orc {
         postscriptLength(_postscriptLength),
         footer(contents->footer.get()) {
     isMetadataLoaded = false;
+    contents->stripeStatistics = options.getSharedStripeStatistics();
+    // Standalone readers need a private lazy loader; physical siblings supply 
one
+    // handle for the same immutable file, even before any predicate needs it.
+    if (!contents->stripeStatistics) {
+      contents->stripeStatistics = std::make_shared<SharedStripeStatistics>();
+    }
     checkOrcVersion();
     numberOfStripes = static_cast<uint64_t>(footer->stripes_size());
     contents->schema = convertType(footer->types(0), *footer);
@@ -983,28 +989,47 @@ namespace orc {
     return std::unique_ptr<ColumnStatistics>(convertColumnStatistics(col, 
statContext));
   }
 
+  std::shared_ptr<SharedStripeStatistics> 
ReaderImpl::getSharedStripeStatistics() const {
+    return contents->stripeStatistics;
+  }
+
   void ReaderImpl::readMetadata() const {
-    uint64_t metadataSize = contents->postscript->metadatalength();
-    uint64_t footerLength = contents->postscript->footerlength();
-    if (fileLength < metadataSize + footerLength + postscriptLength + 1) {
-      std::stringstream msg;
-      msg << "Invalid Metadata length: fileLength=" << fileLength
-          << ", metadataLength=" << metadataSize << ", footerLength=" << 
footerLength
-          << ", postscriptLength=" << postscriptLength;
-      throw ParseError(msg.str());
-    }
-    uint64_t metadataStart = fileLength - metadataSize - footerLength - 
postscriptLength - 1;
-    if (metadataSize != 0) {
-      std::unique_ptr<SeekableInputStream> pbStream = createDecompressor(
-          contents->compression,
-          std::make_unique<SeekableFileInputStream>(contents->stream.get(), 
metadataStart,
-                                                    metadataSize, 
*contents->pool),
-          contents->blockSize, *contents->pool, contents->readerMetrics);
-      contents->metadata.reset(new proto::Metadata());
-      if (!contents->metadata->ParseFromZeroCopyStream(pbStream.get())) {
-        throw ParseError("Failed to parse the metadata");
+    // Each reader keeps its own stream and metrics. Only the first successful 
loader
+    // publishes parsed statistics; failed loads remain retryable by another 
reader.
+    auto& shared = *contents->stripeStatistics;
+    // Keep exception unwinding within C++: pthread_once-backed call_once 
implementations
+    // do not all reset their flag when a loader throws.
+    std::lock_guard<std::mutex> lock(shared.mutex);
+    if (!shared.loaded) {
+      uint64_t metadataSize = contents->postscript->metadatalength();
+      uint64_t footerLength = contents->postscript->footerlength();
+      if (fileLength < metadataSize + footerLength + postscriptLength + 1) {
+        std::stringstream msg;
+        msg << "Invalid Metadata length: fileLength=" << fileLength
+            << ", metadataLength=" << metadataSize << ", footerLength=" << 
footerLength
+            << ", postscriptLength=" << postscriptLength;
+        throw ParseError(msg.str());
       }
+      uint64_t metadataStart = fileLength - metadataSize - footerLength - 
postscriptLength - 1;
+      if (metadataSize != 0) {
+        std::unique_ptr<SeekableInputStream> pbStream = createDecompressor(
+            contents->compression,
+            std::make_unique<SeekableFileInputStream>(contents->stream.get(), 
metadataStart,
+                                                      metadataSize, 
*contents->pool),
+            contents->blockSize, *contents->pool, contents->readerMetrics);
+        auto metadata = std::make_shared<proto::Metadata>();
+        // Standalone readers may leave the optional metrics pointer unset.
+        if (contents->readerMetrics != nullptr) {
+          contents->readerMetrics->MetadataParseCount.fetch_add(1, 
std::memory_order_relaxed);
+        }
+        if (!metadata->ParseFromZeroCopyStream(pbStream.get())) {
+          throw ParseError("Failed to parse the metadata");
+        }
+        shared.metadata = std::move(metadata);
+      }
+      shared.loaded = true;
     }
+    contents->metadata = shared.metadata;
     isMetadataLoaded = true;
   }
 
@@ -1038,12 +1063,22 @@ namespace orc {
   }
 
   std::vector<int> ReaderImpl::getNeedReadStripes(const RowReaderOptions& 
opts) {
+    return getNeedReadStripes(opts, 0, numberOfStripes);
+  }
+
+  std::vector<int> ReaderImpl::getNeedReadStripes(const RowReaderOptions& opts,
+                                                  uint64_t firstStripe, 
uint64_t stripeCount) {
+    // Reject invalid intervals before subtracting or indexing the statistics 
array.
+    // In particular, [numberOfStripes, numberOfStripes + 1) is not a valid 
child.
+    if (firstStripe > numberOfStripes || stripeCount > numberOfStripes - 
firstStripe) {
+      throw std::out_of_range("Invalid stripe statistics range");
+    }
     if (opts.getSearchArgument() && !isMetadataLoaded) {
       // load stripe statistics for PPD
       readMetadata();
     }
 
-    std::vector<int> allStripesNeeded(numberOfStripes, 1);
+    std::vector<int> allStripesNeeded(stripeCount, 1);
 
     if (opts.getSearchArgument() && footer->rowindexstride() > 0) {
       auto sargs = opts.getSearchArgument();
@@ -1055,14 +1090,18 @@ namespace orc {
         return allStripesNeeded;
       }
 
-      for (uint64_t currentStripeIndex = 0; currentStripeIndex < 
numberOfStripes;
-           currentStripeIndex++) {
+      for (uint64_t currentStripeIndex = firstStripe;
+           currentStripeIndex < firstStripe + stripeCount; 
currentStripeIndex++) {
         const auto& currentStripeStats =
             
contents->metadata->stripestats(static_cast<int>(currentStripeIndex));
         // Not need add mMetrics,so use 0.
-        allStripesNeeded[currentStripeIndex] =
+        // Default ReaderOptions leaves metrics disabled, including for SARG 
callers.
+        if (contents->readerMetrics != nullptr) {
+          contents->readerMetrics->StripeStatisticsEvaluationCount.fetch_add(
+              1, std::memory_order_relaxed);
+        }
+        allStripesNeeded[currentStripeIndex - firstStripe] =
             sargsApplier->evaluateStripeStatistics(currentStripeStats, 0);
-        ;
       }
       contents->sargsApplier = std::move(sargsApplier);
     }
diff --git a/c++/src/Reader.hh b/c++/src/Reader.hh
index 05990851a74..537a3a5908a 100644
--- a/c++/src/Reader.hh
+++ b/c++/src/Reader.hh
@@ -19,6 +19,8 @@
 #ifndef ORC_READER_IMPL_HH
 #define ORC_READER_IMPL_HH
 
+#include <mutex>
+
 #include "orc/Exceptions.hh"
 #include "orc/Int128.hh"
 #include "orc/OrcFile.hh"
@@ -83,9 +85,17 @@ namespace orc {
     }
   };
 
-  /**
-   * State shared between Reader and Row Reader
-   */
+  // The mutex protects loaded and metadata. A successful load publishes 
immutable Metadata
+  // without retaining a reader, its stream or its predicate state.
+  class SharedStripeStatistics {
+   public:
+    std::mutex mutex;
+    // A zero-length Metadata section is successfully loaded even though 
metadata stays null.
+    bool loaded = false;
+    std::shared_ptr<const proto::Metadata> metadata;
+  };
+
+  /** State shared between one Reader and its RowReaders. */
   struct FileContents {
     std::unique_ptr<InputStream> stream;
     std::unique_ptr<proto::PostScript> postscript;
@@ -98,7 +108,8 @@ namespace orc {
     /// Decimal64 in ORCv2 uses RLE to store values. This flag indicates 
whether
     /// this new encoding is used.
     bool isDecimalAsLong;
-    std::unique_ptr<proto::Metadata> metadata;
+    std::shared_ptr<const proto::Metadata> metadata;
+    std::shared_ptr<SharedStripeStatistics> stripeStatistics;
     ReaderMetrics* readerMetrics;
     std::unique_ptr<SargsApplier> sargsApplier;
   };
@@ -318,6 +329,9 @@ namespace orc {
     uint64_t numberOfStripes;
     std::unique_ptr<SargsApplier> sargsApplier;
     std::vector<int> getNeedReadStripes(const RowReaderOptions& opts) override;
+    std::vector<int> getNeedReadStripes(const RowReaderOptions& opts, uint64_t 
firstStripe,
+                                        uint64_t stripeCount) override;
+    std::shared_ptr<SharedStripeStatistics> getSharedStripeStatistics() const 
override;
     uint64_t getMemoryUse(int stripeIx, std::vector<bool>& selectedColumns);
 
     // internal methods
@@ -440,7 +454,8 @@ namespace orc {
     std::map<uint32_t, BloomFilterIndex> getBloomFilters(
         uint32_t stripeIndex, const std::set<uint32_t>& included) const 
override;
 
-    void getSelectedColumns(const std::list<std::string>& names, 
std::vector<bool>& selectedColumns) override;
+    void getSelectedColumns(const std::list<std::string>& names,
+                            std::vector<bool>& selectedColumns) override;
   };
 }  // namespace orc
 
diff --git a/c++/test/TestPredicatePushdown.cc 
b/c++/test/TestPredicatePushdown.cc
index 8922a9da2ce..230c40f4ec4 100644
--- a/c++/test/TestPredicatePushdown.cc
+++ b/c++/test/TestPredicatePushdown.cc
@@ -16,10 +16,12 @@
  * limitations under the License.
  */
 
+#include <thread>
 #include <tuple>
 
 #include "MemoryInputStream.hh"
 #include "MemoryOutputStream.hh"
+#include "orc/Exceptions.hh"
 #include "orc/OrcFile.hh"
 #include "orc/sargs/SearchArgument.hh"
 #include "wrap/gtest-wrapper.h"
@@ -28,6 +30,211 @@ namespace orc {
 
   static const int DEFAULT_MEM_STREAM_SIZE = 10 * 1024 * 1024;  // 10M
 
+  class MetadataFailureInputStream : public MemoryInputStream {
+   public:
+    enum class Failure { NONE, READ, PARSE };
+
+    MetadataFailureInputStream(const char* buffer, size_t size, Failure 
failure)
+        : MemoryInputStream(buffer, size), failure_(failure) {}
+
+    void read(void* buf, uint64_t length, uint64_t offset) override {
+      ++reads_;
+      if (failure_ == Failure::READ) {
+        throw ParseError("injected metadata read failure");
+      }
+      MemoryInputStream::read(buf, length, offset);
+      if (failure_ == Failure::PARSE) {
+        // Wire type 7 is invalid, so the Metadata protobuf cannot parse this 
tag.
+        *static_cast<char*>(buf) = 0x0f;
+      }
+    }
+
+    uint64_t getReadCount() const {
+      return reads_;
+    }
+
+   private:
+    Failure failure_;
+    uint64_t reads_ = 0;
+  };
+
+  TEST(TestPredicatePushdown, SharedStripeStatisticsRetryAfterLoadFailure) {
+    constexpr uint64_t STRIPES = 2;
+    constexpr uint64_t ROWS = 1000;
+    MemoryOutputStream stream(DEFAULT_MEM_STREAM_SIZE);
+    auto type = 
std::unique_ptr<Type>(Type::buildTypeFromString("struct<id:bigint>"));
+    WriterOptions writeOptions;
+    
writeOptions.setStripeSize(1).setCompression(CompressionKind_NONE).setRowIndexStride(1000);
+    auto writer = createWriter(*type, &stream, writeOptions);
+    auto batch = writer->createRowBatch(ROWS);
+    auto& root = dynamic_cast<StructVectorBatch&>(*batch);
+    auto& ids = dynamic_cast<LongVectorBatch&>(*root.fields[0]);
+    for (uint64_t stripe = 0; stripe < STRIPES; ++stripe) {
+      for (uint64_t row = 0; row < ROWS; ++row) {
+        ids.data[row] = static_cast<int64_t>(stripe * ROWS + row);
+      }
+      root.numElements = ids.numElements = ROWS;
+      writer->add(*batch);
+    }
+    writer->close();
+
+    using Failure = MetadataFailureInputStream::Failure;
+    for (auto failure : {Failure::READ, Failure::PARSE}) {
+      SCOPED_TRACE(failure == Failure::READ ? "read failure" : "parse 
failure");
+      auto parent =
+          createReader(std::make_unique<MemoryInputStream>(stream.getData(), 
stream.getLength()),
+                       ReaderOptions());
+      ASSERT_EQ(STRIPES, parent->getNumberOfStripes());
+      auto shared = parent->getSharedStripeStatistics();
+      const auto tail = parent->getSerializedFileTail();
+      parent.reset();
+
+      RowReaderOptions rowOptions;
+      rowOptions.searchArgument(
+          SearchArgumentFactory::newBuilder()
+              ->lessThan("id", PredicateDataType::LONG, 
Literal(static_cast<int64_t>(ROWS)))
+              .build());
+      ReaderMetrics failedMetrics;
+      ReaderOptions failedOptions;
+      
failedOptions.setSerializedFileTail(tail).setSharedStripeStatistics(shared).setReaderMetrics(
+          &failedMetrics);
+      auto failedStream = 
std::make_unique<MetadataFailureInputStream>(stream.getData(),
+                                                                       
stream.getLength(), failure);
+      auto* failedInput = failedStream.get();
+      auto failedReader = createReader(std::move(failedStream), failedOptions);
+      ASSERT_EQ(0, failedInput->getReadCount());
+      EXPECT_THROW(failedReader->getNeedReadStripes(rowOptions, 1, 1), 
ParseError);
+      EXPECT_EQ(1, failedInput->getReadCount());
+      EXPECT_EQ(1, failedMetrics.MetadataParseCount.load());
+
+      ReaderMetrics retryMetrics;
+      ReaderOptions retryOptions;
+      
retryOptions.setSerializedFileTail(tail).setSharedStripeStatistics(shared).setReaderMetrics(
+          &retryMetrics);
+      auto retryStream = std::make_unique<MetadataFailureInputStream>(
+          stream.getData(), stream.getLength(), Failure::NONE);
+      auto* retryInput = retryStream.get();
+      auto retryReader = createReader(std::move(retryStream), retryOptions);
+      EXPECT_EQ(std::vector<int>({0}), 
retryReader->getNeedReadStripes(rowOptions, 1, 1));
+      EXPECT_EQ(STRIPES, retryReader->getNumberOfStripeStatistics());
+      EXPECT_EQ(1, retryInput->getReadCount());
+      EXPECT_EQ(1, retryMetrics.MetadataParseCount.load());
+      retryReader.reset();
+
+      // The failed reader must adopt the successful result without using its 
failing stream again.
+      EXPECT_EQ(std::vector<int>({1}), 
failedReader->getNeedReadStripes(rowOptions, 0, 1));
+      EXPECT_EQ(1, failedInput->getReadCount());
+      EXPECT_EQ(1, failedMetrics.MetadataParseCount.load());
+      failedReader.reset();
+
+      // A new sibling also needs no I/O, even after both earlier readers are 
destroyed.
+      auto warmStream = std::make_unique<MetadataFailureInputStream>(
+          stream.getData(), stream.getLength(), Failure::READ);
+      auto* warmInput = warmStream.get();
+      auto warmReader = createReader(std::move(warmStream), retryOptions);
+      EXPECT_EQ(std::vector<int>({1, 0}), 
warmReader->getNeedReadStripes(rowOptions));
+      EXPECT_EQ(0, warmInput->getReadCount());
+      EXPECT_EQ(1, retryMetrics.MetadataParseCount.load());
+    }
+  }
+
+  TEST(TestPredicatePushdown, SharedStripeStatisticsKeepReaderLocalPredicates) 
{
+    constexpr uint64_t STRIPES = 8;
+    constexpr uint64_t ROWS = 3000;
+    MemoryOutputStream stream(DEFAULT_MEM_STREAM_SIZE);
+    auto type = 
std::unique_ptr<Type>(Type::buildTypeFromString("struct<id:bigint>"));
+    WriterOptions writeOptions;
+    
writeOptions.setStripeSize(1).setCompression(CompressionKind_NONE).setRowIndexStride(1000);
+    auto writer = createWriter(*type, &stream, writeOptions);
+    auto batch = writer->createRowBatch(ROWS);
+    auto& root = dynamic_cast<StructVectorBatch&>(*batch);
+    auto& ids = dynamic_cast<LongVectorBatch&>(*root.fields[0]);
+    for (uint64_t stripe = 0; stripe < STRIPES; ++stripe) {
+      for (uint64_t row = 0; row < ROWS; ++row) {
+        ids.data[row] = static_cast<int64_t>(stripe * ROWS + row);
+      }
+      root.numElements = ids.numElements = ROWS;
+      writer->add(*batch);
+    }
+    writer->close();
+
+    for (bool warm : {false, true}) {
+      ReaderMetrics parentMetrics;
+      ReaderOptions parentOptions;
+      parentOptions.setReaderMetrics(&parentMetrics);
+      auto parent = createReader(
+          std::make_unique<MemoryInputStream>(stream.getData(), 
stream.getLength()), parentOptions);
+      ASSERT_EQ(STRIPES, parent->getNumberOfStripes());
+      auto shared = parent->getSharedStripeStatistics();
+      const auto tail = parent->getSerializedFileTail();
+      EXPECT_EQ(0, parentMetrics.MetadataParseCount.load());
+      if (warm) {
+        RowReaderOptions options;
+        options.searchArgument(
+            SearchArgumentFactory::newBuilder()
+                ->lessThan("id", PredicateDataType::LONG, 
Literal(int64_t(1000000)))
+                .build());
+        EXPECT_EQ(std::vector<int>(STRIPES, 1), 
parent->getNeedReadStripes(options));
+      }
+      parent.reset();
+
+      std::atomic<uint64_t> parses{parentMetrics.MetadataParseCount.load()};
+      std::atomic<uint64_t> 
evaluations{parentMetrics.StripeStatisticsEvaluationCount.load()};
+      std::atomic<uint64_t> ready{0};
+      std::atomic<bool> start{false};
+      std::vector<std::thread> children;
+      for (uint64_t stripe = 0; stripe < STRIPES; ++stripe) {
+        children.emplace_back([&, stripe] {
+          ReaderMetrics metrics;
+          ReaderOptions options;
+          
options.setSerializedFileTail(tail).setSharedStripeStatistics(shared).setReaderMetrics(
+              &metrics);
+          auto reader = createReader(
+              std::make_unique<MemoryInputStream>(stream.getData(), 
stream.getLength()), options);
+          auto info = reader->getStripe(stripe);
+          RowReaderOptions rowOptions;
+          const uint64_t expectedRows = stripe % 2 == 0 ? 1000 : 2000;
+          rowOptions.range(info->getOffset(), info->getLength());
+          rowOptions.searchArgument(
+              SearchArgumentFactory::newBuilder()
+                  ->lessThan("id", PredicateDataType::LONG,
+                             Literal(static_cast<int64_t>(stripe * ROWS + 
expectedRows)))
+                  .build());
+          ready.fetch_add(1, std::memory_order_release);
+          while (!start.load(std::memory_order_acquire)) {
+            std::this_thread::yield();
+          }
+          EXPECT_EQ(std::vector<int>({1}), 
reader->getNeedReadStripes(rowOptions, stripe, 1));
+          auto rows = reader->createRowReader(rowOptions);
+          auto result = rows->createRowBatch(256);
+          uint64_t count = 0;
+          while (rows->next(*result)) {
+            EXPECT_EQ(stripe * ROWS + count, rows->getRowNumber());
+            auto& values = dynamic_cast<LongVectorBatch&>(
+                *dynamic_cast<StructVectorBatch&>(*result).fields[0]);
+            for (uint64_t row = 0; row < result->numElements; ++row) {
+              EXPECT_EQ(static_cast<int64_t>(stripe * ROWS + count++), 
values.data[row]);
+            }
+          }
+          EXPECT_EQ(expectedRows, count);
+          EXPECT_LT(metrics.SelectedRowGroupCount.load(), 
metrics.EvaluatedRowGroupCount.load());
+          EXPECT_THROW(reader->getNeedReadStripes(rowOptions, STRIPES, 1), 
std::out_of_range);
+          parses.fetch_add(metrics.MetadataParseCount.load());
+          
evaluations.fetch_add(metrics.StripeStatisticsEvaluationCount.load());
+        });
+      }
+      while (ready.load(std::memory_order_acquire) != STRIPES) {
+        std::this_thread::yield();
+      }
+      start.store(true, std::memory_order_release);
+      for (auto& child : children) {
+        child.join();
+      }
+      EXPECT_EQ(1, parses.load());
+      EXPECT_EQ(STRIPES * (warm ? 2 : 1), evaluations.load());
+    }
+  }
+
   class SeekPastFilteredStripeTest
       : public testing::TestWithParam<std::tuple<uint64_t, uint64_t, bool, 
bool, bool>> {};
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to