This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch orc
in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git
The following commit(s) were added to refs/heads/orc by this push:
new 4ec25901b12 [fix](orc) Share stripe statistics between range readers
(#435)
4ec25901b12 is described below
commit 4ec25901b1266235f56ebe5c948277d6f3c2c616
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Wed Nov 18 11:13:10 2026 +0800
[fix](orc) Share stripe statistics between range readers (#435)
* [fix](orc) Share stripe statistics between range readers
### Background
Independent ORC readers use separate streams and predicate appliers. Stripe
statistics live in a file-level Metadata protobuf, separate from the serialized
file tail.
### What problem does this PR solve?
Issue Number: None
Related PR: apache/doris#70664 (physical stripe readers)
PR Type: Bugfix
Problem Summary:
Physical stripe readers repeat complete Metadata parsing and all-file SARG
evaluation. Eight one-stripe readers plus their planning reader can parse
Metadata nine times and evaluate 72 stripe statistics. Share lazy immutable
Metadata and add bounded stripe evaluation while preserving reader-local
predicates and row-group filtering.
### Details
- **Frequency:** Found by code review; triggered by independent readers
with SARG over the same multi-stripe file.
- **Root cause:** Serialized tails omit Metadata, and getNeedReadStripes
evaluates the whole file regardless of the reader's range.
- **Why existing tests missed it:** Predicate tests check pruning results
without counting repeated setup across independent readers.
- **Fix:** Add an opaque thread-safe lazy statistics handle, an interval
overload and operation counters. A successful load publishes immutable
Metadata; failed loads remain retryable. The handle retains no streams,
allocators, metrics or SARG appliers.
- **Other paths and branches:** The existing full-file API delegates to the
interval implementation. No format or default changes; native consumers must
rebuild for the extended interface.
### Testing
Local Doris ASAN BE build passed with this SDK candidate. Standard BE unit
tests, including the new concurrent cold/warm-handle test, are running; results
will be updated before merge. The test checks one parse after parent
destruction, bounded evaluation, independent predicates, row-group pruning and
absolute row positions.
### Release note
None
### Check List (For Author)
- Behavior changed: No
- Does this need documentation: No; SDK interface contracts are documented
in the header.
- Pre-submission status: Partially verified: unit tests are running.
### AI Disclosure
- Tool: Codex
- Generated: SDK implementation, tests and description.
- Human review: not stated
- Prompts: Fix F-01 from the review of apache/doris#70664 and update that
PR.
* [fix](orc) Preserve optional reader metrics
### Background
Standalone ORC readers can leave ReaderOptions metrics unset. Statistics
loading and predicate pruning support those callers.
### What problem does this PR solve?
Issue Number: None
Related PR: apache/doris#70664
PR Type: Bugfix
Problem Summary:
The new statistics counters dereferenced the optional metrics pointer. Keep
instrumentation conditional so default ReaderOptions continues to support
statistics reads and SARG pruning.
### Details
- **Frequency:** Always when the new counters run without configured
metrics.
- **Root cause:** The metrics pointer defaults to nullptr, but the added
counters assumed it was configured.
- **Why existing tests missed it:** The new counter tests supplied metrics
explicitly; running the existing standalone-reader cases exposed the missing
guards.
- **Fix:** Follow SargsApplier's existing optional-metrics pattern for both
Metadata parsing and stripe evaluation.
- **Other paths and branches:** Both full-file and interval APIs use the
guarded counters. No format or config changes.
### Testing
All 238 selected Doris ORC and SDK reader/predicate tests passed locally
under ASAN, including default ReaderOptions, concurrent cold/warm statistics
sharing, and exact stripe ownership.
### Release note
None
### Check List (For Author)
- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready
### AI Disclosure
- Tool: Codex
- Generated: Metrics guards and this message.
- Human review: not stated
* [chore](orc) Declare thread linkage for statistics loading
### Background
The shared statistics loader uses std::call_once. ORC already discovers the
platform thread dependency during configuration.
### What problem does this PR solve?
Issue Number: None
Related PR: apache/doris#70664
PR Type: Build/CI/Chore
Problem Summary:
Declare Threads::Threads on the ORC library so standalone native consumers
inherit the loader's thread linkage.
### Details
- **Affected:** ORC library link interface.
- **Developer and release impact:** Standalone consumers receive the
required platform thread flags; Doris already links threads.
- **License and third-party:** No new dependency; use the existing
discovered system thread target.
- **Rollback:** Remove the target dependency together with the shared
loader.
### Testing
Reconfigured through the standard Doris BE unit script and passed all 238
selected ORC reader/predicate tests under ASAN.
### Release note
None
### Check List (For Author)
- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready
### AI Disclosure
- Tool: Codex
- Generated: Link dependency and this message.
- Human review: not stated
* [fix](orc) Keep shared statistics retryable after load failures
### Background
Independent readers share lazily parsed immutable stripe statistics.
### What problem does this PR solve?
Issue Number: None
Related PR: apache/doris#70664 (downstream physical stripe readers)
PR Type: Bugfix
Problem Summary:
A throwing call_once initializer relies on runtime exception-reset behavior.
Replace it with a mutex and an explicit loaded state so failed loaders
cannot
publish success and other readers can retry with their own streams.
### Details
- **Frequency:** Reported in SDK review on an affected runtime combination;
not reproduced on the local macOS runtime or in a running Doris BE.
- **Root cause:** pthread_once-backed call_once implementations can fail to
reset initialization state after an exception.
- **Why existing tests missed it:** Successful cold/warm sharing tests did
not inject Metadata load failures.
- **Fix:** Protect shared state with a mutex, parse locally, and mark
loaded only after success. Keep successful empty Metadata distinct from an
unloaded handle.
- **Other paths and branches:** All Metadata accessors use this loader.
This corrects code introduced in the same unmerged SDK PR; no public API or
file-format changes in this followup.
### Testing
All 239 selected Doris/SDK reader and predicate tests passed under ASAN,
including 196 Doris ORC tests. The new test injects both a read exception
and an
invalid protobuf tag after exporting a cold handle and destroying its
parent.
It checks retry through a different reader and cache reuse by the failed
reader
and a later sibling after both loaders are destroyed.
The same new test passes on the unchanged call_once implementation locally;
this is a conforming-runtime control, not a reproduced failing baseline.
### Release note
None
### Check List (For Author)
- Behavior changed: No
- Does this need documentation: No; existing API contracts remain unchanged.
- Pre-submission status: Partially verified: the affected runtime
combination is unavailable locally.
### AI Disclosure
- Tool: Codex
- Generated: implementation, tests and description.
- Human review: not stated
- Prompts: Fix the review finding on apache/doris-thirdparty#435 and update
that PR.
---------
Co-authored-by: morningman <[email protected]>
---
c++/include/orc/Reader.hh | 23 ++++-
c++/src/CMakeLists.txt | 1 +
c++/src/Options.hh | 11 ++
c++/src/Reader.cc | 87 +++++++++++-----
c++/src/Reader.hh | 25 ++++-
c++/test/TestPredicatePushdown.cc | 207 ++++++++++++++++++++++++++++++++++++++
6 files changed, 324 insertions(+), 30 deletions(-)
diff --git a/c++/include/orc/Reader.hh b/c++/include/orc/Reader.hh
index 1ab60b1d2c5..e45f1ce450e 100644
--- a/c++/include/orc/Reader.hh
+++ b/c++/include/orc/Reader.hh
@@ -41,11 +41,16 @@ namespace orc {
struct ReaderOptionsPrivate;
struct RowReaderOptionsPrivate;
class InputStream;
+ class SharedStripeStatistics;
/**
* Expose the reader metrics including the latency and
* number of calls of the decompression/decoding/IO modules.
*/
struct ReaderMetrics {
+ // Metadata protobuf parse attempts and getNeedReadStripes evaluations
only.
+ // RowReader's range-local stripe/row-group PPD retains its existing
metrics.
+ std::atomic<uint64_t> MetadataParseCount{0};
+ std::atomic<uint64_t> StripeStatisticsEvaluationCount{0};
std::atomic<uint64_t> ReaderCall{0};
// ReaderInclusiveLatencyUs contains the latency of
// the decompression/decoding/IO modules.
@@ -96,6 +101,14 @@ namespace orc {
*/
ReaderOptions& setSerializedFileTail(const std::string& serialization);
+ /**
+ * Share lazily loaded, immutable stripe statistics between independent
readers.
+ * The handle must come from the same immutable file version as this
reader's tail.
+ * Streams, metrics and predicate appliers remain reader-local.
+ */
+ ReaderOptions&
setSharedStripeStatistics(std::shared_ptr<SharedStripeStatistics> statistics);
+ std::shared_ptr<SharedStripeStatistics> getSharedStripeStatistics() const;
+
/**
* Set the memory allocator.
*/
@@ -661,7 +674,15 @@ namespace orc {
virtual std::vector<int> getNeedReadStripes(const RowReaderOptions& opts)
= 0;
- virtual void getSelectedColumns(const std::list<std::string>& names,
std::vector<bool>& selectedColumns) = 0;
+ /** Return decisions for [firstStripe, firstStripe + stripeCount), indexed
from zero. */
+ virtual std::vector<int> getNeedReadStripes(const RowReaderOptions& opts,
uint64_t firstStripe,
+ uint64_t stripeCount) = 0;
+
+ /** Export a thread-safe lazy statistics handle without reading Metadata.
*/
+ virtual std::shared_ptr<SharedStripeStatistics>
getSharedStripeStatistics() const = 0;
+
+ virtual void getSelectedColumns(const std::list<std::string>& names,
+ std::vector<bool>& selectedColumns) = 0;
};
/**
diff --git a/c++/src/CMakeLists.txt b/c++/src/CMakeLists.txt
index b9904160b83..9aab4255b13 100644
--- a/c++/src/CMakeLists.txt
+++ b/c++/src/CMakeLists.txt
@@ -200,6 +200,7 @@ endif(BUILD_LIBHDFSPP)
add_library (orc STATIC ${SOURCE_FILES})
target_link_libraries (orc
+ Threads::Threads
orc::protobuf
orc::zlib
orc::snappy
diff --git a/c++/src/Options.hh b/c++/src/Options.hh
index 014d0ef0e3b..70819dbd3ce 100644
--- a/c++/src/Options.hh
+++ b/c++/src/Options.hh
@@ -50,6 +50,7 @@ namespace orc {
MemoryPool* memoryPool;
std::string serializedTail;
ReaderMetrics* metrics;
+ std::shared_ptr<SharedStripeStatistics> stripeStatistics;
ReaderOptionsPrivate() {
tailLocation = std::numeric_limits<uint64_t>::max();
@@ -84,6 +85,16 @@ namespace orc {
// PASS
}
+ ReaderOptions& ReaderOptions::setSharedStripeStatistics(
+ std::shared_ptr<SharedStripeStatistics> statistics) {
+ privateBits->stripeStatistics = std::move(statistics);
+ return *this;
+ }
+
+ std::shared_ptr<SharedStripeStatistics>
ReaderOptions::getSharedStripeStatistics() const {
+ return privateBits->stripeStatistics;
+ }
+
ReaderOptions& ReaderOptions::setMemoryPool(MemoryPool& pool) {
privateBits->memoryPool = &pool;
return *this;
diff --git a/c++/src/Reader.cc b/c++/src/Reader.cc
index af704655f35..1f7cb3716c2 100644
--- a/c++/src/Reader.cc
+++ b/c++/src/Reader.cc
@@ -741,6 +741,12 @@ namespace orc {
postscriptLength(_postscriptLength),
footer(contents->footer.get()) {
isMetadataLoaded = false;
+ contents->stripeStatistics = options.getSharedStripeStatistics();
+ // Standalone readers need a private lazy loader; physical siblings supply
one
+ // handle for the same immutable file, even before any predicate needs it.
+ if (!contents->stripeStatistics) {
+ contents->stripeStatistics = std::make_shared<SharedStripeStatistics>();
+ }
checkOrcVersion();
numberOfStripes = static_cast<uint64_t>(footer->stripes_size());
contents->schema = convertType(footer->types(0), *footer);
@@ -983,28 +989,47 @@ namespace orc {
return std::unique_ptr<ColumnStatistics>(convertColumnStatistics(col,
statContext));
}
+ std::shared_ptr<SharedStripeStatistics>
ReaderImpl::getSharedStripeStatistics() const {
+ return contents->stripeStatistics;
+ }
+
void ReaderImpl::readMetadata() const {
- uint64_t metadataSize = contents->postscript->metadatalength();
- uint64_t footerLength = contents->postscript->footerlength();
- if (fileLength < metadataSize + footerLength + postscriptLength + 1) {
- std::stringstream msg;
- msg << "Invalid Metadata length: fileLength=" << fileLength
- << ", metadataLength=" << metadataSize << ", footerLength=" <<
footerLength
- << ", postscriptLength=" << postscriptLength;
- throw ParseError(msg.str());
- }
- uint64_t metadataStart = fileLength - metadataSize - footerLength -
postscriptLength - 1;
- if (metadataSize != 0) {
- std::unique_ptr<SeekableInputStream> pbStream = createDecompressor(
- contents->compression,
- std::make_unique<SeekableFileInputStream>(contents->stream.get(),
metadataStart,
- metadataSize,
*contents->pool),
- contents->blockSize, *contents->pool, contents->readerMetrics);
- contents->metadata.reset(new proto::Metadata());
- if (!contents->metadata->ParseFromZeroCopyStream(pbStream.get())) {
- throw ParseError("Failed to parse the metadata");
+ // Each reader keeps its own stream and metrics. Only the first successful
loader
+ // publishes parsed statistics; failed loads remain retryable by another
reader.
+ auto& shared = *contents->stripeStatistics;
+ // Keep exception unwinding within C++: pthread_once-backed call_once
implementations
+ // do not all reset their flag when a loader throws.
+ std::lock_guard<std::mutex> lock(shared.mutex);
+ if (!shared.loaded) {
+ uint64_t metadataSize = contents->postscript->metadatalength();
+ uint64_t footerLength = contents->postscript->footerlength();
+ if (fileLength < metadataSize + footerLength + postscriptLength + 1) {
+ std::stringstream msg;
+ msg << "Invalid Metadata length: fileLength=" << fileLength
+ << ", metadataLength=" << metadataSize << ", footerLength=" <<
footerLength
+ << ", postscriptLength=" << postscriptLength;
+ throw ParseError(msg.str());
}
+ uint64_t metadataStart = fileLength - metadataSize - footerLength -
postscriptLength - 1;
+ if (metadataSize != 0) {
+ std::unique_ptr<SeekableInputStream> pbStream = createDecompressor(
+ contents->compression,
+ std::make_unique<SeekableFileInputStream>(contents->stream.get(),
metadataStart,
+ metadataSize,
*contents->pool),
+ contents->blockSize, *contents->pool, contents->readerMetrics);
+ auto metadata = std::make_shared<proto::Metadata>();
+ // Standalone readers may leave the optional metrics pointer unset.
+ if (contents->readerMetrics != nullptr) {
+ contents->readerMetrics->MetadataParseCount.fetch_add(1,
std::memory_order_relaxed);
+ }
+ if (!metadata->ParseFromZeroCopyStream(pbStream.get())) {
+ throw ParseError("Failed to parse the metadata");
+ }
+ shared.metadata = std::move(metadata);
+ }
+ shared.loaded = true;
}
+ contents->metadata = shared.metadata;
isMetadataLoaded = true;
}
@@ -1038,12 +1063,22 @@ namespace orc {
}
std::vector<int> ReaderImpl::getNeedReadStripes(const RowReaderOptions&
opts) {
+ return getNeedReadStripes(opts, 0, numberOfStripes);
+ }
+
+ std::vector<int> ReaderImpl::getNeedReadStripes(const RowReaderOptions& opts,
+ uint64_t firstStripe,
uint64_t stripeCount) {
+ // Reject invalid intervals before subtracting or indexing the statistics
array.
+ // In particular, [numberOfStripes, numberOfStripes + 1) is not a valid
child.
+ if (firstStripe > numberOfStripes || stripeCount > numberOfStripes -
firstStripe) {
+ throw std::out_of_range("Invalid stripe statistics range");
+ }
if (opts.getSearchArgument() && !isMetadataLoaded) {
// load stripe statistics for PPD
readMetadata();
}
- std::vector<int> allStripesNeeded(numberOfStripes, 1);
+ std::vector<int> allStripesNeeded(stripeCount, 1);
if (opts.getSearchArgument() && footer->rowindexstride() > 0) {
auto sargs = opts.getSearchArgument();
@@ -1055,14 +1090,18 @@ namespace orc {
return allStripesNeeded;
}
- for (uint64_t currentStripeIndex = 0; currentStripeIndex <
numberOfStripes;
- currentStripeIndex++) {
+ for (uint64_t currentStripeIndex = firstStripe;
+ currentStripeIndex < firstStripe + stripeCount;
currentStripeIndex++) {
const auto& currentStripeStats =
contents->metadata->stripestats(static_cast<int>(currentStripeIndex));
// Not need add mMetrics,so use 0.
- allStripesNeeded[currentStripeIndex] =
+ // Default ReaderOptions leaves metrics disabled, including for SARG
callers.
+ if (contents->readerMetrics != nullptr) {
+ contents->readerMetrics->StripeStatisticsEvaluationCount.fetch_add(
+ 1, std::memory_order_relaxed);
+ }
+ allStripesNeeded[currentStripeIndex - firstStripe] =
sargsApplier->evaluateStripeStatistics(currentStripeStats, 0);
- ;
}
contents->sargsApplier = std::move(sargsApplier);
}
diff --git a/c++/src/Reader.hh b/c++/src/Reader.hh
index 05990851a74..537a3a5908a 100644
--- a/c++/src/Reader.hh
+++ b/c++/src/Reader.hh
@@ -19,6 +19,8 @@
#ifndef ORC_READER_IMPL_HH
#define ORC_READER_IMPL_HH
+#include <mutex>
+
#include "orc/Exceptions.hh"
#include "orc/Int128.hh"
#include "orc/OrcFile.hh"
@@ -83,9 +85,17 @@ namespace orc {
}
};
- /**
- * State shared between Reader and Row Reader
- */
+ // The mutex protects loaded and metadata. A successful load publishes
immutable Metadata
+ // without retaining a reader, its stream or its predicate state.
+ class SharedStripeStatistics {
+ public:
+ std::mutex mutex;
+ // A zero-length Metadata section is successfully loaded even though
metadata stays null.
+ bool loaded = false;
+ std::shared_ptr<const proto::Metadata> metadata;
+ };
+
+ /** State shared between one Reader and its RowReaders. */
struct FileContents {
std::unique_ptr<InputStream> stream;
std::unique_ptr<proto::PostScript> postscript;
@@ -98,7 +108,8 @@ namespace orc {
/// Decimal64 in ORCv2 uses RLE to store values. This flag indicates
whether
/// this new encoding is used.
bool isDecimalAsLong;
- std::unique_ptr<proto::Metadata> metadata;
+ std::shared_ptr<const proto::Metadata> metadata;
+ std::shared_ptr<SharedStripeStatistics> stripeStatistics;
ReaderMetrics* readerMetrics;
std::unique_ptr<SargsApplier> sargsApplier;
};
@@ -318,6 +329,9 @@ namespace orc {
uint64_t numberOfStripes;
std::unique_ptr<SargsApplier> sargsApplier;
std::vector<int> getNeedReadStripes(const RowReaderOptions& opts) override;
+ std::vector<int> getNeedReadStripes(const RowReaderOptions& opts, uint64_t
firstStripe,
+ uint64_t stripeCount) override;
+ std::shared_ptr<SharedStripeStatistics> getSharedStripeStatistics() const
override;
uint64_t getMemoryUse(int stripeIx, std::vector<bool>& selectedColumns);
// internal methods
@@ -440,7 +454,8 @@ namespace orc {
std::map<uint32_t, BloomFilterIndex> getBloomFilters(
uint32_t stripeIndex, const std::set<uint32_t>& included) const
override;
- void getSelectedColumns(const std::list<std::string>& names,
std::vector<bool>& selectedColumns) override;
+ void getSelectedColumns(const std::list<std::string>& names,
+ std::vector<bool>& selectedColumns) override;
};
} // namespace orc
diff --git a/c++/test/TestPredicatePushdown.cc
b/c++/test/TestPredicatePushdown.cc
index 8922a9da2ce..230c40f4ec4 100644
--- a/c++/test/TestPredicatePushdown.cc
+++ b/c++/test/TestPredicatePushdown.cc
@@ -16,10 +16,12 @@
* limitations under the License.
*/
+#include <thread>
#include <tuple>
#include "MemoryInputStream.hh"
#include "MemoryOutputStream.hh"
+#include "orc/Exceptions.hh"
#include "orc/OrcFile.hh"
#include "orc/sargs/SearchArgument.hh"
#include "wrap/gtest-wrapper.h"
@@ -28,6 +30,211 @@ namespace orc {
static const int DEFAULT_MEM_STREAM_SIZE = 10 * 1024 * 1024; // 10M
+ class MetadataFailureInputStream : public MemoryInputStream {
+ public:
+ enum class Failure { NONE, READ, PARSE };
+
+ MetadataFailureInputStream(const char* buffer, size_t size, Failure
failure)
+ : MemoryInputStream(buffer, size), failure_(failure) {}
+
+ void read(void* buf, uint64_t length, uint64_t offset) override {
+ ++reads_;
+ if (failure_ == Failure::READ) {
+ throw ParseError("injected metadata read failure");
+ }
+ MemoryInputStream::read(buf, length, offset);
+ if (failure_ == Failure::PARSE) {
+ // Wire type 7 is invalid, so the Metadata protobuf cannot parse this
tag.
+ *static_cast<char*>(buf) = 0x0f;
+ }
+ }
+
+ uint64_t getReadCount() const {
+ return reads_;
+ }
+
+ private:
+ Failure failure_;
+ uint64_t reads_ = 0;
+ };
+
+ TEST(TestPredicatePushdown, SharedStripeStatisticsRetryAfterLoadFailure) {
+ constexpr uint64_t STRIPES = 2;
+ constexpr uint64_t ROWS = 1000;
+ MemoryOutputStream stream(DEFAULT_MEM_STREAM_SIZE);
+ auto type =
std::unique_ptr<Type>(Type::buildTypeFromString("struct<id:bigint>"));
+ WriterOptions writeOptions;
+
writeOptions.setStripeSize(1).setCompression(CompressionKind_NONE).setRowIndexStride(1000);
+ auto writer = createWriter(*type, &stream, writeOptions);
+ auto batch = writer->createRowBatch(ROWS);
+ auto& root = dynamic_cast<StructVectorBatch&>(*batch);
+ auto& ids = dynamic_cast<LongVectorBatch&>(*root.fields[0]);
+ for (uint64_t stripe = 0; stripe < STRIPES; ++stripe) {
+ for (uint64_t row = 0; row < ROWS; ++row) {
+ ids.data[row] = static_cast<int64_t>(stripe * ROWS + row);
+ }
+ root.numElements = ids.numElements = ROWS;
+ writer->add(*batch);
+ }
+ writer->close();
+
+ using Failure = MetadataFailureInputStream::Failure;
+ for (auto failure : {Failure::READ, Failure::PARSE}) {
+ SCOPED_TRACE(failure == Failure::READ ? "read failure" : "parse
failure");
+ auto parent =
+ createReader(std::make_unique<MemoryInputStream>(stream.getData(),
stream.getLength()),
+ ReaderOptions());
+ ASSERT_EQ(STRIPES, parent->getNumberOfStripes());
+ auto shared = parent->getSharedStripeStatistics();
+ const auto tail = parent->getSerializedFileTail();
+ parent.reset();
+
+ RowReaderOptions rowOptions;
+ rowOptions.searchArgument(
+ SearchArgumentFactory::newBuilder()
+ ->lessThan("id", PredicateDataType::LONG,
Literal(static_cast<int64_t>(ROWS)))
+ .build());
+ ReaderMetrics failedMetrics;
+ ReaderOptions failedOptions;
+
failedOptions.setSerializedFileTail(tail).setSharedStripeStatistics(shared).setReaderMetrics(
+ &failedMetrics);
+ auto failedStream =
std::make_unique<MetadataFailureInputStream>(stream.getData(),
+
stream.getLength(), failure);
+ auto* failedInput = failedStream.get();
+ auto failedReader = createReader(std::move(failedStream), failedOptions);
+ ASSERT_EQ(0, failedInput->getReadCount());
+ EXPECT_THROW(failedReader->getNeedReadStripes(rowOptions, 1, 1),
ParseError);
+ EXPECT_EQ(1, failedInput->getReadCount());
+ EXPECT_EQ(1, failedMetrics.MetadataParseCount.load());
+
+ ReaderMetrics retryMetrics;
+ ReaderOptions retryOptions;
+
retryOptions.setSerializedFileTail(tail).setSharedStripeStatistics(shared).setReaderMetrics(
+ &retryMetrics);
+ auto retryStream = std::make_unique<MetadataFailureInputStream>(
+ stream.getData(), stream.getLength(), Failure::NONE);
+ auto* retryInput = retryStream.get();
+ auto retryReader = createReader(std::move(retryStream), retryOptions);
+ EXPECT_EQ(std::vector<int>({0}),
retryReader->getNeedReadStripes(rowOptions, 1, 1));
+ EXPECT_EQ(STRIPES, retryReader->getNumberOfStripeStatistics());
+ EXPECT_EQ(1, retryInput->getReadCount());
+ EXPECT_EQ(1, retryMetrics.MetadataParseCount.load());
+ retryReader.reset();
+
+ // The failed reader must adopt the successful result without using its
failing stream again.
+ EXPECT_EQ(std::vector<int>({1}),
failedReader->getNeedReadStripes(rowOptions, 0, 1));
+ EXPECT_EQ(1, failedInput->getReadCount());
+ EXPECT_EQ(1, failedMetrics.MetadataParseCount.load());
+ failedReader.reset();
+
+ // A new sibling also needs no I/O, even after both earlier readers are
destroyed.
+ auto warmStream = std::make_unique<MetadataFailureInputStream>(
+ stream.getData(), stream.getLength(), Failure::READ);
+ auto* warmInput = warmStream.get();
+ auto warmReader = createReader(std::move(warmStream), retryOptions);
+ EXPECT_EQ(std::vector<int>({1, 0}),
warmReader->getNeedReadStripes(rowOptions));
+ EXPECT_EQ(0, warmInput->getReadCount());
+ EXPECT_EQ(1, retryMetrics.MetadataParseCount.load());
+ }
+ }
+
+ TEST(TestPredicatePushdown, SharedStripeStatisticsKeepReaderLocalPredicates)
{
+ constexpr uint64_t STRIPES = 8;
+ constexpr uint64_t ROWS = 3000;
+ MemoryOutputStream stream(DEFAULT_MEM_STREAM_SIZE);
+ auto type =
std::unique_ptr<Type>(Type::buildTypeFromString("struct<id:bigint>"));
+ WriterOptions writeOptions;
+
writeOptions.setStripeSize(1).setCompression(CompressionKind_NONE).setRowIndexStride(1000);
+ auto writer = createWriter(*type, &stream, writeOptions);
+ auto batch = writer->createRowBatch(ROWS);
+ auto& root = dynamic_cast<StructVectorBatch&>(*batch);
+ auto& ids = dynamic_cast<LongVectorBatch&>(*root.fields[0]);
+ for (uint64_t stripe = 0; stripe < STRIPES; ++stripe) {
+ for (uint64_t row = 0; row < ROWS; ++row) {
+ ids.data[row] = static_cast<int64_t>(stripe * ROWS + row);
+ }
+ root.numElements = ids.numElements = ROWS;
+ writer->add(*batch);
+ }
+ writer->close();
+
+ for (bool warm : {false, true}) {
+ ReaderMetrics parentMetrics;
+ ReaderOptions parentOptions;
+ parentOptions.setReaderMetrics(&parentMetrics);
+ auto parent = createReader(
+ std::make_unique<MemoryInputStream>(stream.getData(),
stream.getLength()), parentOptions);
+ ASSERT_EQ(STRIPES, parent->getNumberOfStripes());
+ auto shared = parent->getSharedStripeStatistics();
+ const auto tail = parent->getSerializedFileTail();
+ EXPECT_EQ(0, parentMetrics.MetadataParseCount.load());
+ if (warm) {
+ RowReaderOptions options;
+ options.searchArgument(
+ SearchArgumentFactory::newBuilder()
+ ->lessThan("id", PredicateDataType::LONG,
Literal(int64_t(1000000)))
+ .build());
+ EXPECT_EQ(std::vector<int>(STRIPES, 1),
parent->getNeedReadStripes(options));
+ }
+ parent.reset();
+
+ std::atomic<uint64_t> parses{parentMetrics.MetadataParseCount.load()};
+ std::atomic<uint64_t>
evaluations{parentMetrics.StripeStatisticsEvaluationCount.load()};
+ std::atomic<uint64_t> ready{0};
+ std::atomic<bool> start{false};
+ std::vector<std::thread> children;
+ for (uint64_t stripe = 0; stripe < STRIPES; ++stripe) {
+ children.emplace_back([&, stripe] {
+ ReaderMetrics metrics;
+ ReaderOptions options;
+
options.setSerializedFileTail(tail).setSharedStripeStatistics(shared).setReaderMetrics(
+ &metrics);
+ auto reader = createReader(
+ std::make_unique<MemoryInputStream>(stream.getData(),
stream.getLength()), options);
+ auto info = reader->getStripe(stripe);
+ RowReaderOptions rowOptions;
+ const uint64_t expectedRows = stripe % 2 == 0 ? 1000 : 2000;
+ rowOptions.range(info->getOffset(), info->getLength());
+ rowOptions.searchArgument(
+ SearchArgumentFactory::newBuilder()
+ ->lessThan("id", PredicateDataType::LONG,
+ Literal(static_cast<int64_t>(stripe * ROWS +
expectedRows)))
+ .build());
+ ready.fetch_add(1, std::memory_order_release);
+ while (!start.load(std::memory_order_acquire)) {
+ std::this_thread::yield();
+ }
+ EXPECT_EQ(std::vector<int>({1}),
reader->getNeedReadStripes(rowOptions, stripe, 1));
+ auto rows = reader->createRowReader(rowOptions);
+ auto result = rows->createRowBatch(256);
+ uint64_t count = 0;
+ while (rows->next(*result)) {
+ EXPECT_EQ(stripe * ROWS + count, rows->getRowNumber());
+ auto& values = dynamic_cast<LongVectorBatch&>(
+ *dynamic_cast<StructVectorBatch&>(*result).fields[0]);
+ for (uint64_t row = 0; row < result->numElements; ++row) {
+ EXPECT_EQ(static_cast<int64_t>(stripe * ROWS + count++),
values.data[row]);
+ }
+ }
+ EXPECT_EQ(expectedRows, count);
+ EXPECT_LT(metrics.SelectedRowGroupCount.load(),
metrics.EvaluatedRowGroupCount.load());
+ EXPECT_THROW(reader->getNeedReadStripes(rowOptions, STRIPES, 1),
std::out_of_range);
+ parses.fetch_add(metrics.MetadataParseCount.load());
+
evaluations.fetch_add(metrics.StripeStatisticsEvaluationCount.load());
+ });
+ }
+ while (ready.load(std::memory_order_acquire) != STRIPES) {
+ std::this_thread::yield();
+ }
+ start.store(true, std::memory_order_release);
+ for (auto& child : children) {
+ child.join();
+ }
+ EXPECT_EQ(1, parses.load());
+ EXPECT_EQ(STRIPES * (warm ? 2 : 1), evaluations.load());
+ }
+ }
+
class SeekPastFilteredStripeTest
: public testing::TestWithParam<std::tuple<uint64_t, uint64_t, bool,
bool, bool>> {};
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]