This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c20fcddcd0e [feature](iceberg) Read encrypted Parquet indexes and
Bloom filters (#70314)
c20fcddcd0e is described below
commit c20fcddcd0ed2dd26f17293f454bcac236e08871
Author: Gabriel <[email protected]>
AuthorDate: Tue Nov 17 14:17:09 2026 +0800
[feature](iceberg) Read encrypted Parquet indexes and Bloom filters (#70314)
### Background
The native Iceberg encryption reader can authenticate Parquet footers,
dictionaries and data pages. Its optional encrypted indexes and Bloom
filters are still bypassed, so selective scans cannot use those modules
safely.
### What problem does this PR solve?
Issue Number: close #70313
Related PR: #70246 (file crypto), #70248 (native encrypted pages)
PR Type: Feature
Problem Summary:
Implement D7: authenticate and consume ColumnIndex, OffsetIndex and
split-block Bloom filters in the native reader. Indexed reads preserve
the physical data-page ordinal after seeking past earlier pages, and
compare index rectangles against serialized disk sizes. Authentication
and malformed encrypted-module failures propagate; genuinely absent
optional indexes retain conservative reads. A cache-hit header also
initializes its disk size before index reconciliation, preserving
plaintext indexed scans.
### Details
- **Change map:** Production code: +438/-102 lines for authenticated
metadata parsing, index loading, Bloom framing and indexed-page AAD
integration. Tests cover native reader scenarios, an independent Java
fixture generator, SQL checks and goldens, plus four small binary
fixture copies. The fixture README documents reproduction and the
remaining integration boundary.
- **Scope:** Encrypted ColumnIndex/OffsetIndex, independently
authenticated Bloom header/bitset, page skipping and plaintext cache-hit
compatibility.
- **Design:** Reuse Parquet's GCM implementation with module-specific
AAD bound to the file, row group and physical column. Keep physical page
ordinals across seeks and exclude dictionary pages from that sequence.
Bound module allocations and decoded Thrift containers. Preserve
coalesced index I/O and one live decoded Bloom payload per leaf. Split
adjacent index blocks at the per-read allocation budget, so a wide row
group can load all individually valid modules even when their combined
size exceeds 64 MiB. Charge known index list fields by their generated
native element sizes, independently of untrusted wire element tags.
Index list counts follow the cumulative native-byte budget so nested
histograms may exceed 65,536 entries; general/footer limits are
unchanged. Unknown or skipped containers use the conservative index
maximum. Only index protocols track fields and depth, preserving the
general page-header path.
- **Usage:** The existing physical reader encryption descriptor enables
these modules in controlled reader integrations. No new configuration or
Thrift field is introduced.
- **Default behavior:** SQL encrypted-read admission remains closed.
Encrypted plaintext page/footer caches remain bypassed, and dictionary
pruning remains conservative because its existing optional-probe path
suppresses reader errors. Ordinary dictionary decoding still
authenticates.
- **Compatibility:** Plaintext optional-index fallback is preserved. An
absent encrypted index/Bloom filter is optional; a present module with
an invalid range, framing or authentication tag fails the read.
### Testing
Latest verification after the review fixes:
- Selected Parquet, file-encryption and scan-range suites under ASAN
with PME-enabled dependencies: **512 tests, 510 passed, 2 existing
opt-in microbenchmarks skipped, no failures or errors**. The run uses
focused test sources plus existing common test support; the full BE UT
suite was not run.
- A real encrypted file generated by Arrow 24 has two row groups with
30,000 data pages each. The native loader retains every ColumnIndex and
OffsetIndex entry. This test previously failed with `INVALID_METADATA`
under footer-based allocation accounting.
- Real Arrow 24 encrypted nested files cover 22,000 pages with 66,000
definition-level histogram bins and 30,000 pages with 3.03 million bins.
Both fail before the histogram fix and pass afterward, with all
index/page entries and histogram boundary counts checked. The latter's
i64 storage fits the native budget while string-sized charging would
reject it. A forward-compatible unknown struct before that histogram
also parses, verifying restoration after Thrift's template skip path.
- Twelve Java-fixture physical-reader scans compare encrypted/plaintext
and pruned/fully unpruned results. The reference request fences every
metadata-pruning conjunct while retaining residual row filtering;
counters assert zero row-group/page pruning and all 512 physical rows
consumed. Enabled-pruning cases still assert effective page and Bloom
filtering. The new baseline assertions failed before the request fence
was added.
- A malformed ColumnIndex advertises BYTE lists where generated code
allocates strings, and repeats those fields beyond the cumulative native
budget. Parsing rejects it before the next vector resize. Existing
footer LIST/SET/MAP allocation defenses, tampered module/AAD tests, and
the 65-leaf bounded coalesced-I/O cases also pass.
- The SQL case now describes and toggles Bloom probes only, with native
footer/page pruning active. Query text, dynamic golden names and
expected results are preserved. The SQL suite and full FE/BE builds were
not rerun for these focused fixes.
- Clang-format 16, full-file clang-tidy on both changed C++ files, build
hygiene, whitespace and sensitive-data checks passed.
Earlier verification at the initial feature head (the full builds and
SQL regression were not repeated for these focused fixes):
- `build.sh --be -j 6` with ASAN: passed, including the GLIBC 2.17
compatibility gate. `build.sh --fe -j 6`: passed.
- `run-be-ut.sh --run
--filter='*Parquet*:*FileEncryption*:*FileScanRange*' -j 6` with ASAN
and PME-enabled dependencies: 380 tests, 378 passed and 2 existing
opt-in microbenchmarks skipped. A focused test source selection was
used; this is not a claim that the full BE UT suite ran.
- New reader cases cover Page V1/V2, dictionary/plain pages, multiple
row groups, physical page skipping, module/file/row-group/column
binding, corrupt index/Bloom tags, oversized lengths, invalid ranges,
absent Bloom filters and cached plaintext headers.
- Parquet Java 1.15.2 generated encrypted/plaintext twins: 512 rows in
six row groups, indexes and Bloom filters on both columns. Twelve
physical-reader scans compared exact rows across encrypted/plaintext
twins and query-option settings; the follow-up verification above adds
an explicitly fully unpruned reference.
- `run-regression-test.sh --run -d external_table_p0/iceberg -s
test_iceberg_encryption_native_reader`: passed against this branch's
ASAN BE. Six new golden queries compare the plaintext twin across
Bloom-filter settings while native footer/page pruning remains active;
encrypted-only and mixed-file keyless scans remain rejected, including
negative Bloom probes. Goldens were generated through the regression
framework and then verified in comparison mode.
- The new index tests failed against the previous disabled-index
behavior. The invalid-range test also reproduced a silent fallback
before the fix. SQL regression exposed the cached-header ordering issue
before it was corrected and covered by a dedicated UT.
- Clang-format 16, changed-line clang-tidy, build hygiene, whitespace
checks and secret scanning: passed.
### Release note
None. This advances the native encrypted reader implementation;
encrypted Iceberg SQL admission remains separate work.
### Check List (For Author)
- Behavior changed: Yes (see Details)
- Does this need documentation: No; internal staged reader integration
and test-fixture documentation are included.
- Pre-submission status: Ready. Encrypted SQL admission and shared
decrypted cache support are explicitly outside this PR.
### AI Disclosure
- Tool: OpenAI Codex (GPT-6)
- Generated: Implementation, tests, fixtures generator and this
description.
- Human review: Pending maintainer review; the agent self-reviewed the
full diff and ran the verification above.
---
be/src/format_v2/parquet/parquet_crypto.cpp | 162 ++++-
be/src/format_v2/parquet/parquet_crypto.h | 16 +
be/src/format_v2/parquet/parquet_file_context.cpp | 144 +++--
be/src/format_v2/parquet/parquet_reader.cpp | 1 -
be/src/format_v2/parquet/parquet_scan.cpp | 12 -
be/src/format_v2/parquet/parquet_statistics.cpp | 180 +++++-
.../parquet/reader/native/column_reader.cpp | 3 -
.../parquet/reader/native/page_reader.cpp | 22 +-
be/test/format_v2/parquet/native_decoder_test.cpp | 39 ++
.../format_v2/parquet/parquet_encryption_test.cpp | 689 ++++++++++++++++++++-
.../encryption/GenerateEncryptedIndexes.java | 63 ++
.../parquet/test_data/encryption/README.md | 50 +-
.../test_data/encryption/indexes_java.parquet | Bin 0 -> 70965 bytes
.../encryption/indexes_java_plain.parquet | Bin 0 -> 29038 bytes
.../iceberg/encryption/indexes_java.parquet | Bin 0 -> 70965 bytes
.../iceberg/encryption/indexes_java_plain.parquet | Bin 0 -> 29038 bytes
.../test_iceberg_encryption_native_reader.out | 20 +
.../test_iceberg_encryption_native_reader.groovy | 31 +-
18 files changed, 1311 insertions(+), 121 deletions(-)
diff --git a/be/src/format_v2/parquet/parquet_crypto.cpp
b/be/src/format_v2/parquet/parquet_crypto.cpp
index ebc7807b9c2..9ec681afacf 100644
--- a/be/src/format_v2/parquet/parquet_crypto.cpp
+++ b/be/src/format_v2/parquet/parquet_crypto.cpp
@@ -21,10 +21,12 @@
#include <algorithm>
#include <limits>
#include <string>
+#include <type_traits>
#include <utility>
#include "gen_cpp/PlanNodes_types.h"
#include "util/coding.h"
+#include "util/defer_op.h"
#if __has_include(<parquet/encryption/encryption_internal.h>)
#include <arrow/util/secure_string.h>
@@ -51,45 +53,134 @@ using apache::thrift::protocol::TList;
using apache::thrift::protocol::TMap;
using apache::thrift::protocol::TProtocolException;
using apache::thrift::protocol::TSet;
+using apache::thrift::protocol::TType;
using apache::thrift::transport::TMemoryBuffer;
-class BoundedCryptoProtocol final : public TCompactProtocol {
+constexpr size_t MAX_FOOTER_NATIVE_ELEMENT_BYTES =
+ std::max({sizeof(tparquet::ColumnChunk),
sizeof(tparquet::SchemaElement),
+ sizeof(tparquet::RowGroup)});
+constexpr size_t MAX_INDEX_NATIVE_ELEMENT_BYTES =
+ std::max({sizeof(std::string), sizeof(tparquet::PageLocation),
sizeof(int64_t)});
+
+class BoundedCryptoProtocol : public TCompactProtocol {
public:
- explicit BoundedCryptoProtocol(const std::shared_ptr<TMemoryBuffer>&
transport, int limit)
- : TCompactProtocol(transport, limit, 65536) {}
+ explicit BoundedCryptoProtocol(const std::shared_ptr<TMemoryBuffer>&
transport, int limit,
+ size_t max_element_size =
MAX_FOOTER_NATIVE_ELEMENT_BYTES,
+ int container_limit = 65536)
+ : TCompactProtocol(transport, limit, container_limit),
+ _max_element_size(max_element_size) {}
protected:
void checkReadBytesAvailable(TList& list) override {
TCompactProtocol::checkReadBytesAvailable(list);
- charge(list.size_);
+ charge(list.size_, _max_element_size);
}
void checkReadBytesAvailable(TSet& set) override {
TCompactProtocol::checkReadBytesAvailable(set);
- charge(set.size_);
+ charge(set.size_, _max_element_size);
}
void checkReadBytesAvailable(TMap& map) override {
TCompactProtocol::checkReadBytesAvailable(map);
// Both the key and value containers contribute to the native
allocation budget.
- charge(map.size_);
- charge(map.size_);
+ charge(map.size_, _max_element_size);
+ charge(map.size_, _max_element_size);
}
-private:
- void charge(uint32_t count) {
+ void charge(uint32_t count, size_t element_size) {
// Generated readers resize their statically typed vectors without
checking the wire element
// type: a list advertised as BYTE can still allocate ColumnChunks.
Charge every element at
- // the largest native size; trusting the wire type would undercount
those allocations.
- constexpr size_t unit =
- std::max({sizeof(tparquet::ColumnChunk),
sizeof(tparquet::SchemaElement),
- sizeof(tparquet::RowGroup)});
- if (count > _budget / unit) {
+ // its expected native size, independently of the wire type.
+ if (count > _budget / element_size) {
throw TProtocolException(TProtocolException::SIZE_LIMIT);
}
- _budget -= count * unit;
+ _budget -= count * element_size;
}
+
+private:
+ const size_t _max_element_size;
size_t _budget = MAX_ENCRYPTED_PARQUET_MODULE_BYTES;
};
+class BoundedIndexProtocol final : public BoundedCryptoProtocol {
+public:
+ BoundedIndexProtocol(const std::shared_ptr<TMemoryBuffer>& transport, int
limit,
+ bool column_index)
+ // A 22,000-page nested column can have 66,000 histogram bins.
Index lists are bounded
+ // by native bytes, while the general/footer protocol retains its
65,536-element cap.
+ : BoundedCryptoProtocol(
+ transport, limit, MAX_INDEX_NATIVE_ELEMENT_BYTES,
+ static_cast<int>(MAX_ENCRYPTED_PARQUET_MODULE_BYTES /
sizeof(int64_t))),
+ _column_index(column_index) {}
+
+protected:
+ using BoundedCryptoProtocol::checkReadBytesAvailable;
+
+ uint32_t readStructBegin_virt(std::string& name) override {
+ const auto bytes = TCompactProtocol::readStructBegin(name);
+ ++_struct_depth;
+ return bytes;
+ }
+ uint32_t readStructEnd_virt() override {
+ const auto bytes = TCompactProtocol::readStructEnd();
+ --_struct_depth;
+ return bytes;
+ }
+ uint32_t readFieldBegin_virt(std::string& name, TType& type, int16_t& id)
override {
+ const auto bytes = TCompactProtocol::readFieldBegin(name, type, id);
+ _field_type = type;
+ _field_id = id;
+ return bytes;
+ }
+ uint32_t skip_virt(TType type) override {
+ // Thrift's template skip bypasses the virtual field/depth hooks. Its
nested containers
+ // must use the conservative root maximum rather than borrow a known
field's smaller unit.
+ const bool previous = std::exchange(_skipping, true);
+ Defer restore {[this, previous] { _skipping = previous; }};
+ return TCompactProtocol::skip(type);
+ }
+ void checkReadBytesAvailable(TList& list) override {
+ TCompactProtocol::checkReadBytesAvailable(list);
+ charge(list.size_, native_element_size());
+ }
+
+private:
+ size_t native_element_size() const {
+ if (_skipping || _struct_depth != 1 || _field_type !=
apache::thrift::protocol::T_LIST) {
+ return MAX_INDEX_NATIVE_ELEMENT_BYTES;
+ }
+ // These are expected Parquet field IDs, not the untrusted advertised
list element type.
+ // Charging i64 histograms as strings would reject 3M counters needing
only 24 MiB.
+ if (_column_index) {
+ switch (_field_id) {
+ case 1: // null_pages: conservatively cover the packed bitmap's
native words.
+ case 5: // null_counts
+ case 6: // repetition_level_histograms
+ case 7: // definition_level_histograms
+ return sizeof(int64_t);
+ case 2: // min_values
+ case 3: // max_values
+ return sizeof(std::string);
+ default:
+ return MAX_INDEX_NATIVE_ELEMENT_BYTES;
+ }
+ }
+ switch (_field_id) {
+ case 1: // page_locations
+ return sizeof(tparquet::PageLocation);
+ case 2: // unencoded_byte_array_data_bytes
+ return sizeof(int64_t);
+ default:
+ return MAX_INDEX_NATIVE_ELEMENT_BYTES;
+ }
+ }
+
+ const bool _column_index;
+ uint32_t _struct_depth = 0;
+ TType _field_type = apache::thrift::protocol::T_STOP;
+ int16_t _field_id = 0;
+ bool _skipping = false;
+};
+
template <typename T>
Status deserialize_bounded(std::span<const uint8_t> bytes, uint32_t* consumed,
T* metadata) {
if (bytes.empty() || bytes.size() > MAX_ENCRYPTED_PARQUET_MODULE_BYTES) {
@@ -102,8 +193,16 @@ Status deserialize_bounded(std::span<const uint8_t> bytes,
uint32_t* consumed, T
auto transport =
std::make_shared<TMemoryBuffer>(const_cast<uint8_t*>(bytes.data()),
static_cast<uint32_t>(bytes.size()),
TMemoryBuffer::OBSERVE, configuration);
- BoundedCryptoProtocol protocol(transport,
static_cast<int>(bytes.size()));
- metadata->read(&protocol);
+ if constexpr (std::is_same_v<T, tparquet::ColumnIndex> ||
+ std::is_same_v<T, tparquet::OffsetIndex>) {
+ BoundedIndexProtocol protocol(transport,
static_cast<int>(bytes.size()),
+ std::is_same_v<T,
tparquet::ColumnIndex>);
+ metadata->read(&protocol);
+ } else {
+ // Page headers keep their existing protocol and avoid index-only
field tracking.
+ BoundedCryptoProtocol protocol(transport,
static_cast<int>(bytes.size()));
+ metadata->read(&protocol);
+ }
*consumed = static_cast<uint32_t>(bytes.size()) -
transport->available_read();
return Status::OK();
} catch (const Exception& e) {
@@ -167,6 +266,19 @@ Status deserialize_crypto_metadata(std::span<const
uint8_t> bytes, uint32_t* con
return deserialize_bounded(bytes, consumed, metadata);
}
+Status deserialize_crypto_metadata(std::span<const uint8_t> bytes, uint32_t*
consumed,
+ tparquet::ColumnIndex* metadata) {
+ return deserialize_bounded(bytes, consumed, metadata);
+}
+Status deserialize_crypto_metadata(std::span<const uint8_t> bytes, uint32_t*
consumed,
+ tparquet::OffsetIndex* metadata) {
+ return deserialize_bounded(bytes, consumed, metadata);
+}
+Status deserialize_crypto_metadata(std::span<const uint8_t> bytes, uint32_t*
consumed,
+ tparquet::BloomFilterHeader* metadata) {
+ return deserialize_bounded(bytes, consumed, metadata);
+}
+
struct ParquetFileCrypto::Impl {
#if __has_include(<parquet/encryption/encryption_internal.h>)
std::shared_ptr<const crypto_detail::FileKey> key;
@@ -291,6 +403,22 @@ Status ParquetFileCrypto::column_crypto(const
tparquet::ColumnChunk& column, int
#endif
}
+Status ParquetColumnCrypto::decrypt_metadata(ParquetMetadataModule module,
+ std::span<const uint8_t> bytes,
+ DorisVector<uint8_t>* plaintext) {
+#if __has_include(<parquet/encryption/encryption_internal.h>)
+ // Index and Bloom modules bind the physical column and row group, but
have no page ordinal.
+ // In particular, a ColumnIndex (6) cannot authenticate an OffsetIndex (7)
for the same column.
+ return decrypt_module(_impl->header.get(), *_impl->key,
+ ::parquet::encryption::CreateModuleAad(
+ _impl->key->file_aad,
static_cast<int8_t>(module),
+ _impl->row_group, _impl->column, -1),
+ bytes, plaintext);
+#else
+ return Status::NotSupported("File encryption
[PME_DEPENDENCY_UNAVAILABLE]");
+#endif
+}
+
Status ParquetColumnCrypto::plaintext_size(int32_t ciphertext_size, int32_t*
size) const {
#if __has_include(<parquet/encryption/encryption_internal.h>)
if (ciphertext_size < 0 ||
diff --git a/be/src/format_v2/parquet/parquet_crypto.h
b/be/src/format_v2/parquet/parquet_crypto.h
index 77f94aab731..05086b4295e 100644
--- a/be/src/format_v2/parquet/parquet_crypto.h
+++ b/be/src/format_v2/parquet/parquet_crypto.h
@@ -42,6 +42,20 @@ Status deserialize_crypto_metadata(std::span<const uint8_t>
bytes, uint32_t* con
Status deserialize_crypto_metadata(std::span<const uint8_t> bytes, uint32_t*
consumed,
tparquet::PageHeader* metadata);
+Status deserialize_crypto_metadata(std::span<const uint8_t> bytes, uint32_t*
consumed,
+ tparquet::ColumnIndex* metadata);
+Status deserialize_crypto_metadata(std::span<const uint8_t> bytes, uint32_t*
consumed,
+ tparquet::OffsetIndex* metadata);
+Status deserialize_crypto_metadata(std::span<const uint8_t> bytes, uint32_t*
consumed,
+ tparquet::BloomFilterHeader* metadata);
+
+enum class ParquetMetadataModule : int8_t {
+ COLUMN_INDEX = 6,
+ OFFSET_INDEX = 7,
+ BLOOM_FILTER_HEADER = 8,
+ BLOOM_FILTER_BITSET = 9,
+};
+
class ParquetColumnCrypto;
// Immutable file key/AAD owner. Encrypted metadata is never admitted to
shared caches.
@@ -68,6 +82,8 @@ private:
// References extend one column's reader stack only; never reuse an instance
between readers.
class ParquetColumnCrypto {
public:
+ Status decrypt_metadata(ParquetMetadataModule module, std::span<const
uint8_t> bytes,
+ DorisVector<uint8_t>* plaintext);
~ParquetColumnCrypto();
Status plaintext_size(int32_t ciphertext_size, int32_t* size) const;
Status decrypt_header(std::span<const uint8_t> bytes, bool dictionary,
int32_t page,
diff --git a/be/src/format_v2/parquet/parquet_file_context.cpp
b/be/src/format_v2/parquet/parquet_file_context.cpp
index c2baa8efa6b..d164d878369 100644
--- a/be/src/format_v2/parquet/parquet_file_context.cpp
+++ b/be/src/format_v2/parquet/parquet_file_context.cpp
@@ -378,14 +378,29 @@ Status ParquetFileContext::open(io::FileReaderSPtr
input_file_reader, io::IOCont
return Status::OK();
}
+namespace {
+template <typename T>
+Status parse_native_index(std::span<const uint8_t> bytes, ParquetColumnCrypto*
crypto,
+ ParquetMetadataModule module, T* index, bool*
parsed) {
+ auto consumed = static_cast<uint32_t>(bytes.size());
+ if (crypto == nullptr) {
+ *parsed = deserialize_thrift_msg(bytes.data(), &consumed, true,
index).ok();
+ return Status::OK();
+ }
+ DorisVector<uint8_t> plaintext;
+ RETURN_IF_ERROR(crypto->decrypt_metadata(module, bytes, &plaintext));
+ RETURN_IF_ERROR(deserialize_crypto_metadata(plaintext, &consumed, index));
+ RETURN_IF_ERROR(validate_crypto_metadata_padding(plaintext, consumed));
+ *parsed = true;
+ return Status::OK();
+}
+} // namespace
+
Status ParquetFileContext::load_native_offset_indexes(
int row_group_id, const std::unordered_set<int>& leaf_column_ids,
std::unordered_map<int, tparquet::OffsetIndex>* offset_indexes) const {
DORIS_CHECK(offset_indexes != nullptr);
offset_indexes->clear();
- if (native_metadata->encryption() != nullptr) {
- return Status::OK();
- }
if (leaf_column_ids.empty()) {
return Status::OK();
}
@@ -405,31 +420,47 @@ Status ParquetFileContext::load_native_offset_indexes(
}
const auto& column_chunk =
native_row_group.columns[leaf_column_id];
if (!column_chunk.__isset.offset_index_offset ||
- !column_chunk.__isset.offset_index_length ||
- column_chunk.offset_index_length <= 0) {
+ !column_chunk.__isset.offset_index_length) {
continue;
}
const int64_t index_offset = column_chunk.offset_index_offset;
const int64_t index_length = column_chunk.offset_index_length;
if (!detail::is_serialized_index_range_safe(native_file->size(),
index_offset,
index_length)) {
- // OffsetIndex is optional. A malformed range must not
allocate from untrusted
- // footer values or redirect the native reader outside the
file.
+ // An absent index is optional; an authenticated descriptor
with an impossible
+ // range (for example offset -1) is corrupt and must fail
before allocation or I/O.
+ if (native_metadata->encryption() != nullptr) {
+ return Status::Corruption("File encryption
[INVALID_INDEX_RANGE]");
+ }
continue;
}
- std::vector<uint8_t>
serialized_index(static_cast<size_t>(index_length));
+ std::shared_ptr<ParquetColumnCrypto> crypto;
+ if (native_metadata->encryption() != nullptr) {
+ RETURN_IF_ERROR(native_metadata->encryption()->column_crypto(
+ column_chunk, row_group_id, leaf_column_id, &crypto));
+ }
+ DorisVector<uint8_t>
serialized_index(static_cast<size_t>(index_length));
Slice index_slice(serialized_index.data(),
serialized_index.size());
size_t bytes_read = 0;
- if (!native_file->read_at(index_offset, index_slice, &bytes_read,
native_io_ctx).ok() ||
- bytes_read != serialized_index.size()) {
+ const auto read_status =
+ native_file->read_at(index_offset, index_slice,
&bytes_read, native_io_ctx);
+ if (!read_status.ok() || bytes_read != serialized_index.size()) {
+ if (crypto != nullptr) {
+ RETURN_IF_ERROR(read_status);
+ return Status::Corruption("File encryption
[INDEX_READ_FAILED]");
+ }
continue;
}
- uint32_t thrift_length =
static_cast<uint32_t>(serialized_index.size());
tparquet::OffsetIndex native_index;
- if (!deserialize_thrift_msg(serialized_index.data(),
&thrift_length, true,
- &native_index)
- .ok() ||
- native_index.page_locations.empty()) {
+ bool parsed = false;
+ RETURN_IF_ERROR(parse_native_index(serialized_index, crypto.get(),
+
ParquetMetadataModule::OFFSET_INDEX, &native_index,
+ &parsed));
+ if (!parsed || native_index.page_locations.empty()) {
+ if (crypto != nullptr) {
+ // An authenticated empty map cannot describe a non-empty
column chunk.
+ return Status::Corruption("File encryption
[INVALID_INDEX]");
+ }
continue;
}
native::ColumnChunkRange chunk_range;
@@ -440,13 +471,19 @@ Status ParquetFileContext::load_native_offset_indexes(
native_index, chunk_range,
native_row_group.columns[leaf_column_id].meta_data.data_page_offset,
native_row_group.num_rows)) {
- // OffsetIndex is optional. Reject the complete index instead
of letting one bad
- // location redirect an indexed reader outside its owning
column chunk.
+ // Reject an authenticated skip map with an invalid physical
rectangle. A location
+ // outside its owning chunk must never determine a page's
authenticated ordinal.
+ if (crypto != nullptr) {
+ return Status::Corruption("File encryption
[INVALID_INDEX]");
+ }
continue;
}
offset_indexes->emplace(leaf_column_id, std::move(native_index));
}
} catch (const std::exception&) {
+ if (native_metadata->encryption() != nullptr) {
+ return Status::Corruption("File encryption [INVALID_INDEX]");
+ }
// OffsetIndex is optional. Selected logical ranges still enforce
correctness, while the
// native reader conservatively falls back to sequential page
traversal.
offset_indexes->clear();
@@ -460,9 +497,6 @@ Status ParquetFileContext::load_native_page_indexes(
int64_t* parse_time) const {
DORIS_CHECK(page_indexes != nullptr);
page_indexes->clear();
- if (native_metadata->encryption() != nullptr) {
- return Status::OK();
- }
if (leaf_column_ids.empty()) {
return Status::OK();
}
@@ -483,6 +517,7 @@ Status ParquetFileContext::load_native_page_indexes(
NativeParquetPageIndex indexes;
bool has_column_index = false;
bool has_offset_index = false;
+ std::shared_ptr<ParquetColumnCrypto> crypto;
};
std::vector<SerializedIndexRange> column_index_ranges;
std::vector<SerializedIndexRange> offset_index_ranges;
@@ -503,38 +538,56 @@ Status ParquetFileContext::load_native_page_indexes(
}
if (!valid_index_range(chunk.column_index_offset,
chunk.column_index_length) ||
!valid_index_range(chunk.offset_index_offset,
chunk.offset_index_length)) {
+ // The encrypted footer authenticated these locations; invalid
ranges are corrupt
+ // descriptors, while genuinely missing indexes above still retain
all candidate rows.
+ if (native_metadata->encryption() != nullptr) {
+ return Status::Corruption("File encryption
[INVALID_INDEX_RANGE]");
+ }
continue;
}
column_index_ranges.push_back(
{leaf_column_id, chunk.column_index_offset,
chunk.column_index_length});
offset_index_ranges.push_back(
{leaf_column_id, chunk.offset_index_offset,
chunk.offset_index_length});
- pending_indexes.try_emplace(leaf_column_id);
+ auto& pending = pending_indexes[leaf_column_id];
+ if (native_metadata->encryption() != nullptr) {
+ RETURN_IF_ERROR(native_metadata->encryption()->column_crypto(
+ chunk, row_group_id, leaf_column_id, &pending.crypto));
+ }
}
auto read_coalesced_indexes = [&](std::vector<SerializedIndexRange>*
ranges,
- bool column_index) {
+ bool column_index) -> Status {
std::sort(ranges->begin(), ranges->end(),
[](const auto& lhs, const auto& rhs) { return lhs.offset <
rhs.offset; });
size_t range_begin = 0;
while (range_begin < ranges->size()) {
+ const int64_t span_offset = (*ranges)[range_begin].offset;
size_t range_end = range_begin + 1;
int64_t span_end = (*ranges)[range_begin].offset +
(*ranges)[range_begin].length;
while (range_end < ranges->size() && (*ranges)[range_end].offset
<= span_end) {
- span_end = std::max(span_end,
- (*ranges)[range_end].offset +
(*ranges)[range_end].length);
+ const int64_t next_span_end = std::max(
+ span_end, (*ranges)[range_end].offset +
(*ranges)[range_end].length);
+ // A wide row group can have valid adjacent modules totaling
more than one read's
+ // budget. Split the I/O block before that limit instead of
rejecting the file.
+ if (!detail::is_serialized_index_span_safe(span_offset,
next_span_end)) {
+ break;
+ }
+ span_end = next_span_end;
++range_end;
}
- const int64_t span_offset = (*ranges)[range_begin].offset;
if (!detail::is_serialized_index_span_safe(span_offset, span_end))
{
- // Optional indexes share one allocation per contiguous footer
block. Skipping the
- // whole block prevents many individually small ranges from
bypassing the budget.
+ // Optional indexes share one allocation per contiguous footer
block. Bound the
+ // whole block so adjacent small descriptors cannot bypass the
allocation budget.
+ if (native_metadata->encryption() != nullptr) {
+ return Status::Corruption("File encryption
[INVALID_INDEX_RANGE]");
+ }
range_begin = range_end;
continue;
}
const int64_t span_length = span_end - span_offset;
- std::vector<uint8_t> serialized(static_cast<size_t>(span_length));
+ DorisVector<uint8_t> serialized(static_cast<size_t>(span_length));
Slice slice(serialized.data(), serialized.size());
size_t bytes_read = 0;
Status read_status;
@@ -555,34 +608,45 @@ Status ParquetFileContext::load_native_page_indexes(
serialized.data() +
static_cast<size_t>(range.offset - span_offset);
int64_t parse_time_sink = 0;
SCOPED_RAW_TIMER(parse_time == nullptr ? &parse_time_sink
: parse_time);
+ const auto bytes = std::span(thrift_data,
static_cast<size_t>(thrift_length));
if (column_index) {
- pending->second.has_column_index =
- deserialize_thrift_msg(thrift_data,
&thrift_length, true,
-
&pending->second.indexes.column_index)
- .ok();
+ RETURN_IF_ERROR(parse_native_index(bytes,
pending->second.crypto.get(),
+
ParquetMetadataModule::COLUMN_INDEX,
+
&pending->second.indexes.column_index,
+
&pending->second.has_column_index));
} else {
- pending->second.has_offset_index =
- deserialize_thrift_msg(thrift_data,
&thrift_length, true,
-
&pending->second.indexes.offset_index)
- .ok();
+ RETURN_IF_ERROR(parse_native_index(bytes,
pending->second.crypto.get(),
+
ParquetMetadataModule::OFFSET_INDEX,
+
&pending->second.indexes.offset_index,
+
&pending->second.has_offset_index));
}
}
}
+ if (native_metadata->encryption() != nullptr &&
+ (!read_status.ok() || bytes_read != serialized.size())) {
+ RETURN_IF_ERROR(read_status);
+ return Status::Corruption("File encryption
[INDEX_READ_FAILED]");
+ }
range_begin = range_end;
}
+ return Status::OK();
};
// Parquet writers place each index kind in a contiguous block. Reading
overlapping/adjacent
// ranges as one span keeps cold small-file planning from paying two
remote round trips per
// projected leaf, while refusing gaps avoids amplifying reads from
untrusted footer offsets.
- read_coalesced_indexes(&column_index_ranges, true);
- read_coalesced_indexes(&offset_index_ranges, false);
+ RETURN_IF_ERROR(read_coalesced_indexes(&column_index_ranges, true));
+ RETURN_IF_ERROR(read_coalesced_indexes(&offset_index_ranges, false));
for (auto& [leaf_column_id, pending] : pending_indexes) {
const auto& chunk = row_group.columns[leaf_column_id];
auto& indexes = pending.indexes;
if (!pending.has_column_index || !pending.has_offset_index ||
indexes.column_index.null_pages.size() !=
indexes.offset_index.page_locations.size()) {
+ if (pending.crypto != nullptr) {
+ // A decoded page bitmap must have one entry for every
authenticated page location.
+ return Status::Corruption("File encryption [INVALID_INDEX]");
+ }
continue;
}
native::ColumnChunkRange chunk_range;
@@ -590,6 +654,10 @@ Status ParquetFileContext::load_native_page_indexes(
chunk.meta_data, native_file->size(),
compat.parquet_816_padding, &chunk_range));
if (!native::validate_offset_index(indexes.offset_index, chunk_range,
chunk.meta_data.data_page_offset,
row_group.num_rows)) {
+ if (pending.crypto != nullptr) {
+ // Invalid physical locations cannot supply authenticated page
ordinals to a seek.
+ return Status::Corruption("File encryption [INVALID_INDEX]");
+ }
continue;
}
page_indexes->emplace(leaf_column_id, std::move(indexes));
diff --git a/be/src/format_v2/parquet/parquet_reader.cpp
b/be/src/format_v2/parquet/parquet_reader.cpp
index afac87943e1..d9ad0c8df03 100644
--- a/be/src/format_v2/parquet/parquet_reader.cpp
+++ b/be/src/format_v2/parquet/parquet_reader.cpp
@@ -655,7 +655,6 @@ Status ParquetReader::init(RuntimeState* state) {
_clear_decryption_parameters();
RETURN_IF_ERROR(footer_status);
if (_state->file_context.native_metadata->encryption() != nullptr) {
- _state->enable_bloom_filter = false;
_state->enable_page_cache = false;
}
}
diff --git a/be/src/format_v2/parquet/parquet_scan.cpp
b/be/src/format_v2/parquet/parquet_scan.cpp
index d6b51bf56bd..4ecea4ab08e 100644
--- a/be/src/format_v2/parquet/parquet_scan.cpp
+++ b/be/src/format_v2/parquet/parquet_scan.cpp
@@ -763,12 +763,6 @@ Status finalize_native_row_group_read_plan(
return Status::OK();
}
row_group_plan->expensive_pruning_pending = false;
- // Encrypted indexes and dictionary pruning are a separate integration
stage. Keep this
- // path sequential, including callers that finalize row groups lazily in
the scheduler.
- if (metadata.encryption() != nullptr) {
- pruning_stats->selected_row_ranges +=
row_group_plan->selected_ranges.size();
- return Status::OK();
- }
const auto& thrift = metadata.to_thrift();
const auto row_group_leaf_ids =
row_group_leaf_column_ids(file_schema, request, *row_group_plan,
request_leaf_ids);
@@ -880,12 +874,6 @@ Status plan_parquet_row_groups(const
NativeParquetMetadata& metadata,
RETURN_IF_ERROR(build_native_row_group_read_plans(metadata, file_schema,
request,
scan_range_selected,
row_group_first_rows,
plan, timezone,
runtime_state, file_context));
- if (metadata.encryption() != nullptr) {
- plan->enable_bloom_filter = false;
- plan->pruning_stats.total_row_groups =
cast_set<int64_t>(scan_range_selected.size());
- plan->pruning_stats.selected_row_groups = plan->row_groups.size();
- return Status::OK();
- }
std::vector<RowGroupReadPlan> metadata_selected_plans;
metadata_selected_plans.reserve(plan->row_groups.size());
for (auto& row_group_plan : plan->row_groups) {
diff --git a/be/src/format_v2/parquet/parquet_statistics.cpp
b/be/src/format_v2/parquet/parquet_statistics.cpp
index 3e15a5bc3f4..478bbbce026 100644
--- a/be/src/format_v2/parquet/parquet_statistics.cpp
+++ b/be/src/format_v2/parquet/parquet_statistics.cpp
@@ -43,6 +43,7 @@
#include "exprs/vliteral.h"
#include "exprs/vslot_ref.h"
#include "format_v2/parquet/parquet_column_schema.h"
+#include "format_v2/parquet/parquet_crypto.h"
#include "format_v2/parquet/parquet_file_context.h"
#include "format_v2/parquet/reader/native/block_split_bloom_filter.h"
#include "format_v2/parquet/reader/native/column_chunk_reader.h"
@@ -53,6 +54,7 @@
#include "storage/index/bloom_filter/bloom_filter.h"
#include "storage/index/zone_map/zone_map_index.h"
#include "storage/index/zone_map/zonemap_eval_context.h"
+#include "util/coding.h"
#include "util/thrift_util.h"
#include "util/unaligned.h"
@@ -145,12 +147,94 @@ enum class ParquetRowGroupPruneReason {
BLOOM_FILTER, // excluded by bloom filter
};
+Status read_encrypted_bloom_module(const io::FileReaderSPtr& file,
io::IOContext* io_ctx,
+ size_t offset, size_t available,
ParquetColumnCrypto* crypto,
+ ParquetMetadataModule module,
DorisVector<uint8_t>* plaintext,
+ size_t* wire_size) {
+ if (available < 4) {
+ return Status::Corruption("File encryption [INVALID_MODULE_SIZE]");
+ }
+ uint8_t prefix[4];
+ size_t read = 0;
+ RETURN_IF_ERROR(file->read_at(offset, Slice(prefix, sizeof(prefix)),
&read, io_ctx));
+ if (read != sizeof(prefix)) {
+ return Status::Corruption("File encryption [INVALID_MODULE_SIZE]");
+ }
+ const uint64_t length = static_cast<uint64_t>(decode_fixed32_le(prefix)) +
4;
+ if (length > available || length > MAX_ENCRYPTED_PARQUET_MODULE_BYTES) {
+ return Status::Corruption("File encryption [INVALID_MODULE_SIZE]");
+ }
+ DorisVector<uint8_t> bytes(static_cast<size_t>(length));
+ RETURN_IF_ERROR(file->read_at(offset, Slice(bytes.data(), bytes.size()),
&read, io_ctx));
+ if (read != bytes.size()) {
+ return Status::Corruption("File encryption [INVALID_MODULE_SIZE]");
+ }
+ RETURN_IF_ERROR(crypto->decrypt_metadata(module, bytes, plaintext));
+ *wire_size = bytes.size();
+ return Status::OK();
+}
+
+Status read_encrypted_bloom_filter(const tparquet::ColumnMetaData& metadata,
+ const io::FileReaderSPtr& file,
io::IOContext* io_ctx,
+ ParquetColumnCrypto* crypto,
+
std::unique_ptr<native::BlockSplitBloomFilter>* result) {
+ if (metadata.bloom_filter_offset < 0 ||
+ std::cmp_greater_equal(metadata.bloom_filter_offset, file->size())) {
+ return Status::Corruption("File encryption [INVALID_BLOOM_RANGE]");
+ }
+ const auto offset = static_cast<size_t>(metadata.bloom_filter_offset);
+ size_t available = file->size() - offset;
+ if (metadata.__isset.bloom_filter_length) {
+ if (metadata.bloom_filter_length <= 0 ||
+ std::cmp_greater(metadata.bloom_filter_length, available)) {
+ return Status::Corruption("File encryption [INVALID_BLOOM_RANGE]");
+ }
+ available = static_cast<size_t>(metadata.bloom_filter_length);
+ }
+ DorisVector<uint8_t> plaintext;
+ size_t header_wire_size = 0;
+ RETURN_IF_ERROR(read_encrypted_bloom_module(file, io_ctx, offset,
available, crypto,
+
ParquetMetadataModule::BLOOM_FILTER_HEADER,
+ &plaintext,
&header_wire_size));
+ tparquet::BloomFilterHeader header;
+ uint32_t consumed = 0;
+ RETURN_IF_ERROR(deserialize_crypto_metadata(plaintext, &consumed,
&header));
+ RETURN_IF_ERROR(validate_crypto_metadata_padding(plaintext, consumed));
+ if (!header.algorithm.__isset.BLOCK ||
!header.compression.__isset.UNCOMPRESSED ||
+ !header.hash.__isset.XXHASH) {
+ return Status::NotSupported("Unsupported Parquet Bloom filter
encoding");
+ }
+ if (std::cmp_less(header.numBytes, segment_v2::BloomFilter::MINIMUM_BYTES)
||
+ header.numBytes % 32 != 0 ||
+ std::cmp_greater(header.numBytes, MAX_ENCRYPTED_PARQUET_MODULE_BYTES -
32)) {
+ return Status::Corruption("File encryption [INVALID_BLOOM_SIZE]");
+ }
+ size_t bitset_wire_size = 0;
+ // Header and bitset are independently framed/authenticated. Never
interpret the encrypted
+ // header's byte length as its decoded Thrift length when locating the
following bitset.
+ RETURN_IF_ERROR(read_encrypted_bloom_module(
+ file, io_ctx, offset + header_wire_size, available -
header_wire_size, crypto,
+ ParquetMetadataModule::BLOOM_FILTER_BITSET, &plaintext,
&bitset_wire_size));
+ if (std::cmp_not_equal(plaintext.size(), header.numBytes)) {
+ return Status::Corruption("File encryption [INVALID_BLOOM_SIZE]");
+ }
+ auto bloom = std::make_unique<native::BlockSplitBloomFilter>();
+ RETURN_IF_ERROR(bloom->init(reinterpret_cast<const
char*>(plaintext.data()), plaintext.size(),
+ segment_v2::HashStrategyPB::XX_HASH_64));
+ *result = std::move(bloom);
+ return Status::OK();
+}
+
Status read_native_bloom_filter(const tparquet::ColumnMetaData& metadata,
const io::FileReaderSPtr& file, io::IOContext*
io_ctx,
-
std::unique_ptr<native::BlockSplitBloomFilter>* result) {
+
std::unique_ptr<native::BlockSplitBloomFilter>* result,
+ ParquetColumnCrypto* crypto = nullptr) {
if (result == nullptr || file == nullptr ||
!metadata.__isset.bloom_filter_offset) {
return Status::NotSupported("Parquet Bloom filter is unavailable");
}
+ if (crypto != nullptr) {
+ return read_encrypted_bloom_filter(metadata, file, io_ctx, crypto,
result);
+ }
constexpr size_t MAX_BLOOM_HEADER_BYTES = 64;
if (metadata.bloom_filter_offset < 0 ||
(metadata.__isset.bloom_filter_length && metadata.bloom_filter_length
<= 0)) {
@@ -1370,8 +1454,58 @@ ParquetRowGroupPruneReason
native_dictionary_prune_reason(
return ParquetRowGroupPruneReason::NONE;
}
-ParquetRowGroupPruneReason native_bloom_filter_prune_reason(
- const tparquet::RowGroup& row_group,
+Result<std::unique_ptr<native::BlockSplitBloomFilter>>
read_row_group_bloom_filter(
+ const tparquet::ColumnChunk& chunk, int row_group_idx, int
leaf_column_id,
+ ParquetFileContext* file_context, ParquetPruningStats* pruning_stats) {
+ if (!chunk.meta_data.__isset.bloom_filter_offset) {
+ // A missing Bloom filter is optional even in an encrypted file. Only
a present module
+ // must authenticate successfully before it can participate in pruning.
+ if (pruning_stats != nullptr) {
+ ++pruning_stats->bloom_filter_conservative_fallbacks;
+ }
+ return std::unique_ptr<native::BlockSplitBloomFilter>();
+ }
+ std::shared_ptr<ParquetColumnCrypto> crypto;
+ if (file_context->native_metadata != nullptr &&
+ file_context->native_metadata->encryption() != nullptr) {
+
RETURN_IF_ERROR_RESULT(file_context->native_metadata->encryption()->column_crypto(
+ chunk, row_group_idx, leaf_column_id, &crypto));
+ }
+ std::unique_ptr<native::BlockSplitBloomFilter> bloom_filter;
+ Status bloom_status;
+ int64_t timer_sink = 0;
+ {
+ SCOPED_RAW_TIMER(pruning_stats == nullptr ? &timer_sink
+ :
&pruning_stats->bloom_filter_read_time);
+ bloom_status =
+ read_native_bloom_filter(chunk.meta_data,
file_context->native_file,
+ file_context->native_io_ctx,
&bloom_filter, crypto.get());
+ if (!bloom_status.ok()) {
+ bloom_filter.reset();
+ }
+ }
+ if (crypto != nullptr && !bloom_status.ok()) {
+ // Authentication failures are query failures, not optional-index
misses. Retaining
+ // rows here would hide a swapped or modified Bloom module from the
encrypted scan.
+ return unexpected(std::move(bloom_status));
+ }
+ if (bloom_filter == nullptr) {
+ if (pruning_stats != nullptr) {
+ ++pruning_stats->bloom_filter_conservative_fallbacks;
+ if (bloom_status.is<ErrorCode::CORRUPTION>()) {
+ ++pruning_stats->bloom_filter_corrupt_rejections;
+ }
+ }
+ return std::unique_ptr<native::BlockSplitBloomFilter>();
+ }
+ if (pruning_stats != nullptr) {
+ ++pruning_stats->bloom_filter_probe_successes;
+ }
+ return bloom_filter;
+}
+
+Result<ParquetRowGroupPruneReason> native_bloom_filter_prune_reason(
+ const tparquet::RowGroup& row_group, int row_group_idx,
const std::vector<std::unique_ptr<ParquetColumnSchema>>& file_schema,
const format::FileScanRequest& request, ParquetFileContext*
file_context,
ParquetPruningStats* pruning_stats) {
@@ -1447,31 +1581,15 @@ ParquetRowGroupPruneReason
native_bloom_filter_prune_reason(
}
continue;
}
- std::unique_ptr<native::BlockSplitBloomFilter> bloom_filter;
- Status bloom_status;
- int64_t timer_sink = 0;
- {
- SCOPED_RAW_TIMER(pruning_stats == nullptr ? &timer_sink
- :
&pruning_stats->bloom_filter_read_time);
- bloom_status =
read_native_bloom_filter(row_group.columns[leaf_column_id].meta_data,
- file_context->native_file,
-
file_context->native_io_ctx, &bloom_filter);
- if (!bloom_status.ok()) {
- bloom_filter.reset();
- }
+ auto bloom_filter_result = read_row_group_bloom_filter(chunk,
row_group_idx, leaf_column_id,
+ file_context,
pruning_stats);
+ if (!bloom_filter_result.has_value()) {
+ return unexpected(std::move(bloom_filter_result.error()));
}
+ auto bloom_filter = std::move(*bloom_filter_result);
if (bloom_filter == nullptr) {
- if (pruning_stats != nullptr) {
- ++pruning_stats->bloom_filter_conservative_fallbacks;
- if (bloom_status.is<ErrorCode::CORRUPTION>()) {
- ++pruning_stats->bloom_filter_corrupt_rejections;
- }
- }
continue;
}
- if (pruning_stats != nullptr) {
- ++pruning_stats->bloom_filter_probe_successes;
- }
// Keep at most one decoded payload live while reusing it for every
predicate on this leaf.
for (const auto& probe : leaf_group.probes) {
if (ParquetStatisticsUtils::NativeBloomFilterExcludes(
@@ -1582,16 +1700,24 @@ Status select_row_groups_by_metadata(
null_count_trusted)))) {
prune_reason = ParquetRowGroupPruneReason::STATISTICS;
}
+ // Dictionary probes currently treat reader errors as optional misses.
Keep encrypted
+ // dictionaries on the ordinary decode path, where an authentication
error fails the scan.
if (probe_mode != ParquetMetadataProbeMode::FOOTER_ONLY &&
- prune_reason == ParquetRowGroupPruneReason::NONE) {
+ prune_reason == ParquetRowGroupPruneReason::NONE &&
+ (file_context == nullptr || file_context->native_metadata ==
nullptr ||
+ file_context->native_metadata->encryption() == nullptr)) {
prune_reason =
native_dictionary_prune_reason(row_group, row_group_idx,
file_schema, request,
timezone, file_context,
column_reader_profile);
}
if (probe_mode != ParquetMetadataProbeMode::FOOTER_ONLY &&
prune_reason == ParquetRowGroupPruneReason::NONE &&
enable_bloom_filter) {
- prune_reason = native_bloom_filter_prune_reason(row_group,
file_schema, request,
- file_context,
pruning_stats);
+ auto bloom_reason = native_bloom_filter_prune_reason(
+ row_group, row_group_idx, file_schema, request,
file_context, pruning_stats);
+ if (!bloom_reason.has_value()) {
+ return std::move(bloom_reason).error();
+ }
+ prune_reason = *bloom_reason;
}
if (prune_reason == ParquetRowGroupPruneReason::NONE) {
selected_row_groups->push_back(row_group_idx);
diff --git a/be/src/format_v2/parquet/reader/native/column_reader.cpp
b/be/src/format_v2/parquet/reader/native/column_reader.cpp
index 1b64d3842c2..cc1fd565209 100644
--- a/be/src/format_v2/parquet/reader/native/column_reader.cpp
+++ b/be/src/format_v2/parquet/reader/native/column_reader.cpp
@@ -684,9 +684,6 @@ Status ColumnReader::_create_scalar_reader(const
ColumnReaderCreateContext& cont
}
std::shared_ptr<ParquetColumnCrypto> crypto;
if (context.file_crypto != nullptr) {
- if (offset_index != nullptr) {
- return Status::NotSupported("File encryption
[INDEX_READ_UNAVAILABLE]");
- }
RETURN_IF_ERROR(context.file_crypto->column_crypto(chunk,
context.row_group_ordinal,
physical_index,
&crypto));
}
diff --git a/be/src/format_v2/parquet/reader/native/page_reader.cpp
b/be/src/format_v2/parquet/reader/native/page_reader.cpp
index 374c383435e..fb0305d1e29 100644
--- a/be/src/format_v2/parquet/reader/native/page_reader.cpp
+++ b/be/src/format_v2/parquet/reader/native/page_reader.cpp
@@ -23,6 +23,7 @@
#include <stdint.h>
#include <algorithm>
+#include <limits>
#include <utility>
#include "common/compiler_util.h" // IWYU pragma: keep
@@ -154,8 +155,10 @@ void PageReader<IN_COLLECTION,
OFFSET_INDEX>::_reconcile_offset_index_location(
const auto& location = _offset_index->page_locations[_page_index];
const bool data_page = _cur_page_header.type ==
tparquet::PageType::DATA_PAGE ||
_cur_page_header.type ==
tparquet::PageType::DATA_PAGE_V2;
- const uint64_t serialized_size = static_cast<uint64_t>(header_size) +
-
static_cast<uint64_t>(_cur_page_header.compressed_page_size);
+ // OffsetIndex sizes describe ciphertext on disk. Using the decrypted
compressed size
+ // drops GCM framing/tag bytes, falsely invalidates the index and loses
the seek's row range.
+ const uint64_t serialized_size =
+ static_cast<uint64_t>(header_size) +
static_cast<uint64_t>(_serialized_page_size);
const bool matches = data_page && location.offset >= 0 &&
location.compressed_page_size > 0 &&
header_offset ==
static_cast<uint64_t>(location.offset) &&
serialized_size ==
static_cast<uint64_t>(location.compressed_page_size);
@@ -207,6 +210,8 @@ Status PageReader<IN_COLLECTION,
OFFSET_INDEX>::_try_cached_header(bool* loaded)
&real_header_size, true,
&_cur_page_header);
RETURN_IF_ERROR(st);
RETURN_IF_ERROR(_validate_page_header(real_header_size));
+ // Index reconciliation needs this page's disk size on cache hits
as well as cold reads.
+ _serialized_page_size = _cur_page_header.compressed_page_size;
_reconcile_offset_index_location(_offset, real_header_size);
_update_sequential_row_range();
if (_cur_page_header.type == tparquet::PageType::DATA_PAGE ||
@@ -232,7 +237,6 @@ Status PageReader<IN_COLLECTION,
OFFSET_INDEX>::_try_cached_header(bool* loaded)
_last_header_size = real_header_size;
_page_statistics.parse_page_header_num++;
_offset += real_header_size;
- _serialized_page_size = _cur_page_header.compressed_page_size;
_next_header_offset = _offset + _serialized_page_size;
_state = HEADER_PARSED;
*loaded = true;
@@ -273,6 +277,16 @@ Status PageReader<IN_COLLECTION,
OFFSET_INDEX>::_parse_encrypted_header(uint32_t
_metadata.dictionary_page_offset <
_metadata.data_page_offset &&
std::cmp_equal(_offset,
_metadata.dictionary_page_offset);
_crypto_page = _next_crypto_page;
+ if constexpr (OFFSET_INDEX) {
+ if (_offset_index != nullptr) {
+ // An indexed seek from page 0 to page 3 must authenticate
physical ordinal 3,
+ // rather than the number of headers actually read. Dictionary
pages consume no ordinal.
+ if (std::cmp_greater(_page_index,
std::numeric_limits<int16_t>::max())) {
+ return Status::NotSupported("File encryption
[ORDINAL_OUT_OF_RANGE]");
+ }
+ _crypto_page = static_cast<int32_t>(_page_index);
+ }
+ }
DorisVector<uint8_t> decrypted;
RETURN_IF_ERROR(_page_read_ctx.encryption->decrypt_header(
std::span(bytes, encrypted_size), _crypto_dictionary,
_crypto_page, &decrypted));
@@ -359,7 +373,7 @@ Status PageReader<IN_COLLECTION,
OFFSET_INDEX>::parse_page_header() {
RETURN_IF_ERROR(_validate_page_header(header_size));
if (_cur_page_header.type == tparquet::PageType::DATA_PAGE ||
_cur_page_header.type == tparquet::PageType::DATA_PAGE_V2) {
- ++_next_crypto_page;
+ _next_crypto_page = _crypto_page + 1;
}
}
_reconcile_offset_index_location(_offset, header_size);
diff --git a/be/test/format_v2/parquet/native_decoder_test.cpp
b/be/test/format_v2/parquet/native_decoder_test.cpp
index efa6e86b2eb..98627b3d675 100644
--- a/be/test/format_v2/parquet/native_decoder_test.cpp
+++ b/be/test/format_v2/parquet/native_decoder_test.cpp
@@ -3589,6 +3589,45 @@ TEST(ParquetV2NativeDecoderTest,
PageHeaderRejectsSignedAndV2LevelSizeCorruption
EXPECT_TRUE(parse_header(competing_layouts).is<ErrorCode::CORRUPTION>());
}
+TEST(ParquetV2NativeDecoderTest, CachedHeaderKeepsValidOffsetIndexRectangle) {
+ const auto first_page = serialize_plain_int32_page({10});
+ const auto second_page = serialize_plain_int32_page({20});
+ std::vector<uint8_t> bytes = first_page;
+ bytes.insert(bytes.end(), second_page.begin(), second_page.end());
+ tparquet::OffsetIndex offset_index;
+ std::vector<tparquet::PageLocation> locations(2);
+ locations[0].__set_offset(0);
+ locations[0].__set_compressed_page_size(first_page.size());
+ locations[0].__set_first_row_index(0);
+ locations[1].__set_offset(first_page.size());
+ locations[1].__set_compressed_page_size(second_page.size());
+ locations[1].__set_first_row_index(1);
+ offset_index.__set_page_locations(locations);
+ const std::string cache_key = "valid-index-cached-header-disk-size";
+ auto* page = new DataPage(first_page.size(), true, segment_v2::DATA_PAGE);
+ memcpy(page->data(), first_page.data(), first_page.size());
+ page->reset_size(first_page.size());
+ PageCacheHandle handle;
+ StoragePageCache::instance()->insert(StoragePageCache::CacheKey(cache_key,
bytes.size(), 0),
+ page, &handle, segment_v2::DATA_PAGE);
+ tparquet::ColumnMetaData metadata;
+ metadata.__set_type(tparquet::Type::INT32);
+ metadata.__set_codec(tparquet::CompressionCodec::UNCOMPRESSED);
+ metadata.__set_num_values(2);
+ metadata.__set_total_compressed_size(bytes.size());
+ metadata.__set_data_page_offset(0);
+ MemoryBufferedReader stream(std::move(bytes));
+ PageReader<false, true> reader(&stream, nullptr, 0,
metadata.total_compressed_size, 2, metadata,
+ ParquetPageReadContext(true, cache_key),
&offset_index);
+ ASSERT_TRUE(reader.parse_page_header().ok());
+ EXPECT_EQ(reader._page_statistics.page_cache_hit_counter, 1);
+ EXPECT_EQ(reader._offset_index, &offset_index);
+ reader.skip_page_data();
+ ASSERT_TRUE(reader.next_page().ok());
+ ASSERT_TRUE(reader.parse_page_header().ok());
+ EXPECT_EQ(reader._offset_index, &offset_index);
+}
+
TEST(ParquetV2NativeDecoderTest, ShiftedOffsetIndexFallsBackToSequentialPages)
{
auto data_page = [](int32_t value) {
tparquet::PageHeader header;
diff --git a/be/test/format_v2/parquet/parquet_encryption_test.cpp
b/be/test/format_v2/parquet/parquet_encryption_test.cpp
index 1fef0675b9a..52b626189ee 100644
--- a/be/test/format_v2/parquet/parquet_encryption_test.cpp
+++ b/be/test/format_v2/parquet/parquet_encryption_test.cpp
@@ -29,13 +29,22 @@
#include <cstring>
#include <filesystem>
#include <fstream>
+#include <limits>
#include <memory>
+#include <set>
#include <string>
+#include <unordered_set>
#include <utility>
#include <vector>
+#include "common/config.h"
#include "core/block/block.h"
#include "core/data_type/data_type_number.h"
+#include "exprs/expr_zonemap_filter.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
#include "format_v2/parquet/parquet_column_schema.h"
#include "format_v2/parquet/parquet_crypto.h"
#include "format_v2/parquet/parquet_file_context.h"
@@ -50,6 +59,7 @@
#include "runtime/exec_env.h"
#include "runtime/runtime_profile.h"
#include "runtime/runtime_state.h"
+#include "storage/index/zone_map/zonemap_eval_context.h"
#include "testutil/scoped_temp_dir.h"
#include "util/coding.h"
#include "util/defer_op.h"
@@ -65,7 +75,7 @@ const std::string AAD = "synthetic-file-aad";
std::vector<uint8_t> encrypted_file(
bool dictionary = true,
::parquet::ParquetDataPageVersion page_version =
::parquet::ParquetDataPageVersion::V1,
- const std::string& key = KEY, bool plaintext_footer = false) {
+ const std::string& key = KEY, bool plaintext_footer = false, int32_t
rows_per_group = 4) {
auto sink = ::arrow::io::BufferOutputStream::Create().ValueOrDie();
auto schema = ::parquet::schema::GroupNode::Make(
"schema", ::parquet::Repetition::REQUIRED,
@@ -92,11 +102,14 @@ std::vector<uint8_t> encrypted_file(
auto writer = ::parquet::ParquetFileWriter::Open(
sink,
std::static_pointer_cast<::parquet::schema::GroupNode>(schema),
properties.build());
- for (int32_t first : {1, 5}) {
+ for (int32_t first : {1, 1 + rows_per_group}) {
auto* group = writer->AppendRowGroup();
auto* column =
static_cast<::parquet::Int32Writer*>(group->NextColumn());
- const int32_t values[] = {first, first + 1, first + 2, first + 3};
- column->WriteBatch(4, nullptr, nullptr, values);
+ std::vector<int32_t> values(rows_per_group);
+ for (int32_t row = 0; row < rows_per_group; ++row) {
+ values[row] = first + row;
+ }
+ column->WriteBatch(rows_per_group, nullptr, nullptr, values.data());
group->Close();
}
writer->Close();
@@ -118,6 +131,7 @@ public:
bool closed() const override { return _closed; }
int64_t mtime() const override { return 1; }
[[maybe_unused]] size_t read_count() const { return _read_count; }
+ [[maybe_unused]] size_t max_read_size() const { return _max_read_size; }
protected:
Status read_at_impl(size_t offset, Slice result, size_t* bytes_read,
@@ -126,6 +140,7 @@ protected:
return Status::IOError("Synthetic encryption fixture read exceeds
file");
}
++_read_count;
+ _max_read_size = std::max(_max_read_size, result.size);
memcpy(result.data, _bytes.data() + offset, result.size);
*bytes_read = result.size;
return Status::OK();
@@ -136,6 +151,7 @@ private:
io::Path _path {"synthetic-encrypted.parquet"};
bool _closed = false;
size_t _read_count = 0;
+ size_t _max_read_size = 0;
};
std::vector<uint8_t> java_fixture(const std::string& name =
"iceberg_java.parquet") {
@@ -335,6 +351,111 @@ TEST(ParquetEncryptionTest,
NativeContainerBudgetDoesNotTrustWireElementTypes) {
}
}
+TEST(ParquetEncryptionTest, IndexContainerBudgetDoesNotTrustWireElementTypes) {
+ using namespace apache::thrift::protocol;
+ constexpr uint32_t values_per_container = 4096;
+ const size_t container_count =
+ MAX_ENCRYPTED_PARQUET_MODULE_BYTES /
+ (values_per_container * std::max(sizeof(std::string),
sizeof(int64_t))) +
+ 1;
+ auto transport =
std::make_shared<apache::thrift::transport::TMemoryBuffer>();
+ TCompactProtocol writer(transport);
+ writer.writeStructBegin("ColumnIndex");
+ writer.writeFieldBegin("null_pages", T_LIST, 1);
+ writer.writeListBegin(T_BOOL, 0);
+ writer.writeListEnd();
+ writer.writeFieldEnd();
+ writer.writeFieldBegin("max_values", T_LIST, 3);
+ writer.writeListBegin(T_STRING, 0);
+ writer.writeListEnd();
+ writer.writeFieldEnd();
+ writer.writeFieldBegin("boundary_order", T_I32, 4);
+ writer.writeI32(tparquet::BoundaryOrder::UNORDERED);
+ writer.writeFieldEnd();
+ // Generated code still resizes vector<string> for a BYTE-advertised
min_values field.
+ // Repeated fields exceed the cumulative native budget but fit a
wire-type-based budget.
+ for (size_t field = 0; field < container_count; ++field) {
+ writer.writeFieldBegin("min_values", T_LIST, 2);
+ writer.writeListBegin(T_BYTE, values_per_container);
+ for (uint32_t value = 0; value < values_per_container; ++value) {
+ writer.writeByte(0);
+ }
+ writer.writeListEnd();
+ writer.writeFieldEnd();
+ }
+ writer.writeFieldStop();
+ writer.writeStructEnd();
+ uint8_t* bytes = nullptr;
+ uint32_t size = 0;
+ transport->getBuffer(&bytes, &size);
+ ASSERT_LT(size, 4 << 20);
+ tparquet::ColumnIndex parsed;
+ uint32_t consumed = 0;
+ const auto status = deserialize_crypto_metadata(std::span(bytes, size),
&consumed, &parsed);
+ EXPECT_FALSE(status.ok());
+ EXPECT_EQ(consumed, 0);
+ // Generated code clears the vector before reading the next list header;
rejection precedes resize.
+ EXPECT_TRUE(parsed.min_values.empty());
+}
+
+TEST(ParquetEncryptionTest, UnknownIndexFieldsDoNotPoisonHistogramAccounting) {
+ using namespace apache::thrift::protocol;
+ constexpr uint32_t pages = 30000;
+ constexpr uint32_t bins_per_page = 101;
+ auto transport =
std::make_shared<apache::thrift::transport::TMemoryBuffer>();
+ TCompactProtocol writer(transport);
+ writer.writeStructBegin("ColumnIndex");
+ // Forward-compatible unknown metadata enters Thrift's nonvirtual template
skip path.
+ writer.writeFieldBegin("future", T_STRUCT, 100);
+ writer.writeStructBegin("FutureMetadata");
+ writer.writeFieldBegin("nested", T_LIST, 7);
+ writer.writeListBegin(T_BYTE, 1);
+ writer.writeByte(0);
+ writer.writeListEnd();
+ writer.writeFieldEnd();
+ writer.writeFieldStop();
+ writer.writeStructEnd();
+ writer.writeFieldEnd();
+ writer.writeFieldBegin("null_pages", T_LIST, 1);
+ writer.writeListBegin(T_BOOL, pages);
+ for (uint32_t page = 0; page < pages; ++page) {
+ writer.writeBool(false);
+ }
+ writer.writeListEnd();
+ writer.writeFieldEnd();
+ for (int16_t field : {2, 3}) {
+ writer.writeFieldBegin("bounds", T_LIST, field);
+ writer.writeListBegin(T_STRING, pages);
+ for (uint32_t page = 0; page < pages; ++page) {
+ writer.writeBinary(std::string(4, '\0'));
+ }
+ writer.writeListEnd();
+ writer.writeFieldEnd();
+ }
+ writer.writeFieldBegin("boundary_order", T_I32, 4);
+ writer.writeI32(tparquet::BoundaryOrder::UNORDERED);
+ writer.writeFieldEnd();
+ writer.writeFieldBegin("definition_level_histograms", T_LIST, 7);
+ writer.writeListBegin(T_I64, pages * bins_per_page);
+ for (uint32_t bin = 0; bin < pages * bins_per_page; ++bin) {
+ writer.writeI64(bin % bins_per_page == bins_per_page - 1 ? 1 : 0);
+ }
+ writer.writeListEnd();
+ writer.writeFieldEnd();
+ writer.writeFieldStop();
+ writer.writeStructEnd();
+ uint8_t* bytes = nullptr;
+ uint32_t size = 0;
+ transport->getBuffer(&bytes, &size);
+ tparquet::ColumnIndex parsed;
+ uint32_t consumed = 0;
+ const auto status = deserialize_crypto_metadata(std::span(bytes, size),
&consumed, &parsed);
+ ASSERT_TRUE(status.ok()) << status;
+ EXPECT_EQ(consumed, size);
+ EXPECT_EQ(parsed.min_values.size(), pages);
+ EXPECT_EQ(parsed.definition_level_histograms.size(), pages *
bins_per_page);
+}
+
tparquet::FileCryptoMetaData crypto_metadata() {
tparquet::AesGcmV1 aes;
aes.__set_aad_file_unique("12345678");
@@ -877,26 +998,562 @@ TEST(ParquetEncryptionTest,
SharedMetadataCacheCannotAuthorizeAnotherKey) {
EXPECT_TRUE(native_rows(bytes, parameters(), &rows).ok());
}
-TEST(ParquetEncryptionTest, EncryptedIndexBytesAreNotReadBySequentialStage) {
- auto file = std::make_shared<EncryptionMemoryReader>(encrypted_file());
+class EncryptedIndexPredicate final : public VExpr {
+public:
+ EncryptedIndexPredicate(int32_t value, bool equality)
+ : VExpr(std::make_shared<DataTypeUInt8>(), false), _value(value),
_equality(equality) {
+ _fn.name.function_name = equality ? "eq" : "gt";
+ set_node_type(TExprNodeType::BINARY_PRED);
+ auto type = std::make_shared<DataTypeInt32>();
+ add_child(VSlotRef::create_shared(0, 0, -1, type, "id"));
+ add_child(VLiteral::create_shared(type,
Field::create_field<TYPE_INT>(value)));
+ }
+ const std::string& expr_name() const override { return _name; }
+ bool can_evaluate_zonemap_filter() const override { return true; }
+ bool can_evaluate_bloom_filter() const override { return _equality; }
+ void collect_slot_column_ids(std::set<int>& ids) const override {
ids.insert(0); }
+ ZoneMapFilterResult evaluate_zonemap_filter(const ZoneMapEvalContext& ctx)
const override {
+ const auto map = ctx.zone_map(0);
+ if (map == nullptr) {
+ return unsupported_zonemap_filter(ctx);
+ }
+ const auto value = Field::create_field<TYPE_INT>(_value);
+ const bool excluded = !map->has_not_null || map->max_value < value ||
+ (!_equality && map->max_value == value) ||
+ (_equality && map->min_value > value);
+ return excluded ? ZoneMapFilterResult::kNoMatch :
ZoneMapFilterResult::kMayMatch;
+ }
+ ZoneMapFilterResult evaluate_bloom_filter(const BloomFilterEvalContext&
ctx) const override {
+ return expr_zonemap::eval_eq_bloom_filter(ctx,
+ {.slot_index = 0,
+ .slot_type =
get_child(0)->data_type(),
+ .literal =
Field::create_field<TYPE_INT>(_value),
+ .literal_type =
get_child(0)->data_type(),
+ .literal_on_left = false});
+ }
+ Status execute_column_impl(VExprContext*, const Block* block, const
Selector* selector,
+ size_t count, ColumnPtr& result_column) const
override {
+ const auto& input = *block->get_by_position(0).column;
+ auto result = ColumnUInt8::create(count);
+ for (size_t row = 0; row < count; ++row) {
+ const auto value = input[selector == nullptr ? row :
(*selector)[row]].get<TYPE_INT>();
+ result->get_data()[row] = _equality ? value == _value : value >
_value;
+ }
+ result_column = std::move(result);
+ return Status::OK();
+ }
+
+private:
+ int32_t _value;
+ bool _equality;
+ const std::string _name = "EncryptedIndexPredicate";
+};
+
+// GTest assertions expand into branches; keep the scan and pruning/reference
counters together.
+// NOLINTNEXTLINE(readability-function-cognitive-complexity)
+void verify_java_index_scan(const std::filesystem::path& path, bool encrypted,
bool pruning,
+ int32_t value, bool equality, const
std::vector<int32_t>& expected) {
+ auto properties = std::make_shared<io::FileSystemProperties>();
+ properties->system_type = TFileType::FILE_LOCAL;
+ auto description = std::make_unique<io::FileDescription>();
+ description->path = path;
+ description->file_size = std::filesystem::file_size(path);
+ TQueryOptions options;
+ const bool previous_page_index = config::enable_parquet_page_index;
+ config::enable_parquet_page_index = pruning;
+ Defer restore_page_index {[&] { config::enable_parquet_page_index =
previous_page_index; }};
+ options.__set_enable_parquet_filter_by_bloom_filter(pruning);
+ RuntimeState state {options, TQueryGlobals()};
+ RuntimeProfile profile("encrypted_index_scan");
+ ParquetReader reader(
+ properties, description, nullptr, &profile, std::nullopt, false,
false, std::nullopt,
+ false, nullptr,
+ encrypted ? std::optional(parameters(KEY, "synthetic-index-aad"))
: std::nullopt);
+ ASSERT_TRUE(reader.init(&state).ok());
+ auto request = std::make_shared<FileScanRequest>();
+ request->predicate_columns =
{LocalColumnIndex::top_level(LocalColumnId(0))};
+ request->local_positions.emplace(LocalColumnId(0), LocalIndex(0));
+ auto context =
+
VExprContext::create_shared(std::make_shared<EncryptedIndexPredicate>(value,
equality));
+ context->_prepared = true;
+ context->_opened = true;
+ request->conjuncts = {context};
+ // The native reader does not use the legacy min/max option. Fence
metadata pruning while
+ // retaining the same residual predicate so the reference really decodes
every input row.
+ request->metadata_pruning_safe_conjunct_count = pruning ? 1 : 0;
+ ASSERT_TRUE(reader.open(request).ok());
+ std::vector<int32_t> actual;
+ bool eof = false;
+ while (!eof) {
+ Block block;
+ block.insert({std::make_shared<DataTypeInt32>()->create_column(),
+ std::make_shared<DataTypeInt32>(), "id"});
+ size_t read = 0;
+ auto status = reader.get_block(&block, &read, &eof);
+ ASSERT_TRUE(status.ok()) << status;
+ for (size_t row = 0; row < read; ++row) {
+
actual.push_back((*block.get_by_position(0).column)[row].get<TYPE_INT>());
+ }
+ }
+ EXPECT_EQ(actual, expected);
+ ASSERT_TRUE(reader.close().ok());
+ if (!pruning) {
+ EXPECT_EQ(reader._parquet_profile.filtered_row_groups->value(), 0);
+ EXPECT_EQ(reader._parquet_profile.filtered_page_rows->value(), 0);
+ EXPECT_EQ(reader._parquet_profile.raw_rows_read->value(), 512);
+ }
+ if (pruning && equality && expected.empty()) {
+
EXPECT_GT(reader._parquet_profile.filtered_row_groups_by_bloom_filter->value(),
0);
+
EXPECT_GT(reader._parquet_profile.bloom_filter_probe_successes->value(), 0);
+ }
+ if (pruning && !equality) {
+ EXPECT_GT(reader._parquet_profile.filtered_page_rows->value(), 0);
+ }
+}
+
+TEST(ParquetEncryptionTest,
JavaIndexesAndBloomPruningMatchUnprunedAndPlaintextScans) {
+ test::ScopedTempDirectory directory("encrypted-index-reference");
+ std::vector<int32_t> tail;
+ for (int32_t value = 902; value < 1024; value += 2) {
+ tail.push_back(value);
+ }
+ for (bool encrypted : {false, true}) {
+ const auto bytes =
+ java_fixture(encrypted ? "indexes_java.parquet" :
"indexes_java_plain.parquet");
+ const auto path = directory.path() / (encrypted ? "encrypted.parquet"
: "plain.parquet");
+ {
+ std::ofstream out(path, std::ios::binary);
+ out.write(reinterpret_cast<const char*>(bytes.data()),
bytes.size());
+ }
+ for (bool pruning : {false, true}) {
+ verify_java_index_scan(path, encrypted, pruning, 900, false, tail);
+ verify_java_index_scan(path, encrypted, pruning, 3, true, {});
+ verify_java_index_scan(path, encrypted, pruning, 900, true, {900});
+ }
+ }
+}
+
+TEST(ParquetEncryptionTest,
MissingEncryptedBloomFilterRetainsCandidateRowGroups) {
+ auto info = parameters();
+ ParquetFileContext context;
+ io::FileDescription description;
+
ASSERT_TRUE(context.open(std::make_shared<EncryptionMemoryReader>(encrypted_file()),
nullptr,
+ false, description, false, false, false, &info)
+ .ok());
+ std::vector<std::unique_ptr<ParquetColumnSchema>> schema;
+ ASSERT_TRUE(build_parquet_column_schema(context.native_metadata->schema(),
&schema).ok());
+ FileScanRequest request;
+ request.local_positions.emplace(LocalColumnId(0), LocalIndex(0));
+ request.predicate_columns =
{LocalColumnIndex::top_level(LocalColumnId(0))};
+ request.conjuncts = {
+
VExprContext::create_shared(std::make_shared<EncryptedIndexPredicate>(3,
true))};
+ std::vector<int> selected;
+ ParquetPruningStats stats;
+ auto status = select_row_groups_by_metadata(
+ context.native_metadata->to_thrift(), schema, request, nullptr,
&selected, true, &stats,
+ nullptr, nullptr, &context, {},
ParquetMetadataProbeMode::EXPENSIVE_ONLY);
+ ASSERT_TRUE(status.ok()) << status;
+ EXPECT_EQ(selected, (std::vector<int> {0, 1}));
+ EXPECT_EQ(stats.bloom_filter_conservative_fallbacks, 2);
+ ASSERT_TRUE(context.close().ok());
+}
+
+void verify_index_module_binding(
+ const std::vector<uint8_t>& bytes, const tparquet::ColumnChunk& chunk,
+ const std::shared_ptr<ParquetColumnCrypto>& correct,
+ const std::vector<std::shared_ptr<ParquetColumnCrypto>>& mismatched) {
+ for (auto module : {ParquetMetadataModule::COLUMN_INDEX,
ParquetMetadataModule::OFFSET_INDEX}) {
+ const bool column = module == ParquetMetadataModule::COLUMN_INDEX;
+ const auto offset = column ? chunk.column_index_offset :
chunk.offset_index_offset;
+ const auto length = column ? chunk.column_index_length :
chunk.offset_index_length;
+ const auto ciphertext = std::span(bytes).subspan(offset, length);
+ DorisVector<uint8_t> plaintext;
+ ASSERT_TRUE(correct->decrypt_metadata(module, ciphertext,
&plaintext).ok());
+ for (const auto& crypto : mismatched) {
+ EXPECT_FALSE(crypto->decrypt_metadata(module, ciphertext,
&plaintext).ok());
+ }
+ const auto swapped =
+ column ? ParquetMetadataModule::OFFSET_INDEX :
ParquetMetadataModule::COLUMN_INDEX;
+ EXPECT_FALSE(correct->decrypt_metadata(swapped, ciphertext,
&plaintext).ok());
+ }
+}
+
+TEST(ParquetEncryptionTest,
IndexAuthenticationBindsModuleRowGroupColumnAndFile) {
+ auto bytes = encrypted_file();
auto info = parameters();
ParquetFileContext context;
+ ParquetFileContext other_file;
io::FileDescription description;
- ASSERT_TRUE(context.open(file, nullptr, true, description, false, false,
false, &info).ok());
+ ASSERT_TRUE(context.open(std::make_shared<EncryptionMemoryReader>(bytes),
nullptr, false,
+ description, false, false, false, &info)
+ .ok());
+ ASSERT_TRUE(other_file
+
.open(std::make_shared<EncryptionMemoryReader>(encrypted_file()), nullptr,
+ false, description, false, false, false, &info)
+ .ok());
const auto& chunk =
context.native_metadata->to_thrift().row_groups[0].columns[0];
- ASSERT_TRUE(chunk.__isset.column_index_offset);
- ASSERT_TRUE(chunk.__isset.offset_index_offset);
- auto read_count = file->read_count();
- std::unordered_map<int, tparquet::OffsetIndex> offsets;
+ std::shared_ptr<ParquetColumnCrypto> correct;
+ std::shared_ptr<ParquetColumnCrypto> wrong_group;
+ std::shared_ptr<ParquetColumnCrypto> wrong_column;
+ std::shared_ptr<ParquetColumnCrypto> wrong_file;
+ ASSERT_TRUE(context.native_metadata->encryption()->column_crypto(chunk, 0,
0, &correct).ok());
+ ASSERT_TRUE(
+ context.native_metadata->encryption()->column_crypto(chunk, 1, 0,
&wrong_group).ok());
+ ASSERT_TRUE(
+ context.native_metadata->encryption()->column_crypto(chunk, 0, 1,
&wrong_column).ok());
+ ASSERT_TRUE(
+ other_file.native_metadata->encryption()->column_crypto(chunk, 0,
0, &wrong_file).ok());
+ verify_index_module_binding(bytes, chunk, correct, {wrong_group,
wrong_column, wrong_file});
+ ASSERT_TRUE(context.close().ok());
+ ASSERT_TRUE(other_file.close().ok());
+}
+
+TEST(ParquetEncryptionTest, EncryptedIndexRangesRejectInvalidOffsetsBeforeIo) {
+ auto bytes = encrypted_file();
+ auto info = parameters();
+ for (bool offset_only : {false, true}) {
+ auto file = std::make_shared<EncryptionMemoryReader>(bytes);
+ ParquetFileContext context;
+ io::FileDescription description;
+ ASSERT_TRUE(
+ context.open(file, nullptr, false, description, false, false,
false, &info).ok());
+ auto metadata = context.native_metadata->to_thrift();
+ metadata.row_groups[0].columns[0].__set_offset_index_offset(-1);
+ auto crypto = context.native_metadata->encryption();
+ context.native_metadata_owner =
+ std::make_unique<NativeParquetMetadata>(std::move(metadata),
0, crypto);
+ context.native_metadata = context.native_metadata_owner.get();
+ const auto reads = file->read_count();
+ std::unordered_map<int, NativeParquetPageIndex> pages;
+ std::unordered_map<int, tparquet::OffsetIndex> offsets;
+ const auto status = offset_only ?
context.load_native_offset_indexes(0, {0}, &offsets)
+ : context.load_native_page_indexes(0,
{0}, &pages);
+ EXPECT_FALSE(status.ok());
+ EXPECT_EQ(file->read_count(), reads);
+ ASSERT_TRUE(context.close().ok());
+ }
+}
+
+TEST(ParquetEncryptionTest,
EncryptedBloomAuthenticationAndLengthFailuresPropagate) {
+ const auto original = java_fixture("indexes_java.parquet");
+ auto info = parameters(KEY, "synthetic-index-aad");
+ ParquetFileContext base;
+ io::FileDescription description;
+ ASSERT_TRUE(base.open(std::make_shared<EncryptionMemoryReader>(original),
nullptr, false,
+ description, false, false, false, &info)
+ .ok());
+ const auto& metadata =
base.native_metadata->to_thrift().row_groups[0].columns[0].meta_data;
+ ASSERT_TRUE(metadata.__isset.bloom_filter_offset);
+ const auto header_offset =
static_cast<size_t>(metadata.bloom_filter_offset);
+ const auto header_length =
+ static_cast<size_t>(decode_fixed32_le(original.data() +
header_offset)) + 4;
+ for (int failure : {0, 1, 2}) {
+ auto bytes = original;
+ if (failure == 0) {
+ bytes.at(header_offset + header_length - 1) ^= 1;
+ } else if (failure == 1) {
+ const auto offset = header_offset + header_length;
+ const auto length =
static_cast<size_t>(decode_fixed32_le(bytes.data() + offset)) + 4;
+ bytes.at(offset + length - 1) ^= 1;
+ } else {
+ encode_fixed32_le(bytes.data() + header_offset,
std::numeric_limits<uint32_t>::max());
+ }
+ ParquetFileContext context;
+
ASSERT_TRUE(context.open(std::make_shared<EncryptionMemoryReader>(bytes),
nullptr, false,
+ description, false, false, false, &info)
+ .ok());
+ std::vector<std::unique_ptr<ParquetColumnSchema>> schema;
+
ASSERT_TRUE(build_parquet_column_schema(context.native_metadata->schema(),
&schema).ok());
+ FileScanRequest request;
+ request.local_positions.emplace(LocalColumnId(0), LocalIndex(0));
+ request.predicate_columns =
{LocalColumnIndex::top_level(LocalColumnId(0))};
+ request.conjuncts = {
+
VExprContext::create_shared(std::make_shared<EncryptedIndexPredicate>(3,
true))};
+ const std::vector<int> candidate {0};
+ std::vector<int> selected;
+ ParquetPruningStats stats;
+ const auto status =
select_row_groups_by_metadata(context.native_metadata->to_thrift(),
+ schema, request,
&candidate, &selected,
+ true, &stats,
nullptr, nullptr, &context);
+ EXPECT_FALSE(status.ok());
+ EXPECT_NE(
+ status.to_string().find(failure == 2 ? "INVALID_MODULE_SIZE" :
"DECRYPTION_FAILED"),
+ std::string::npos);
+ ASSERT_TRUE(context.close().ok());
+ }
+ ASSERT_TRUE(base.close().ok());
+}
+
+TEST(ParquetEncryptionTest, EncryptedIndexAuthenticationFailureIsNotOptional) {
+ auto original = encrypted_file();
+ auto info = parameters();
+ ParquetFileContext original_context;
+ io::FileDescription description;
+ ASSERT_TRUE(original_context
+
.open(std::make_shared<EncryptionMemoryReader>(original), nullptr, false,
+ description, false, false, false, &info)
+ .ok());
+ const auto chunk =
original_context.native_metadata->to_thrift().row_groups[0].columns[0];
+ for (bool column : {false, true}) {
+ auto bytes = original;
+ const auto offset = column ? chunk.column_index_offset :
chunk.offset_index_offset;
+ const auto length = column ? chunk.column_index_length :
chunk.offset_index_length;
+ bytes.at(offset + length - 1) ^= 1;
+ ParquetFileContext context;
+
ASSERT_TRUE(context.open(std::make_shared<EncryptionMemoryReader>(bytes),
nullptr, false,
+ description, false, false, false, &info)
+ .ok());
+ std::unordered_map<int, NativeParquetPageIndex> pages;
+ auto status = context.load_native_page_indexes(0, {0}, &pages);
+ EXPECT_FALSE(status.ok());
+ EXPECT_NE(status.to_string().find("DECRYPTION_FAILED"),
std::string::npos);
+ ASSERT_TRUE(context.close().ok());
+ }
+ ASSERT_TRUE(original_context.close().ok());
+}
+
+std::vector<uint8_t> wide_index_file(bool encrypted) {
+ auto sink = ::arrow::io::BufferOutputStream::Create().ValueOrDie();
+ ::parquet::schema::NodeVector fields;
+ for (int column = 0; column < 65; ++column) {
+ fields.push_back(::parquet::schema::PrimitiveNode::Make("value_" +
std::to_string(column),
+
::parquet::Repetition::REQUIRED,
+
::parquet::Type::BYTE_ARRAY));
+ }
+ auto schema =
+ ::parquet::schema::GroupNode::Make("schema",
::parquet::Repetition::REQUIRED, fields);
+ ::parquet::WriterProperties::Builder properties;
+ properties.compression(::parquet::Compression::SNAPPY)
+ ->disable_dictionary()
+ ->enable_write_page_index()
+ ->data_pagesize(256)
+ ->write_batch_size(1);
+ if (encrypted) {
+ ::parquet::FileEncryptionProperties::Builder encryption {
+ ::arrow::util::SecureString(std::string(KEY))};
+ encryption.aad_prefix(AAD)->disable_aad_prefix_storage();
+ properties.encryption(encryption.build());
+ }
+ auto writer = ::parquet::ParquetFileWriter::Open(
+ sink,
std::static_pointer_cast<::parquet::schema::GroupNode>(schema),
+ properties.build());
+ auto* group = writer->AppendRowGroup();
+ for (int column = 0; column < 65; ++column) {
+ auto* values =
static_cast<::parquet::ByteArrayWriter*>(group->NextColumn());
+ for (int row = 0; row < 1024; ++row) {
+ std::string value(512, 'x');
+ value.replace(0, 4, std::to_string(1000 + row));
+ const ::parquet::ByteArray bytes(value.size(),
+ reinterpret_cast<const
uint8_t*>(value.data()));
+ values->WriteBatch(1, nullptr, nullptr, &bytes);
+ }
+ }
+ group->Close();
+ writer->Close();
+ auto bytes = sink->Finish().ValueOrDie();
+ return {bytes->data(), bytes->data() + bytes->size()};
+}
+
+// GTest assertions expand into branches; keep the complete bounded-I/O
scenario together.
+// NOLINTNEXTLINE(readability-function-cognitive-complexity)
+void verify_large_index_block(bool encrypted) {
+ auto file =
std::make_shared<EncryptionMemoryReader>(wide_index_file(encrypted));
+ auto info = parameters();
+ ParquetFileContext context;
+ io::FileDescription description;
+ ASSERT_TRUE(context.open(file, nullptr, false, description, false, false,
false,
+ encrypted ? &info : nullptr)
+ .ok());
+ const auto& chunks =
context.native_metadata->to_thrift().row_groups[0].columns;
+ std::unordered_set<int> leaves;
+ int64_t span_end = chunks.front().column_index_offset;
+ for (int column = 0; column < 65; ++column) {
+ ASSERT_EQ(chunks[column].column_index_offset, span_end);
+ ASSERT_LT(chunks[column].column_index_length,
detail::MAX_SERIALIZED_PARQUET_INDEX_BYTES);
+ span_end += chunks[column].column_index_length;
+ leaves.insert(column);
+ }
+ // Each module fits the budget; only the writer's contiguous block exceeds
it.
+ ASSERT_GT(span_end - chunks.front().column_index_offset,
+ detail::MAX_SERIALIZED_PARQUET_INDEX_BYTES);
+ const auto reads = file->read_count();
std::unordered_map<int, NativeParquetPageIndex> pages;
- ASSERT_TRUE(context.load_native_offset_indexes(0, {0}, &offsets).ok());
- ASSERT_TRUE(context.load_native_page_indexes(0, {0}, &pages).ok());
- EXPECT_TRUE(offsets.empty());
- EXPECT_TRUE(pages.empty());
- EXPECT_EQ(file->read_count(), read_count);
+ const auto status = context.load_native_page_indexes(0, leaves, &pages);
+ ASSERT_TRUE(status.ok()) << status;
+ ASSERT_EQ(pages.size(), 65);
+ for (const auto& [column, indexes] : pages) {
+ EXPECT_EQ(indexes.column_index.null_pages.size(), 1024);
+ EXPECT_EQ(indexes.offset_index.page_locations.size(), 1024);
+ }
+ // Two bounded ColumnIndex reads and one coalesced OffsetIndex read retain
all leaves.
+ EXPECT_EQ(file->read_count() - reads, 3);
+ EXPECT_LE(file->max_read_size(),
detail::MAX_SERIALIZED_PARQUET_INDEX_BYTES);
+ if (encrypted) {
+ // A tag failure in the last module must still fail after the earlier
I/O block succeeded.
+ file->_bytes.at(span_end - 1) ^= 1;
+ const auto corrupt = context.load_native_page_indexes(0, leaves,
&pages);
+ EXPECT_FALSE(corrupt.ok());
+ EXPECT_NE(corrupt.to_string().find("DECRYPTION_FAILED"),
std::string::npos);
+ EXPECT_TRUE(pages.empty());
+ }
+ ASSERT_TRUE(context.close().ok());
+}
+
+TEST(ParquetEncryptionTest, LargePlaintextIndexBlocksStayBounded) {
+ verify_large_index_block(false);
+}
+
+TEST(ParquetEncryptionTest, LargeEncryptedIndexBlocksStayBounded) {
+ verify_large_index_block(true);
+}
+
+TEST(ParquetEncryptionTest, ManyPageEncryptedIndexesFitNativeContainerBudget) {
+ constexpr int32_t pages = 30000;
+ auto file = std::make_shared<EncryptionMemoryReader>(
+ encrypted_file(false, ::parquet::ParquetDataPageVersion::V1, KEY,
false, pages));
+ auto info = parameters();
+ ParquetFileContext context;
+ io::FileDescription description;
+ ASSERT_TRUE(context.open(file, nullptr, false, description, false, false,
false, &info).ok());
+ for (int group = 0; group < 2; ++group) {
+ std::unordered_map<int, NativeParquetPageIndex> indexes;
+ const auto status = context.load_native_page_indexes(group, {0},
&indexes);
+ ASSERT_TRUE(status.ok()) << status;
+ ASSERT_EQ(indexes.size(), 1);
+ EXPECT_EQ(indexes.at(0).column_index.min_values.size(), pages);
+ EXPECT_EQ(indexes.at(0).column_index.max_values.size(), pages);
+ ASSERT_EQ(indexes.at(0).offset_index.page_locations.size(), pages);
+
EXPECT_EQ(indexes.at(0).offset_index.page_locations.back().first_row_index,
pages - 1);
+ }
ASSERT_TRUE(context.close().ok());
}
+std::vector<uint8_t> encrypted_histogram_file(int16_t max_definition_level,
int32_t pages) {
+ auto sink = ::arrow::io::BufferOutputStream::Create().ValueOrDie();
+ auto field = ::parquet::schema::PrimitiveNode::Make("value",
::parquet::Repetition::OPTIONAL,
+
::parquet::Type::INT32);
+ for (int level = 1; level < max_definition_level; ++level) {
+ field = ::parquet::schema::GroupNode::Make("level_" +
std::to_string(level),
+
::parquet::Repetition::OPTIONAL, {field});
+ }
+ auto schema =
+ ::parquet::schema::GroupNode::Make("schema",
::parquet::Repetition::REQUIRED, {field});
+ ::parquet::FileEncryptionProperties::Builder encryption {
+ ::arrow::util::SecureString(std::string(KEY))};
+ encryption.aad_prefix(AAD)->disable_aad_prefix_storage();
+ ::parquet::WriterProperties::Builder properties;
+ properties.encryption(encryption.build())
+ ->compression(::parquet::Compression::SNAPPY)
+ ->disable_dictionary()
+ ->enable_write_page_index()
+
->set_size_statistics_level(::parquet::SizeStatisticsLevel::PageAndColumnChunk)
+ ->max_rows_per_page(1)
+ ->data_pagesize(1)
+ ->write_batch_size(1);
+ auto writer = ::parquet::ParquetFileWriter::Open(
+ sink,
std::static_pointer_cast<::parquet::schema::GroupNode>(schema),
+ properties.build());
+ auto* group = writer->AppendRowGroup();
+ auto* column = static_cast<::parquet::Int32Writer*>(group->NextColumn());
+ for (int32_t row = 0; row < pages; ++row) {
+ column->WriteBatch(1, &max_definition_level, nullptr, &row);
+ }
+ group->Close();
+ writer->Close();
+ auto bytes = sink->Finish().ValueOrDie();
+ return {bytes->data(), bytes->data() + bytes->size()};
+}
+
+void verify_encrypted_histogram(int16_t max_definition_level, int32_t pages) {
+ auto file = std::make_shared<EncryptionMemoryReader>(
+ encrypted_histogram_file(max_definition_level, pages));
+ auto info = parameters();
+ ParquetFileContext context;
+ io::FileDescription description;
+ ASSERT_TRUE(context.open(file, nullptr, false, description, false, false,
false, &info).ok());
+ std::unordered_map<int, NativeParquetPageIndex> indexes;
+ const auto status = context.load_native_page_indexes(0, {0}, &indexes);
+ ASSERT_TRUE(status.ok()) << status;
+ ASSERT_EQ(indexes.size(), 1);
+ const auto& index = indexes.at(0);
+ EXPECT_EQ(index.column_index.min_values.size(), pages);
+ ASSERT_EQ(index.column_index.definition_level_histograms.size(),
+ pages * (max_definition_level + 1));
+
EXPECT_EQ(index.column_index.definition_level_histograms[max_definition_level],
1);
+ EXPECT_EQ(index.column_index.definition_level_histograms.back(), 1);
+ EXPECT_EQ(index.offset_index.page_locations.size(), pages);
+ ASSERT_TRUE(context.close().ok());
+}
+
+TEST(ParquetEncryptionTest,
NestedEncryptedHistogramsExceedFooterContainerLimit) {
+ verify_encrypted_histogram(2, 22000);
+}
+
+TEST(ParquetEncryptionTest, DeepEncryptedHistogramsUseNativeCounterBudget) {
+ verify_encrypted_histogram(100, 30000);
+}
+
+void verify_loaded_indexes(ParquetFileContext& context, int group,
+ std::unordered_map<int, tparquet::OffsetIndex>*
offsets) {
+ std::unordered_map<int, NativeParquetPageIndex> pages;
+ ASSERT_TRUE(context.load_native_offset_indexes(group, {0}, offsets).ok());
+ ASSERT_TRUE(context.load_native_page_indexes(group, {0}, &pages).ok());
+ ASSERT_EQ(offsets->size(), 1);
+ ASSERT_EQ(pages.size(), 1);
+ ASSERT_EQ(offsets->at(0).page_locations.size(), 4);
+ EXPECT_EQ(pages.at(0).column_index.null_pages.size(), 4);
+}
+
+void verify_indexed_row_group(ParquetFileContext& context,
+ const
std::vector<std::unique_ptr<ParquetColumnSchema>>& schema,
+ const std::shared_ptr<EncryptionMemoryReader>&
file, int group,
+ bool dictionary) {
+ std::unordered_map<int, tparquet::OffsetIndex> offsets;
+ ASSERT_NO_FATAL_FAILURE(verify_loaded_indexes(context, group, &offsets));
+ // Skip the first two physical pages: the first page we decrypt still has
AAD 2.
+ RuntimeProfile profile("indexed_headers");
+ ParquetColumnReaderProfile counters;
+ counters.parse_page_header_count = profile.add_counter("ParsedHeaders",
TUnit::UNIT);
+ std::unique_ptr<ParquetColumnReader> reader;
+ ASSERT_TRUE(NativeColumnReader::create(*schema[0], nullptr, file,
context.native_metadata,
+ group, {{2, 2}}, offsets, nullptr,
std::nullopt, nullptr,
+ nullptr, false, "", false,
counters, &reader)
+ .ok());
+ auto output = reader->type()->create_column();
+ int64_t read = 0;
+ ASSERT_TRUE(reader->skip(2).ok());
+ const auto status = reader->read(2, output, &read);
+ ASSERT_TRUE(status.ok()) << status;
+ ASSERT_EQ(read, 2);
+ ASSERT_EQ(output->size(), 2);
+ EXPECT_EQ((*output)[0].get<TYPE_INT>(), group * 4 + 3);
+ EXPECT_EQ((*output)[1].get<TYPE_INT>(), group * 4 + 4);
+ reader->flush_profile();
+ EXPECT_LT(counters.parse_page_header_count->value(), 4 +
static_cast<int>(dictionary));
+}
+
+TEST(ParquetEncryptionTest,
EncryptedIndexesLoadAndIndexedReadsKeepPhysicalPageOrdinals) {
+ for (auto version :
+ {::parquet::ParquetDataPageVersion::V1,
::parquet::ParquetDataPageVersion::V2}) {
+ for (bool dictionary : {false, true}) {
+ auto file =
+
std::make_shared<EncryptionMemoryReader>(encrypted_file(dictionary, version));
+ auto info = parameters();
+ ParquetFileContext context;
+ io::FileDescription description;
+ ASSERT_TRUE(context.open(file, nullptr, false, description, false,
false, false, &info)
+ .ok());
+ std::vector<std::unique_ptr<ParquetColumnSchema>> schema;
+ ASSERT_TRUE(
+
build_parquet_column_schema(context.native_metadata->schema(), &schema).ok());
+ for (int group : {0, 1}) {
+ verify_indexed_row_group(context, schema, file, group,
dictionary);
+ }
+ ASSERT_TRUE(context.close().ok());
+ }
+ }
+}
+
TEST(ParquetEncryptionTest, LevelsOnlyReaderDecryptsNullableNestedShape) {
auto file = std::make_shared<EncryptionMemoryReader>(java_fixture());
auto info = parameters(KEY, "0123456789abcdef");
diff --git
a/be/test/format_v2/parquet/test_data/encryption/GenerateEncryptedIndexes.java
b/be/test/format_v2/parquet/test_data/encryption/GenerateEncryptedIndexes.java
new file mode 100644
index 00000000000..1d12f76c971
--- /dev/null
+++
b/be/test/format_v2/parquet/test_data/encryption/GenerateEncryptedIndexes.java
@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import java.nio.charset.StandardCharsets;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.crypto.FileEncryptionProperties;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroupFactory;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.example.ExampleParquetWriter;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.MessageTypeParser;
+
+/** Generates independently written page-index and Bloom-filter
interoperability fixtures. */
+public final class GenerateEncryptedIndexes {
+ public static void main(String[] args) throws Exception {
+ MessageType schema = MessageTypeParser.parseMessageType(
+ "message indexed { required int32 id = 1; optional binary
label (UTF8) = 2; }");
+ Configuration configuration = new Configuration();
+ configuration.set("fs.file.impl",
"org.apache.hadoop.fs.RawLocalFileSystem");
+ for (int file = 0; file < args.length; file++) {
+ ExampleParquetWriter.Builder builder =
ExampleParquetWriter.builder(new Path(args[file]))
+ .withConf(configuration).withType(schema)
+
.withPageSize(128).withPageRowCountLimit(8).withRowGroupSize(2048)
+
.withMinRowCountForPageSizeCheck(1).withMaxRowCountForPageSizeCheck(8)
+ .withDictionaryEncoding(true).withDictionaryPageSize(128)
+ .withBloomFilterEnabled(true)
+ .withBloomFilterNDV("id", 512).withBloomFilterNDV("label",
512);
+ if (file == 0) {
+ builder.withEncryption(FileEncryptionProperties.builder(
+ "synthetic-key123".getBytes(StandardCharsets.UTF_8))
+
.withAADPrefix("synthetic-index-aad".getBytes(StandardCharsets.UTF_8))
+ .withoutAADPrefixStorage().build());
+ }
+ SimpleGroupFactory factory = new SimpleGroupFactory(schema);
+ try (ParquetWriter<Group> writer = builder.build()) {
+ for (int row = 0; row < 512; row++) {
+ Group value = factory.newGroup().append("id", row * 2);
+ if (row % 5 != 0) {
+ value.append("label", "value-" + row);
+ }
+ writer.write(value);
+ }
+ }
+ }
+ }
+}
diff --git a/be/test/format_v2/parquet/test_data/encryption/README.md
b/be/test/format_v2/parquet/test_data/encryption/README.md
index f26d91b54b5..7b702ba4e6f 100644
--- a/be/test/format_v2/parquet/test_data/encryption/README.md
+++ b/be/test/format_v2/parquet/test_data/encryption/README.md
@@ -63,6 +63,54 @@ Run through `run-be-ut.sh --run
--filter='ParquetEncryptionTest.*'` with a Doris
third-party build using `PARQUET_REQUIRE_ENCRYPTION=ON`. The internal PME
headers
installed by that build are required for decryption. Older prebuilt bundles run
the explicit dependency-unavailable contract test instead. Encrypted SQL
admission,
-shared decrypted caches, encrypted indexes and encrypted deletes remain
disabled
+shared decrypted caches and encrypted deletes remain disabled
at this stage; the physical reader's optional parameters support controlled
tests
and subsequent scan integration.
+
+## Encrypted indexes and Bloom filters
+
+`indexes_java.parquet` and `indexes_java_plain.parquet` were generated by
+`GenerateEncryptedIndexes.java` with Parquet Java 1.15.2, as packaged in Spark
+4.0.0. Both files contain 512 rows across six row groups. `id` contains the
even
+integers from 0 through 1022; `label` is null every fifth row. Both physical
+columns have page indexes and split-block Bloom filters. The encrypted twin
uses
+the same public test key and the AAD prefix `synthetic-index-aad`.
+
+Set `PARQUET_REFERENCE_CLASSPATH` to the Parquet Java 1.15.2 example/Hadoop
jars
+and their dependencies. Generate into paths that do not already exist:
+
+```sh
+javac -cp "$PARQUET_REFERENCE_CLASSPATH" -d build/encrypted-parquet-fixture \
+ be/test/format_v2/parquet/test_data/encryption/GenerateEncryptedIndexes.java
+java -cp "build/encrypted-parquet-fixture:$PARQUET_REFERENCE_CLASSPATH" \
+ GenerateEncryptedIndexes \
+ be/test/format_v2/parquet/test_data/encryption/indexes_java.parquet \
+ be/test/format_v2/parquet/test_data/encryption/indexes_java_plain.parquet
+cp be/test/format_v2/parquet/test_data/encryption/indexes_java*.parquet \
+ regression-test/data/external_table_p0/iceberg/encryption/
+```
+
+Native reader tests compare encrypted/plaintext and pruned/unpruned results.
+The reference request excludes all conjuncts from metadata pruning while
keeping
+them for row-level evaluation; counters verify all 512 physical rows are read
+without row-group/page pruning. Pruned scans assert effective page and Bloom
+pruning, and authenticate indexes against their module type, file, row group
and
+physical column. Additional C++ fixtures cover
+Page V1/V2, dictionary/plain pages and indexed seeks past earlier pages.
Modified
+index/Bloom authentication tags, oversized module prefixes and invalid index
+ranges must fail explicitly. Index parsing charges known list fields by their
+generated native element sizes, independently of advertised wire element types;
+unknown and skipped containers retain conservative accounting. Histogram list
+counts follow the decoded-byte budget rather than the footer's 65,536-element
+cap. Recursion and module-size limits remain shared with encrypted footers.
+Generated encrypted files cover 30,000-page indexes, 66,000 shallow-nesting
+histogram bins, and 3.03 million deeply nested histogram counters whose native
+storage still fits the budget. An unknown-field test covers accounting after
+Thrift's recursive skip path.
+
+SQL tests compare the plaintext twin with Bloom filtering enabled and disabled;
+native footer and page pruning remain active in both SQL scans. The fully
+unpruned comparison is provided by the physical-reader unit test above.
+Keyless encrypted scans, including negative Bloom queries, remain rejected by
+the existing admission contract. These SQL checks do not claim that
+encrypted Iceberg SQL admission has been enabled.
diff --git
a/be/test/format_v2/parquet/test_data/encryption/indexes_java.parquet
b/be/test/format_v2/parquet/test_data/encryption/indexes_java.parquet
new file mode 100644
index 00000000000..3d06bf109d5
Binary files /dev/null and
b/be/test/format_v2/parquet/test_data/encryption/indexes_java.parquet differ
diff --git
a/be/test/format_v2/parquet/test_data/encryption/indexes_java_plain.parquet
b/be/test/format_v2/parquet/test_data/encryption/indexes_java_plain.parquet
new file mode 100644
index 00000000000..a3d5a2d91a2
Binary files /dev/null and
b/be/test/format_v2/parquet/test_data/encryption/indexes_java_plain.parquet
differ
diff --git
a/regression-test/data/external_table_p0/iceberg/encryption/indexes_java.parquet
b/regression-test/data/external_table_p0/iceberg/encryption/indexes_java.parquet
new file mode 100644
index 00000000000..3d06bf109d5
Binary files /dev/null and
b/regression-test/data/external_table_p0/iceberg/encryption/indexes_java.parquet
differ
diff --git
a/regression-test/data/external_table_p0/iceberg/encryption/indexes_java_plain.parquet
b/regression-test/data/external_table_p0/iceberg/encryption/indexes_java_plain.parquet
new file mode 100644
index 00000000000..a3d5a2d91a2
Binary files /dev/null and
b/regression-test/data/external_table_p0/iceberg/encryption/indexes_java_plain.parquet
differ
diff --git
a/regression-test/data/external_table_p0/iceberg/test_iceberg_encryption_native_reader.out
b/regression-test/data/external_table_p0/iceberg/test_iceberg_encryption_native_reader.out
index 3e3909c369b..6d5f855fbc1 100644
---
a/regression-test/data/external_table_p0/iceberg/test_iceberg_encryption_native_reader.out
+++
b/regression-test/data/external_table_p0/iceberg/test_iceberg_encryption_native_reader.out
@@ -52,6 +52,26 @@
7 shared [] \N
8 shared [null, 8] 8
+-- !index_tail_false --
+61 902 1022 49
+
+-- !index_membership_false --
+900 \N
+1022 value-511
+
+-- !index_absent_false --
+0
+
+-- !index_tail_true --
+61 902 1022 49
+
+-- !index_membership_true --
+900 \N
+1022 value-511
+
+-- !index_absent_true --
+0
+
-- !plain_after_truncation --
8 6 36
diff --git
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_encryption_native_reader.groovy
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_encryption_native_reader.groovy
index 218ea46221b..f89c66f7e10 100644
---
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_encryption_native_reader.groovy
+++
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_encryption_native_reader.groovy
@@ -36,6 +36,8 @@ suite("test_iceberg_encryption_native_reader", "p0,external")
{
String prefix = "native-pme-${UUID.randomUUID()}"
String plainKey = "${prefix}/mixed/00-plain.parquet"
String encryptedKey = "${prefix}/mixed/10-encrypted.parquet"
+ String indexedPlainKey = "${prefix}/indexed/00-plain.parquet"
+ String indexedEncryptedKey = "${prefix}/indexed/10-encrypted.parquet"
String truncatedKey = "${prefix}/invalid/truncated.parquet"
File fixtures = new File(context.config.dataPath,
"external_table_p0/iceberg/encryption")
File encrypted = new File(fixtures, "iceberg_java.parquet")
@@ -43,7 +45,8 @@ suite("test_iceberg_encryption_native_reader", "p0,external")
{
assertEquals("PARE", new String(Arrays.copyOf(encrypted.bytes, 4),
StandardCharsets.US_ASCII))
assertEquals("PAR1", new String(Arrays.copyOf(plain.bytes, 4),
StandardCharsets.US_ASCII))
def previous = sql """SELECT @@enable_file_scanner_v2,
- @@enable_count_push_down_for_external_table,
@@enable_parquet_file_page_cache"""
+ @@enable_count_push_down_for_external_table,
@@enable_parquet_file_page_cache,
+ @@enable_parquet_filter_by_bloom_filter"""
def client = AmazonS3ClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(new
BasicAWSCredentials("admin", "password")))
.withEndpointConfiguration(new
AwsClientBuilder.EndpointConfiguration(endpoint, "us-east-1"))
@@ -61,6 +64,8 @@ suite("test_iceberg_encryption_native_reader", "p0,external")
{
bucketCreated = true
client.putObject(bucket, plainKey, plain)
client.putObject(bucket, encryptedKey, encrypted)
+ client.putObject(bucket, indexedPlainKey, new File(fixtures,
"indexes_java_plain.parquet"))
+ client.putObject(bucket, indexedEncryptedKey, new File(fixtures,
"indexes_java.parquet"))
byte[] encryptedBytes = encrypted.bytes
truncated.bytes = Arrays.copyOf(encryptedBytes, encryptedBytes.length
- 1)
client.putObject(bucket, truncatedKey, truncated)
@@ -91,6 +96,27 @@ suite("test_iceberg_encryption_native_reader",
"p0,external") {
order_qt_plain_after_rejection rows
}
}
+ // This toggles Bloom probes; native footer/page pruning remains
active. The physical-reader
+ // unit test separately disables all metadata pruning for its
full-scan reference.
+ for (String bloom : ["false", "true"]) {
+ sql "SET enable_parquet_filter_by_bloom_filter=${bloom}"
+ quickTest("index_tail_${bloom}", """SELECT COUNT(*), MIN(id),
MAX(id), COUNT(label)
+ FROM ${tvf(indexedPlainKey)} WHERE id > 900""")
+ quickTest("index_membership_${bloom}", """SELECT id, label FROM
${tvf(indexedPlainKey)}
+ WHERE id IN (3, 900, 1022) ORDER BY id""")
+ quickTest("index_absent_${bloom}", "SELECT COUNT(*) FROM
${tvf(indexedPlainKey)} WHERE id = 3")
+ // A negative Bloom probe must never authorize a keyless scan of
an encrypted file.
+ for (String key : [indexedEncryptedKey, prefix +
'/indexed/*.parquet']) {
+ test {
+ sql "SELECT id FROM ${tvf(key)} WHERE id = 3"
+ exception "File encryption [MISSING_KEY]"
+ }
+ test {
+ sql "SELECT COUNT(*) FROM ${tvf(key)} WHERE id > 900"
+ exception "File encryption [MISSING_KEY]"
+ }
+ }
+ }
test {
sql "SELECT * FROM ${tvf(truncatedKey)}"
exception "Invalid Parquet v2 footer magic"
@@ -101,12 +127,13 @@ suite("test_iceberg_encryption_native_reader",
"p0,external") {
sql "SET enable_file_scanner_v2=${previous[0][0]}"
sql "SET
enable_count_push_down_for_external_table=${previous[0][1]}"
sql "SET enable_parquet_file_page_cache=${previous[0][2]}"
+ sql "SET enable_parquet_filter_by_bloom_filter=${previous[0][3]}"
} finally {
try {
// A failed bucket creation has nothing to clean up. Once
created, this UUID bucket
// belongs only to this attempt, including any object from an
interrupted upload.
if (bucketCreated) {
- for (String key : [plainKey, encryptedKey, truncatedKey]) {
+ for (String key : [plainKey, encryptedKey, truncatedKey,
indexedPlainKey, indexedEncryptedKey]) {
client.deleteObject(bucket, key)
}
client.deleteBucket(bucket)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]