This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 28cbcfc233e [fix](build) Enable Parquet modular encryption in Arrow
(#70087)
28cbcfc233e is described below
commit 28cbcfc233e81218e5a3e38b83676637fa53772e
Author: Gabriel <[email protected]>
AuthorDate: Wed Nov 4 10:12:58 2026 +0800
[fix](build) Enable Parquet modular encryption in Arrow (#70087)
### What problem does this PR solve?
The third-party Arrow build leaves `PARQUET_REQUIRE_ENCRYPTION` at its
default `OFF`. Parquet still installs encryption APIs and exports some
encryption symbols, but its archive contains
`encryption_internal_nossl.cc.o`. Reading a standard PME-encrypted
Parquet file with the correct key then throws:
```text
Calling encryption method in Arrow/Parquet built without OpenSSL
```
Linking `libssl.a` and `libcrypto.a` into the consumer does not replace
those compiled stubs. This change supplies the build prerequisites for
native encrypted Iceberg reads.
The change:
- Enables `PARQUET_REQUIRE_ENCRYPTION` and selects static OpenSSL.
- Pins OpenSSL headers and both archives to the Doris third-party
prefix, and fails before CMake if any required input is missing.
- Installs `encryption_internal.h` and `internal_file_decryptor.h` from
the same pinned Arrow source. Arrow normally excludes these internal
headers, but module-level decryption is needed to retain the native
Parquet scan path.
Existing third-party prebuilt packages must be rebuilt to pick up these
changes. SQL-level encrypted Iceberg read integration remains separate
work.
### Release note
None
### Check List (For Author)
- Test:
- [x] Manual test (details below)
- Behavior changed:
- [x] Yes. Third-party Parquet is built with PME enabled; missing owned
OpenSSL inputs fail before configure.
- Does this need documentation?
- [x] No. This is the third-party build prerequisite, with no new
SQL-facing configuration.
#### Validation
On Linux x86_64, extracted Arrow 24.0.0 from the archive pinned by
`thirdparty/vars.sh`, applied the current Doris Arrow patches, and
executed the changed `build_arrow()` function in an isolated prefix
seeded with existing third-party dependencies. The complete Arrow build
and install succeeded. CMake selected PME `ON` and OpenSSL 1.1.1s from
that prefix. The rebuilt archive contains `encryption_internal.cc.o` /
`openssl_internal.cc.o`, with no `nossl` object.
A standalone C++ reader linked directly to the checked/rebuilt static
archives, including the same prefix's OpenSSL archives, used the public
Apache [Parquet test
fixtures](https://github.com/apache/parquet-testing/tree/master/data):
| Check | Before: checked PME-disabled package | After: rebuilt package
|
| --- | --- | --- |
| `uniform_encryption.parquet.encrypted`, correct public test key |
OpenSSL-disabled runtime error | All-column scan succeeds: 50 rows |
| Same file, wrong key | OpenSSL-disabled runtime error | Authentication
failure: `Failed decryption finalization` |
| Same file, no key | Missing decryption-properties error | Missing
decryption-properties error |
| `alltypes_plain.parquet` | All-column scan succeeds: 8 rows |
All-column scan succeeds: 8 rows |
Additional checks:
- Module-level AES-GCM round trip and rejection of a different data-page
AAD pass.
- Both installed internal headers match the source and compile/link with
`InternalFileDecryptor` and the module decryptors.
- Removing each required OpenSSL input rejects the build before CMake.
The baseline function reaches CMake for the same incomplete inputs.
- Captured Linux and Darwin invocations contain the PME option and
explicit static OpenSSL paths; this is an argument check, not a macOS
build.
- `bash -n thirdparty/build-thirdparty.sh` and `git diff --check` pass.
Validation limits: no full Doris BE build, Doris SQL test, AArch64 build
or native macOS build was run. No runtime reader behavior is enabled by
this PR alone.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
thirdparty/build-thirdparty.sh | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh
index 0f0b0990fc9..c751afc9491 100755
--- a/thirdparty/build-thirdparty.sh
+++ b/thirdparty/build-thirdparty.sh
@@ -1165,6 +1165,16 @@ build_arrow() {
check_if_source_exist "${ARROW_SOURCE}"
cd "${TP_SOURCE_DIR}/${ARROW_SOURCE}/cpp"
+ # CMake otherwise accepts a host OpenSSL when this dependency is missing.
+ # Keep both headers and static libraries in the same Doris third-party
prefix.
+ local openssl_file
+ for openssl_file in include/openssl/ssl.h lib/libssl.a lib/libcrypto.a; do
+ if [[ ! -f "${TP_INSTALL_DIR}/${openssl_file}" ]]; then
+ echo "Missing Arrow OpenSSL dependency:
${TP_INSTALL_DIR}/${openssl_file}"
+ exit 1
+ fi
+ done
+
mkdir -p release
cd release
@@ -1187,10 +1197,17 @@ build_arrow() {
ldflags="-L${TP_LIB_DIR}"
fi
+ # Without this option Parquet links encryption stubs that throw at runtime,
+ # even when the BE itself already links OpenSSL.
LDFLAGS="${ldflags}" \
"${CMAKE_CMD}" -DCMAKE_POLICY_VERSION_MINIMUM=3.5 \
-DCMAKE_CXX_STANDARD="${TP_CXX_STANDARD}" \
-G "${GENERATOR}" -DARROW_PARQUET=ON -DARROW_IPC=ON
-DARROW_BUILD_SHARED=OFF \
+ -DPARQUET_REQUIRE_ENCRYPTION=ON -DARROW_OPENSSL_USE_SHARED=OFF \
+ -DOPENSSL_ROOT_DIR="${TP_INSTALL_DIR}" \
+ -DOPENSSL_INCLUDE_DIR="${TP_INSTALL_DIR}/include" \
+ -DOPENSSL_SSL_LIBRARY="${TP_INSTALL_DIR}/lib/libssl.a" \
+ -DOPENSSL_CRYPTO_LIBRARY="${TP_INSTALL_DIR}/lib/libcrypto.a" \
-DARROW_BUILD_STATIC=ON -DARROW_WITH_BROTLI=ON -DARROW_WITH_LZ4=ON
-DARROW_USE_GLOG=ON \
-DARROW_WITH_SNAPPY=ON -DARROW_WITH_ZLIB=ON -DARROW_WITH_ZSTD=ON
-DARROW_JSON=ON \
-DARROW_WITH_UTF8PROC=OFF -DARROW_WITH_RE2=ON -DARROW_ORC=ON \
@@ -1240,6 +1257,12 @@ build_arrow() {
"${BUILD_SYSTEM}" -j "${PARALLEL}"
"${BUILD_SYSTEM}" install
+ # Arrow excludes internal headers from installation, but the native Parquet
+ # reader needs these module decryptors instead of replacing its scan path.
+ install -m 644 ../src/parquet/encryption/encryption_internal.h \
+ ../src/parquet/encryption/internal_file_decryptor.h \
+ "${TP_INSTALL_DIR}/include/parquet/encryption/"
+
#copy dep libs
cp -rf ./brotli_ep/src/brotli_ep-install/lib/libbrotlienc-static.a
"${TP_INSTALL_DIR}/lib64/libbrotlienc.a"
cp -rf ./brotli_ep/src/brotli_ep-install/lib/libbrotlidec-static.a
"${TP_INSTALL_DIR}/lib64/libbrotlidec.a"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
