Gabriel39 opened a new pull request, #70087:
URL: https://github.com/apache/doris/pull/70087
### What problem does this PR solve?
The third-party Arrow build leaves `PARQUET_REQUIRE_ENCRYPTION` at its
default `OFF`. Parquet still installs encryption APIs and exports some
encryption symbols, but its archive contains `encryption_internal_nossl.cc.o`.
Reading a standard PME-encrypted Parquet file with the correct key then throws:
```text
Calling encryption method in Arrow/Parquet built without OpenSSL
```
Linking `libssl.a` and `libcrypto.a` into the consumer does not replace
those compiled stubs. This change supplies the build prerequisites for native
encrypted Iceberg reads.
The change:
- Enables `PARQUET_REQUIRE_ENCRYPTION` and selects static OpenSSL.
- Pins OpenSSL headers and both archives to the Doris third-party prefix,
and fails before CMake if any required input is missing.
- Installs `encryption_internal.h` and `internal_file_decryptor.h` from the
same pinned Arrow source. Arrow normally excludes these internal headers, but
module-level decryption is needed to retain the native Parquet scan path.
Existing third-party prebuilt packages must be rebuilt to pick up these
changes. SQL-level encrypted Iceberg read integration remains separate work.
### Release note
None
### Check List (For Author)
- Test:
- [x] Manual test (details below)
- Behavior changed:
- [x] Yes. Third-party Parquet is built with PME enabled; missing owned
OpenSSL inputs fail before configure.
- Does this need documentation?
- [x] No. This is the third-party build prerequisite, with no new
SQL-facing configuration.
#### Validation
On Linux x86_64, extracted Arrow 24.0.0 from the archive pinned by
`thirdparty/vars.sh`, applied the current Doris Arrow patches, and executed the
changed `build_arrow()` function in an isolated prefix seeded with existing
third-party dependencies. The complete Arrow build and install succeeded. CMake
selected PME `ON` and OpenSSL 1.1.1s from that prefix. The rebuilt archive
contains `encryption_internal.cc.o` / `openssl_internal.cc.o`, with no `nossl`
object.
A standalone C++ reader linked directly to the checked/rebuilt static
archives, including the same prefix's OpenSSL archives, used the public Apache
[Parquet test
fixtures](https://github.com/apache/parquet-testing/tree/master/data):
| Check | Before: checked PME-disabled package | After: rebuilt package |
| --- | --- | --- |
| `uniform_encryption.parquet.encrypted`, correct public test key |
OpenSSL-disabled runtime error | All-column scan succeeds: 50 rows |
| Same file, wrong key | OpenSSL-disabled runtime error | Authentication
failure: `Failed decryption finalization` |
| Same file, no key | Missing decryption-properties error | Missing
decryption-properties error |
| `alltypes_plain.parquet` | All-column scan succeeds: 8 rows | All-column
scan succeeds: 8 rows |
Additional checks:
- Module-level AES-GCM round trip and rejection of a different data-page AAD
pass.
- Both installed internal headers match the source and compile/link with
`InternalFileDecryptor` and the module decryptors.
- Removing each required OpenSSL input rejects the build before CMake. The
baseline function reaches CMake for the same incomplete inputs.
- Captured Linux and Darwin invocations contain the PME option and explicit
static OpenSSL paths; this is an argument check, not a macOS build.
- `bash -n thirdparty/build-thirdparty.sh` and `git diff --check` pass.
Validation limits: no full Doris BE build, Doris SQL test, AArch64 build or
native macOS build was run. No runtime reader behavior is enabled by this PR
alone.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
