PinkCrow007 opened a new issue, #636: URL: https://github.com/apache/parquet-format/issues/636
We noticed an interoperability issue between Parquet implementations when reading encrypted DataPageV2 pages with non-empty repetition or definition levels. Arrow and parquet-mr handle encryption of the DataPageV2 body differently: - **parquet-mr**: Leaves **rep/def levels in plaintext** and encrypts only the values. Code pointer: [ColumnChunkPageWriteStore.java](https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ColumnChunkPageWriteStore.java) writePageV2 - **Arrow C++ and arrow-rs**: **Encrypt the entire page body**, including both rep/def levels and values. [parquet_modular_encryption_interop_test.py](https://github.com/user-attachments/files/34394744/parquet_modular_encryption_interop_test.py) Code pointer: [column_writer.cc](https://github.com/apache/arrow/blob/main/cpp/src/parquet/column_writer.cc) BuildDataPageV2 As a result, the implementations **cannot read each other**'s encrypted pages when rep/def levels are present. Since repetition and definition levels contain structural data (like null distributions and array lengths), it feels like they should be encrypted alongside the values to keep that data secure. **Question**: What should be the expected behavior? How should we solve this interop issue moving forward? Here is a python script to reproduce this with PyArrow 17.0.0 and Spark 4.1 (parquet-mr 1.16.0). The results are: ``` == V1 data pages Arrow writes, Arrow reads: OK Spark writes, Spark reads: OK Arrow writes, Spark reads: OK Spark writes, Arrow reads: OK == V2 data pages Arrow writes, Arrow reads: OK Spark writes, Spark reads: OK Arrow writes, Spark reads: FAILED: AEADBadTagException: Tag mismatch Spark writes, Arrow reads: FAILED: Negative ciphertext length -541098369 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
