shangxinli commented on code in PR #3935: URL: https://github.com/apache/parquet-java/pull/3935#discussion_r4589794755
########## parquet-hadoop/README.md: ########## @@ -451,6 +451,10 @@ Note: nested column names must be specified as full dot-separated paths for each **Property:** `parquet.encryption.algorithm` **Description:** Parquet encryption algorithm. Can be `AES_GCM_V1` or `AES_GCM_CTR_V1`. +Since 1.19.0, DataPageV2 pages of encrypted columns encrypt the repetition and definition levels with the values, +as the specification requires. Earlier versions wrote the levels in plaintext. Version 1.19.0 reads both layouts. +Versions 1.18 and earlier cannot read the new pages. With `AES_GCM_V1`, decryption fails. With `AES_GCM_CTR_V1`, +the levels are incorrect. Review Comment: Running a parquet-java 1.18.2 reader against the Arrow CTR fixtures in this PR returns wrong nulls and wrong values with no error for flat optional columns, for example 468 of 500 rows wrong on id, even with Snappy, while GCM fails with a tag mismatch. So I think this note understates it. Could we add a writer property to keep the old layout for a release, with the new layout as the default, or at least spell this out in the release notes? Repro: https://gist.github.com/shangxinli/3816371fe541e7cbf11fc6daa554a059 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
