sunchao opened a new pull request, #7720: URL: https://github.com/apache/datafusion-comet/pull/7720
## Which issue does this PR close? Part of #7029. This addresses scans configured for Parquet modular encryption, which #7148 explicitly excludes. ## Rationale for this change In a legacy list, Spark can store an INT96 timestamp at `at.bag.array`. DataFusion 55.2.0 looks for `at.list.array`, misses the leaf, and decodes it as nanoseconds. A value such as `9999-12-31 23:59:59` overflows before Comet can cast it to microseconds. The footer rewrite in #7148 would drop the file decryptor, so that PR skips the rewrite whenever the reader has decryption properties. Comet supplies those properties for every file when modular encryption is configured. The layout fix in apache/datafusion#26464 has merged upstream and is pending a release that Comet can adopt. ## What changes are included in this PR? `CometScanRule` leaves the scan with Spark when modular encryption is configured and the required schema includes a timestamp beneath an array. The check looks through structs, nested arrays and map keys and values, for both `timestamp` and `timestamp_ntz`. Pruning the affected field allows Comet to scan other columns in the same file. The planner sees the projected Spark types before it knows each file's encoding. The fallback therefore also covers standard arrays, INT64 timestamp encodings and plaintext files in these scans. The user guide records that scope. The guard can be removed once Comet picks up apache/datafusion#26464. ## How are these changes tested? A new `ParquetEncryptionITCase` test writes a uniformly encrypted Parquet file with Spark's legacy list layout and INT96 encoding. It verifies the encrypted footer, compares a year-9999 timestamp nested in a map of arrays of structs with Spark, and checks the fallback reason. A second projection of the same file reads a scalar timestamp and an integer array, checks the values against Spark, and requires `CometNativeScanExec`. Ran Spotless on the changed Scala files and `git diff --check`. The regression is included in the hosted Linux scan suite. Tests were deferred to CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
