sunchao opened a new pull request, #7720:
URL: https://github.com/apache/datafusion-comet/pull/7720

   ## Which issue does this PR close?
   
   Part of #7029. This addresses scans configured for Parquet modular 
encryption, which #7148 explicitly excludes.
   
   ## Rationale for this change
   
   In a legacy list, Spark can store an INT96 timestamp at `at.bag.array`. 
DataFusion 55.2.0 looks for `at.list.array`, misses the leaf, and decodes it as 
nanoseconds. A value such as `9999-12-31 23:59:59` overflows before Comet can 
cast it to microseconds.
   
   The footer rewrite in #7148 would drop the file decryptor, so that PR skips 
the rewrite whenever the reader has decryption properties. Comet supplies those 
properties for every file when modular encryption is configured. The layout fix 
in apache/datafusion#26464 has merged upstream and is pending a release that 
Comet can adopt.
   
   ## What changes are included in this PR?
   
   `CometScanRule` leaves the scan with Spark when modular encryption is 
configured and the required schema includes a timestamp beneath an array. The 
check looks through structs, nested arrays and map keys and values, for both 
`timestamp` and `timestamp_ntz`. Pruning the affected field allows Comet to 
scan other columns in the same file.
   
   The planner sees the projected Spark types before it knows each file's 
encoding. The fallback therefore also covers standard arrays, INT64 timestamp 
encodings and plaintext files in these scans. The user guide records that 
scope. The guard can be removed once Comet picks up apache/datafusion#26464.
   
   ## How are these changes tested?
   
   A new `ParquetEncryptionITCase` test writes a uniformly encrypted Parquet 
file with Spark's legacy list layout and INT96 encoding. It verifies the 
encrypted footer, compares a year-9999 timestamp nested in a map of arrays of 
structs with Spark, and checks the fallback reason. A second projection of the 
same file reads a scalar timestamp and an integer array, checks the values 
against Spark, and requires `CometNativeScanExec`.
   
   Ran Spotless on the changed Scala files and `git diff --check`. The 
regression is included in the hosted Linux scan suite. Tests were deferred to 
CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to