viirya commented on code in PR #7720:
URL: https://github.com/apache/datafusion-comet/pull/7720#discussion_r4650224337


##########
docs/source/user-guide/latest/parquet.md:
##########
@@ -38,6 +38,11 @@ described in [Supported Data 
Sources](datasources.md#supported-storages).
 The following features are not supported and cause Comet to fall back to Spark:
 
 - Decimals encoded in binary format.
+- Scans configured for Parquet modular encryption that read a timestamp inside 
an array, including

Review Comment:
   The #7029 entry under "Reads that differ from Spark" still says these lists 
read back wrong in every scan, and #7148 rewrites that same entry to describe 
the encrypted gap. If both merge as written, this page will say encrypted scans 
both read wrong and fall back. Could one of the two PRs own that entry, and 
could this bullet link to it? Also, "Spark decodes the timestamps to avoid 
INT96 overflow" is a little hard to parse. Maybe "so that Spark reads the INT96 
timestamps that DataFusion would overflow"?



##########
spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala:
##########
@@ -275,18 +276,28 @@ case class CometScanRule(session: SparkSession)
     if (!CometNativeScan.isSupported(scanExec)) {
       return None
     }
-    if (encryptionEnabled(hadoopConf) && 
!isEncryptionConfigSupported(hadoopConf)) {
+    val encryptionConfigured = encryptionEnabled(hadoopConf)
+    if (encryptionConfigured && !isEncryptionConfigSupported(hadoopConf)) {
       withFallbackReason(scanExec, "Native Parquet scan does not support 
encryption")
       return None
     }
     // TODO: Remove this fallback once DataFusion can ignore embedded Arrow 
schema hints and
     // preserve Spark's ENUM inference without losing Parquet decryption state.
     // https://github.com/apache/datafusion-comet/issues/5477
-    if (encryptionEnabled(hadoopConf) &&
+    if (encryptionConfigured &&
       scanExec.requiredSchema.exists(field => isVariantType(field.dataType))) {
       withFallbackReason(scanExec, "Native Parquet Variant scans do not 
support encryption")
       return None
     }
+    // Legacy lists can hide INT96 leaves from DataFusion's microsecond 
coercion. Rebuilding
+    // the footer with a schema hint would lose the decryptor. Remove after 
adopting
+    // https://github.com/apache/datafusion/pull/26464.

Review Comment:
   "Remove after adopting #26464" is easy to read as met by the DataFusion 56 
upgrade, but 56.0.0 doesn't contain that fix. Could this use the same `TODO:` 
form as the Variant guard above, link #7029, and say the removal needs a 
DataFusion release that includes #26464?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to