viirya commented on code in PR #7720:
URL: https://github.com/apache/datafusion-comet/pull/7720#discussion_r4650224337
##########
docs/source/user-guide/latest/parquet.md:
##########
@@ -38,6 +38,11 @@ described in [Supported Data
Sources](datasources.md#supported-storages).
The following features are not supported and cause Comet to fall back to Spark:
- Decimals encoded in binary format.
+- Scans configured for Parquet modular encryption that read a timestamp inside
an array, including
Review Comment:
The #7029 entry under "Reads that differ from Spark" still says these lists
read back wrong in every scan, and #7148 rewrites that same entry to describe
the encrypted gap. If both merge as written, this page will say encrypted scans
both read wrong and fall back. Could one of the two PRs own that entry, and
could this bullet link to it? Also, "Spark decodes the timestamps to avoid
INT96 overflow" is a little hard to parse. Maybe "so that Spark reads the INT96
timestamps that DataFusion would overflow"?
##########
spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala:
##########
@@ -275,18 +276,28 @@ case class CometScanRule(session: SparkSession)
if (!CometNativeScan.isSupported(scanExec)) {
return None
}
- if (encryptionEnabled(hadoopConf) &&
!isEncryptionConfigSupported(hadoopConf)) {
+ val encryptionConfigured = encryptionEnabled(hadoopConf)
+ if (encryptionConfigured && !isEncryptionConfigSupported(hadoopConf)) {
withFallbackReason(scanExec, "Native Parquet scan does not support
encryption")
return None
}
// TODO: Remove this fallback once DataFusion can ignore embedded Arrow
schema hints and
// preserve Spark's ENUM inference without losing Parquet decryption state.
// https://github.com/apache/datafusion-comet/issues/5477
- if (encryptionEnabled(hadoopConf) &&
+ if (encryptionConfigured &&
scanExec.requiredSchema.exists(field => isVariantType(field.dataType))) {
withFallbackReason(scanExec, "Native Parquet Variant scans do not
support encryption")
return None
}
+ // Legacy lists can hide INT96 leaves from DataFusion's microsecond
coercion. Rebuilding
+ // the footer with a schema hint would lose the decryptor. Remove after
adopting
+ // https://github.com/apache/datafusion/pull/26464.
Review Comment:
"Remove after adopting #26464" is easy to read as met by the DataFusion 56
upgrade, but 56.0.0 doesn't contain that fix. Could this use the same `TODO:`
form as the Variant guard above, link #7029, and say the removal needs a
DataFusion release that includes #26464?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]