parthchandra commented on code in PR #4752:
URL: https://github.com/apache/datafusion-comet/pull/4752#discussion_r3686066772
##########
spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala:
##########
@@ -882,6 +930,29 @@ case class CometScanTypeChecker() extends DataTypeSupport
with CometTypeShim {
object CometScanRule extends Logging {
+ // Iceberg FileIO implementations whose underlying storage object_store can
reach.
+ // Custom/test FileIO classes (e.g. CustomFileIO in TestSparkExecutorCache)
are not compatible
+ // because Comet's native reader bypasses Java FileIO entirely.
+ private val CompatibleFileIOClasses: Set[String] = Set(
+ "org.apache.iceberg.hadoop.HadoopFileIO",
+ "org.apache.iceberg.aws.s3.S3FileIO",
+ "org.apache.iceberg.gcp.gcs.GCSFileIO",
+ "org.apache.iceberg.io.ResolvingFileIO",
+ "org.apache.iceberg.spark.SparkFileIO",
+ "org.apache.iceberg.azure.adlsv2.ADLSFileIO",
+ "org.apache.iceberg.CachingFileIO")
+
+ // Prefix of the EncryptingFileIO family. An encrypted table's io() is not
the bare
+ // EncryptingFileIO but a nested variant chosen from the wrapped delegate's
capabilities
+ // (e.g. EncryptingFileIO$WithSupportsPrefixOperations when the delegate is
HadoopFileIO), so
+ // an exact class-name match misses it. Comet forwards each file's
key_metadata to iceberg-rust
+ // and reads the ciphertext via object_store, so any EncryptingFileIO
variant is compatible.
+ private val EncryptingFileIOPrefix =
"org.apache.iceberg.encryption.EncryptingFileIO"
Review Comment:
I had to do this because encryption tests failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
