iemejia commented on code in PR #12612:
URL: https://github.com/apache/gluten/pull/12612#discussion_r3822998501
##########
gluten-delta/src/main/scala/org/apache/gluten/execution/DeltaScanTransformer.scala:
##########
@@ -98,20 +99,29 @@ case class DeltaScanTransformer(
override def getSplitInfosFromPartitions(
partitions: Seq[(Partition, ReadFileFormat)]): Seq[SplitInfo] = {
val splitInfos = super.getSplitInfosFromPartitions(partitions)
- val partitionColumnCount = getPartitionSchema.fields.length
- splitInfos.zip(partitions).map {
- case (localFiles: LocalFilesNode, (filePartition: FilePartition, _)) =>
- DeltaDeletionVectorScanInfo
- .normalize(partitionColumnCount, filePartition.files.toSeq)
- .map {
- case (otherMetadataColumns, deltaReadOptions) =>
- DeltaLocalFilesBuilder.makeDeltaLocalFiles(
- localFiles,
- otherMetadataColumns.asJava,
- deltaReadOptions.asJava): SplitInfo
- }
- .getOrElse(localFiles)
- case (splitInfo, _) => splitInfo
+ // Deletion vectors only exist on Delta tables read through a
TahoeFileIndex (which also covers
+ // PreparedDeltaFileIndex). Its `path` is the authoritative table root and
is used to resolve
+ // per-file DV locations. Any other location cannot carry Delta DV
metadata, so the generic
+ // split representation is returned unchanged.
+ relation.location match {
+ case tahoe: TahoeFileIndex =>
+ val tableRootPath = tahoe.path
Review Comment:
Added in ffa69edb5. New `DeltaSuite` test "deletion vector on shallow-cloned
table" (min Spark 3.4): it `DELETE`s on a `SHALLOW CLONE`, whose data files
point absolute into the source while the DV is written clone-root-relative
(`"u"`). The old walk-up would resolve the source root and fail to find the DV
here; sourcing the root from `TahoeFileIndex.path` makes it pass. It asserts
the `DeltaScanTransformer` arm is used (on 3.5+) and checks results, so this
arm is now pinned on 3.5. Compiles clean on Scala 2.12/Spark 3.5 and Scala
2.13/Spark 4.0; will confirm the runtime assertion once CI runs.
##########
.github/workflows/util/delta-spark-ut/known-failures.txt:
##########
@@ -59,8 +59,6 @@ org.apache.spark.sql.delta.CloneTableSQLSuite#shallow clone
across file systems
org.apache.spark.sql.delta.CloneTableSQLWithCatalogOwnedBatch100Suite#shallow
clone across file systems
org.apache.spark.sql.delta.CloneTableSQLWithCatalogOwnedBatch1Suite#shallow
clone across file systems
org.apache.spark.sql.delta.CloneTableSQLWithCatalogOwnedBatch2Suite#shallow
clone across file systems
-org.apache.spark.sql.delta.CloneTableScalaDeletionVectorSuite#Cloning table
with persistent DVs and absolute parquet paths
-org.apache.spark.sql.delta.CloneTableScalaDeletionVectorSuite#Shallow clone
round-trip with DVs
Review Comment:
Follow-up: added the shallow-clone-then-`DELETE` read as a `DeltaSuite` test
in ffa69edb5, so the 3.5 gap is now covered directly (not just via the delta40
clone shards). Details on the `DeltaScanTransformer` thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
