felipepessoto commented on code in PR #13042:
URL: https://github.com/apache/gluten/pull/13042#discussion_r4277981970
##########
gluten-delta/src/main/scala/org/apache/gluten/extension/OffloadDeltaScan.scala:
##########
@@ -35,9 +35,13 @@ case class OffloadDeltaScan(enableNativeDmlRowIndexScan:
Boolean) extends Offloa
// Spark 3.5+ exposes this as
ParquetFileFormat.ROW_INDEX_TEMPORARY_COLUMN_NAME.
private val parquetTemporaryRowIndexColumnName = "_tmp_metadata_row_index"
- // Row-index columns Delta generates as top-level scan outputs.
+ // Row-index columns used by Delta DML with either row-index generation path.
private val generatedRowIndexColumnNames =
Set(DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME,
parquetTemporaryRowIndexColumnName)
+ private val generatedDeletionVectorMetadataColumnNames =
+ Set(
+ DeltaParquetFileFormat.ROW_INDEX_COLUMN_NAME,
+ DeltaParquetFileFormat.IS_ROW_DELETED_COLUMN_NAME)
Review Comment:
Native handling of `_tmp_metadata_row_index` is intentionally retained.
Following malinjawi's feedback, 76ac03f44 narrowed this fallback to
`__delta_internal_row_index` and `__delta_internal_is_row_deleted`; Spark's
temporary Parquet row-index field does not need Delta's JVM synthesis when
`useMetadataRowIndex=false`.
The updated PR description explicitly states that `_metadata.row_index`
(backed by `_tmp_metadata_row_index`) remains eligible for native offload. Both
src-delta33 and src-delta40 also now contain `Delta metadata row-index scan
should offload when useMetadataRowIndex is false`: each executes a real `SELECT
id, _metadata.row_index`, checks the complete value/index pairs, and asserts
native execution.
The requested scope clarification and coverage are therefore already
present. Re-adding the temporary name would restore the unnecessary fallback we
just removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
