JingsongLi commented on code in PR #8094:
URL: https://github.com/apache/paimon/pull/8094#discussion_r3346218212
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/SparkTable.scala:
##########
@@ -115,8 +113,8 @@ object SparkTable {
case fs: FileStoreTable =>
fs.primaryKeys().isEmpty &&
!sparkTable.coreOptions.deletionVectorsEnabled() &&
- !sparkTable.coreOptions.rowTrackingEnabled() &&
- !sparkTable.coreOptions.dataEvolutionEnabled()
+ !sparkTable.coreOptions.dataEvolutionEnabled() &&
+ !SparkTypeUtils.containsCharType(fs.rowType())
Review Comment:
This now enables V2 row-level ops for row-tracking tables on Spark 3.5 as
well. Spark 3.5 does not have the metadata-aware row-level write path:
DataWriter only has write(T) (no write(metadata, data)), MetadataColumn does
not expose preserve flags, and V2Writes builds ReplaceData from rd.dataInput
after stripping metadata columns. Since PaimonBatchWriteBase still creates
PaimonV2MetadataAwareDataWriter whenever row tracking + COW, Spark 3.5 will
call the inherited single-arg write(record) path and rewrite rows with the
plain writer, losing the original _ROW_ID/_SEQUENCE_NUMBER for copied/updated
rows instead of preserving them. Could we keep row-tracking tables on the V1
path for Spark 3.5, e.g. only allow row-tracking V2 ops on Spark >= 4.0, or add
an explicit Spark 3.5 implementation/test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
