JingsongLi commented on code in PR #8486:
URL: https://github.com/apache/paimon/pull/8486#discussion_r3534180636
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/write/PaimonV2Write.scala:
##########
@@ -84,11 +84,16 @@ class PaimonV2Write(
PaimonAddedTableFilesMetric()
)
if (copyOnWriteScan.isEmpty) {
- // todo: support record metrics for row level ops
buffer += PaimonInsertedRecordsMetric()
}
if (copyOnWriteScan.nonEmpty) {
buffer += PaimonDeletedTableFilesMetric()
+ // For DELETE, the number of deleted records is exactly the row count
difference between
+ // the removed files and the rewritten files. For UPDATE and MERGE, the
rewritten files
+ // mix copied rows with modified rows, so record metrics cannot be
derived from file stats.
+ if (operationType.contains(Snapshot.Operation.DELETE)) {
Review Comment:
This condition misses the DELETE plan used by the added test:
`metrics("deletedRecords")` is absent before the write runs. I ran `mvn -pl
paimon-spark/paimon-spark-ut -am -Pfast-build -DfailIfNoTests=false
-DwildcardSuites=org.apache.paimon.spark.sql.PaimonMetricTest -Dtest=none
test`, and `Paimon Metric: delete` fails with `NoSuchElementException: key not
found: deletedRecords` at `PaimonMetricTest.scala:188`. Since Spark builds the
command metrics from `supportedCustomMetrics`, we need to register
`PaimonDeletedRecordsMetric` for the row-level DELETE write path that actually
reaches this test, or ensure `operationType` is populated before
`supportedCustomMetrics` is queried.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
