liujiayi771 opened a new pull request, #10563:
URL: https://github.com/apache/paimon/pull/10563
### Purpose
A V2 write can commit successfully and then fail while reporting optional
driver metrics. `Compatibility.getExecutionMetrics` calls `.get` on
`SQLAppStatusStore.execution`, which returns `None` when the execution has no
stored UI record. The SQL status listener updates that store asynchronously. A
batch write invoked without a SQL execution ID also fails when converting the
missing ID to `Long`.
Return an empty metric sequence when the execution record is absent, and
skip driver metric reporting when the execution ID or metrics are missing.
Apply the same lookup behavior to the Spark 3.2, 3.3 and 3.4 compatibility
implementations. Preserve metric matching and reporting when metrics exist, and
leave data commit, exception propagation and abort behavior unchanged.
### Tests
- Added regression coverage using the real SQL status store with its
listener temporarily removed, while Spark supplies the actual execution ID.
Before the fix, both empty and nonempty DataFrame appends fail with
`NoSuchElementException: None.get` after commit.
- Covered the empty-source `limit(0).count()` / `coalesce(1)` /
`saveAsTable` path and verified `LocalTableScanExec`, unchanged target rows and
unchanged snapshot ID. The ordinary local empty append succeeds before the fix;
missing execution records are exercised deterministically by removing the
status listener.
- Verified nonempty append with missing execution metrics and direct
nonempty batch commit without an execution ID by reading back the committed
rows.
- Verified that a controlled commit failure propagates unchanged and closes
the commit object. Existing abort cleanup and normal metric reporting tests
also pass.
Validation on Spark 3.5.8 / Scala 2.12: **28 tests passed**, including
`WriteMetricsTest`, `PaimonMetricTest`, `SparkWriteITCase` and
`PaimonBatchWriteTest`, with Checkstyle, Spotless and Enforcer enabled:
```shell
mvn -pl paimon-spark/paimon-spark-ut -am -Pspark3 \
-DfailIfNoTests=false \
-DwildcardSuites=org.apache.paimon.spark.sql.WriteMetricsTest,org.apache.paimon.spark.sql.PaimonMetricTest,org.apache.paimon.spark.sql.SparkWriteITCase
\
-Dtest=PaimonBatchWriteTest test
```
Spark 3.2, 3.3, 3.4, 3.5, 4.0 and 4.1 modules also passed `test-compile`
with tests skipped. All builds ran in isolated local staging directories.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
