kerwin-zk opened a new pull request, #8539: URL: https://github.com/apache/paimon/pull/8539
<!-- Please specify the module before the PR name: [core] ... or [flink] ... --> ### Purpose This PR adds delta-based row-level operations (Spark's `SupportsDelta` / `WriteDelta` API) for deletion-vector enabled unaware-bucket append tables. DELETE, UPDATE and MERGE INTO no longer fall back to the V1 commands: deleted rows are marked in deletion vectors keyed by the `(__paimon_file_path, __paimon_row_index)` row ID, and inserted rows / new versions of updated rows go through the regular append writer — no data file is rewritten. Design notes: - `PaimonSparkDeltaOperation` implements `SupportsDelta` with `representUpdateAsDeleteAndInsert = false`: Spark plans a single UPDATE operation per row (no `Expand` doubling) and the delta writer decomposes it into a deletion-vector mark plus an appended row. - Tasks accumulate per-file deletion bitmaps; the driver merges them across tasks (Spark does not cluster delta rows by file), unions them with the files' existing deletion vectors through `BaseAppendDeleteFileMaintainer`, and commits index-only messages together with the appended data messages. - Scope is limited to unaware-bucket deletion-vector append tables: commit conflict detection is deletion-vector aware for them, so concurrent commits touching the same file fail and retry instead of losing deletions. Bucketed deletion-vector append tables keep the V1 path because `IndexManifestFileHandler` overwrites the whole bucket's index without conflict checking. PK, data-evolution, row-tracking and CHAR tables are unchanged. - On Spark 4.1 the built-in rewrites short-circuit on `analyzed=true` UPDATE/MERGE subtrees (same issue as the existing pure-append rules), so `Spark41UpdateTableRewrite` / `Spark41MergeIntoRewrite` transcribe the `buildWriteDeltaPlan` branches, reusing the protected helpers of `RewriteRowLevelCommand`. - The delta writer reports exact `deletedRecords` / `updatedRecords` / `insertedRecords` task metrics for all three commands — numbers the copy-on-write path cannot derive from file statistics (see #8486). The commit-level `insertedRecords` gauge is filtered out on the delta path because `CommitStats.deltaRecordsAppended` mixes ADD and DELETE row counts and would overwrite the exact values. - `__paimon_file_path` / `__paimon_row_index` metadata columns now answer `MetadataColumn.isNullable = false` (required by Spark's row ID contract); their internal `toStructField` / `toAttribute` stay nullable because V1 commands project them through outer joins. ### Tests - New `DeltaRowLevelOpsTest`: DELETE (partial file / whole file / repeated deletes merging vectors / zero rows / partitioned), UPDATE (values, partition-moving rows, NULL, repeated), MERGE INTO (all clause combinations, NOT MATCHED BY SOURCE, cardinality violation), plus negative gating (metadata-only DELETE keeps the V1 truncate fast path; bucketed deletion-vector tables, plain append tables and `use-v2-write=false` keep their current paths). Passed under both the spark3 (3.5) and spark4 (4.1) profiles. - `PaimonMetricTest` extended with exact record metrics assertions for the three delta commands. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
