wwj6591812 opened a new pull request, #8716:
URL: https://github.com/apache/paimon/pull/8716
### Purpose
Related to #8322.
This PR adds a bounded Flink Action named `data_evolution_delete` for
logically deleting rows from non-primary-key Data Evolution append tables by
writing deletion vectors.
The existing Flink `delete` Action emits DELETE changelog records and does
not provide a deletion-vector path for non-primary-key Data Evolution append
tables. Rewriting data files for random deletion is particularly expensive for
Data Evolution tables with dedicated BLOB files. This Action provides a bounded
cleanup path for use cases such as retention policies, removing invalid
samples, and cache eviction.
The Action:
- validates that the target is a non-primary-key, bucket-unaware table with
row tracking, Data Evolution, and deletion vectors enabled;
- evaluates the user-provided Flink SQL predicate against a fixed snapshot
of the `$row_tracking` system table;
- maps matched global row IDs to Data Evolution anchor files and local row
positions;
- merges newly generated deletion vectors with existing deletion vectors;
- commits an index-only `DELETE` snapshot without rewriting existing data
files or dedicated BLOB files;
- gives every Action run a unique commit user so concurrent jobs cannot be
mistaken for committed retries;
- supports configurable sink parallelism;
- registers the new Action factory and documents its CLI, requirements, and
operational constraints.
Example:
```bash
<FLINK_HOME>/bin/flink run \
/path/to/paimon-flink-action-<version>.jar \
data_evolution_delete \
--warehouse <warehouse-path> \
--database <database-name> \
--table <table-name> \
--where "last_access_time < TIMESTAMP '2026-07-01 00:00:00'" \
--sink_parallelism 2
```
This is a logical delete. Physical data and BLOB reclamation still depends
on Data Evolution compaction with
`data-evolution.compaction.rewrite-row-ids=true` and snapshot expiration. The
fixed base snapshot must be retained while the Action runs, and concurrent
`APPEND`, `COMPACT`, `OVERWRITE`, or another delete Action against the same
partition may cause the commit to fail rather than silently overwrite deletion
vectors. For an unpartitioned table, increasing sink parallelism does not
parallelize final deletion-vector aggregation, so large deletes should be split
into bounded batches. Deletes touching many partitions should also be split to
limit repeated deletion-vector metadata scans.
This PR intentionally adds the Action only; a Flink Procedure can be added
separately.
### Tests
```bash
mvn -pl paimon-flink/paimon-flink-common -Pflink1 spotless:check
mvn -pl paimon-flink/paimon-flink-common -am -Pflink1 \
-DskipTests=false \
-Dtest=DataEvolutionDeleteActionITCase,ProcedureTest \
-Dsurefire.failIfNoSpecifiedTests=false test
```
Both commands pass on the latest `apache/paimon:master`. The targeted test
run executes 8 tests with 0 failures, errors, or skips:
- 6 `DataEvolutionDeleteActionITCase` tests covering cross-partition
deletion, merging existing deletion vectors, BLOB rows without data/BLOB
rewrite, no-match behavior, required table options, required `--where`, and two
Actions created from the same base snapshot conflicting instead of being
silently filtered;
- 2 `ProcedureTest` tests validating Action/Procedure coverage and named
arguments.
The full suite will be exercised by CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
