jianguotian opened a new pull request, #9802:
URL: https://github.com/apache/paimon/pull/9802
# [core] Support one-shot forced manifest rewrite
### Purpose
This is the independent follow-up requested during the review of #9784. It
builds on the manifest pruning and sorting changes merged separately in #9791
and #9792.
After manifest sorting is enabled or its sort order changes, manifests from
an existing table may already satisfy the normal full-compaction threshold and
LSM run picker. In that case, `compact_manifest` is a no-op and cannot migrate
the existing manifest layout or populate newly available manifest metadata.
### Changes
- Add `manifest-sort.force-rewrite`, disabled by default.
- When enabled for an explicit manifest compaction:
- bypass the normal full-compaction threshold;
- select all existing sorted runs instead of only runs chosen by the
normal picker;
- combine selected manifests into one global rewrite section so legacy
runs can be reclustered using the current sort order;
- allow an already compacted single manifest to be rewritten.
- Keep the existing `manifest-sort.max-rewrite-size` budget semantics,
including partial rewrites at manifest-file granularity.
- Add the optional `manifest_sort_order` argument to the existing Spark and
Flink `compact_manifest` procedures:
- `bucket-first` rewrites a bucketed table into bucket-first layout;
- `partition-first` rewrites into partition-first layout;
- specifying an order enables manifest sorting and forced rewrite for that
invocation.
- Keep the existing #9792 layout selection unchanged when the argument is
omitted. No `auto` value or state is introduced.
- Reject `bucket-first` for non-bucket tables and explicit layout overrides
for data evolution tables.
- Document the option as a one-shot dynamic maintenance option which should
not be persisted for routine writes.
This reuses the existing `compact_manifest` procedure and rewrites manifest
files only, not data files. The explicit order and the internal force/order
options are applied to a copied table as dynamic options and are not persisted
in the table schema.
### Usage
```sql
CALL sys.compact_manifest(
`table` => 'default.T',
`manifest_sort_order` => 'partition-first',
`manifest_sort_max_rewrite_size` => '1 gb'
);
CALL sys.compact_manifest(
`table` => 'default.T',
`manifest_sort_order` => 'bucket-first',
`manifest_sort_max_rewrite_size` => '1 gb'
);
```
Use a sufficiently large `manifest_sort_max_rewrite_size` when all eligible
manifests should be migrated in one invocation. The rewrite budget follows the
existing file-granular semantics and is not a strict byte-level hard limit.
### Tests
- Verify that default behavior remains a no-op for already compacted
manifests.
- Rewrite legacy manifests without bucket-range metadata into the current
bucket-first order, preserve equivalent entries, and repopulate bucket metadata.
- Verify forced rewriting across multiple existing level runs.
- Verify forced rewriting of a single manifest.
- Verify partial rewrite behavior with a small rewrite budget.
- Verify switching the same manifest entries from partition-first to
bucket-first layout without changing the entries.
- Verify unsupported order values and contradictory/unsupported table
combinations fail clearly.
- Verify that procedure-supplied force and order options are not persisted
and that a subsequent ordinary compaction is a no-op.
Validation performed:
- `ManifestFileMetaTest`: 94 tests, 0 failures, 0 errors (1 existing skip).
- `CompactManifestProcedureITCase`: 7 tests, all passed.
- Spark 3 procedure main and test sources compile successfully with the new
trailing argument.
- Spotless formatting applied successfully to all modified Java and Scala
modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
