SteNicholas opened a new pull request, #227:
URL: https://github.com/apache/paimon-cpp/pull/227
### Purpose
Linked issue: close #204
Adds two independent capabilities to the write and compaction paths.
**1. Managed BLOB storage for primary-key tables**
A BLOB column of a primary-key table can now keep its payload out of the
data file
entirely. Payloads are externalized into rolling `.managed.blob` packs
*before* they
reach the merge-tree write buffer, so neither the buffer nor its spill ever
carries
payload bytes — only the fixed-size descriptor that replaces them.
- `PrimaryKeyBlobExternalizer` sits in front of the merge-tree writer, seals
a pack when
it reaches `blob.target-file-size`, and hands the packs it opened to the
commit message
its writer produces. A retract row drops its payload rather than storing
one, and a
batch of nothing but retracts opens no pack at all. An input that already
holds a
descriptor is re-materialized so the value is stored under this table's
own packs.
- Each data file keeps a `.blobref` sidecar (`ManagedBlobReferenceFile`, a
versioned
binary format with a golden-bytes test against the Java writer) listing
the packs its
rows reference. `ManagedBlobReferenceCollector` writes it on close; it
travels in the
data file's `extra_files`, so snapshot expiration, orphan cleaning and
abort all treat
it as part of the data file.
- Reads go through `ManagedBlobResolvingBatchReader`, which resolves each
stored
descriptor back to payload bytes with one ranged read per value.
- Ownership is explicit: a pack belongs to the commit message whose writer
created it, and
a compaction that merely inherits a pack does not own it.
`UncommittedFileCleaner` rolls
back exactly the packs a failed commit — or a `PrepareCommit` that gives
up partway —
created, and leaves the inherited ones alone.
- `SchemaValidation` enforces the merge-engine, primary-key, sequence-group
ordering and
option rules the layout relies on, so an unsupported combination
(`pk-clustering-override`,
`blob-descriptor.source-table`, a managed BLOB in a key or ordering field)
is rejected at
schema time rather than silently changing semantics.
Only top-level scalar BLOB columns are managed; a BLOB nested in a
ROW/MAP/ARRAY and the
existing inline descriptor/view fields are untouched.
**2. Compaction across the evolved field groups of a data-evolution table**
`AppendCompactCoordinator` previously refused a data-evolution table. It now
plans such a
table with `DataEvolutionCompactPlanner`: files covering the exact same rows
form one
evolved field group, the groups of a contiguous row-id run are bin-packed,
and each task
rewrites one bin into a single normal file holding every non-dedicated
column — preserving
row ids and the merged file-level sequence-number range.
- Because the rewrite keeps every input row, the deletions of the replaced
groups are
*re-keyed* onto the rewritten file and committed in the same snapshot
rather than
dropped. `DataEvolutionCompactDeletionVectorRewriter` performs the
migration and
`ConflictDetection` verifies it: a vector that went missing, shrank, or
was left behind
fails the commit instead of resurrecting deleted rows. Both bitmap32 and
bitmap64 vectors
are supported.
- `MaterializeDeletionVectors` is the heavy alternative entry point: it
applies the
deletions physically, which reassigns the row ids of the surviving rows
and therefore
drops the global indexes over the touched partitions in the same commit.
It is never done
automatically, and a range covered by a dedicated blob or vector-store
file is rejected
rather than corrupted. Its commit uses the new
`RowIdCheckConflictForMaterializeDeletionVectors` range rule, since a
rewrite of whole
row ranges cannot rely on the narrower column-overlap check.
- Both `RunAndCommit` and `MaterializeDeletionVectors` split the row id
space into bounded
rounds and commit each round on its own, so only one round's file metadata
is live at a
time. The split is a soft target: a cut can only land where one data
manifest's row id
coverage ends before the next one's begins, and a snapshot without usable
row id
statistics falls back to a single round. Rounds are independent — a failed
round leaves
the committed ones committed and a later call re-plans the rest.
- Options that decide the compaction path or the physical layout (row
tracking, data
evolution, deletion vectors, bucket, blob layout) are now rejected when an
override would
contradict the persisted schema, and the legacy
`data-evolution.compaction.rewrite-row-ids=true` mode fails instead of
being ignored.
### Tests
**Managed BLOB — unit**
| Case group | Covers |
| --- | --- |
| `PrimaryKeyBlobExternalizerTest` (13) | descriptor replacement, per-field
packs, pack rolling by target size, retract-only batches, `PrepareCommit`
hand-over, seal failure still closing the pack stream, inline descriptor fields
left alone, re-materializing descriptor input, IO failure |
| `ManagedBlobReferenceFileTest` (14) | round trip with sort + dedup, empty
list, non-ASCII, **Java golden bytes**, and rejection of corruption, trailing
bytes, bad magic, unsupported version, negative/oversized count, malformed
surrogate, nested relative path |
| `ManagedBlobReferenceCollectorTest` (8) | collecting only declared
columns, ignoring non-descriptor values, empty sidecar, abort deleting the
sidecar, use-after-close and unknown-field rejection |
| `ManagedBlobResolvingBatchReaderTest` (7) | descriptor resolution
preserving nulls, declared-columns-only, pass-through without managed fields,
missing pack / non-struct / wrong-type / non-descriptor rejection |
| `UncommittedFileCleanerTest` (4) | rolling back data files, sidecars and
*owned* packs; not touching an inherited pack; tolerating missing files; one
unusable message not stranding the rest |
| `SchemaValidationTest.TestPrimaryKeyManagedBlob{,SequenceGroups}` |
merge-engine, key, ordering and unsupported-option rules |
| `SingleFileWriterTest.TestAbortExecutorRemovesCompanionFiles` | an abort
executor removes companion files and stays repeatable |
**Managed BLOB — integration (`PkBlobTableInteTest`, 10 cases)**
`TestWriteAndReadManagedBlob`, `TestReadWithPrefetchAndReadAheadCache`,
`TestCompactionRebuildsExactBlobReferences`,
`TestFirstRowManagedBlobKeepsFirstValue`,
`TestPartialUpdateManagedBlob`, `TestDeleteDropsRow`,
`TestSnapshotExpirationRemovesSidecar`,
`TestAbortDeletesTheManagedBlobPacksItRollsBack`,
`TestAbortOfACompactionKeepsTheHistoricalPacks`,
`TestAbortOfAWriteAndCompactionKeepsOnlyTheHistoricalPacks`.
Plus `CleanInteTest.TestOrphanFilesCleanKeepsCompanionFilesOfLiveDataFiles`.
**Compaction — unit**
| Case group | Covers |
| --- | --- |
| `DataEvolutionCompactPlannerTest` (19) | bin packing by target size and
open-file cost, row-id gaps and large files cutting a bin, evolved groups
packing together, mismatched group ranges / vector-store files rejected, blob
files excluded, per-partition planning, `min-file-num`, and the round windowing
(cut only at coverage gaps, file budget, fallback without row id statistics,
delete-only manifests ignored) |
| `DataEvolutionNormalCompactTaskTest` (3),
`DataEvolutionMaterializeDeletionCompactTaskTest` (5) | input union,
discontiguous / misaligned / dedicated-storage rejection |
| `DataEvolutionCompactDeletionVectorRewriterTest` (6) | short-circuit
without vectors, and rejection of a message that already changed the index, a
bucketed message, a rewrite that moved its rows, several normal outputs |
| `DataEvolutionCompactGlobalIndexDropperTest` (2) | nothing dropped for a
normal compaction or without replaced normal files |
| `DataEvolutionConflictDetectionTest` (10) | the migration check — a
left-behind vector, lost deletions, a shrunken vector, two index files claiming
one data file, an uncovered dropped group; accepting a materialized compaction
and a fully deleted range |
| `ConflictDetectionTest` (+2), `AppendCompactCoordinatorTest` (+2) | plain
append table still refuses dropping data files; row-id existence on compaction;
`RunAndCommit` / `MaterializeDeletionVectors` on a plain append table |
| `RowIdRangeConflictCheckerTest` (2),
`MaterializedIndexChangesProviderTest` (3) | range overlap rule; global indexes
rescanned every attempt, other buckets kept |
**Compaction — integration (`DataEvolutionTableTest`, 33 new cases)**
Field-group compaction (`TestCompactAcrossEvolvedFieldGroups`, with
partitions, with a
partition filter, across schema evolution, after a dropped column),
deletion-vector
migration (`TestCompactKeepsDeletionVectorsOfOneRowRangeGroup`, bitmap64,
shifts across
groups, only the touched index file rewritten, sibling vectors carried,
out-of-group
deletion rejected), materialization
(`TestMaterializeDeletionVectorsReassignsRowIds`,
no-op without deletions, untouched ranges left alone, global indexes
dropped, other
partitions untouched, fully deleted range, option guards), bounded rounds
(`TestRunAndCommitSplitsRowIdSpaceIntoRounds`, with a partition filter,
moving deletion
vectors every round for both bitmap kinds), and concurrency
(`TestStaleCompactMessagePreservesConcurrentPartialUpdate`,
`TestSmallFileCompactConflictsWithConcurrentPartialUpdate`,
`TestCompactKeepsConcurrentAppendForNextSmallFileMerge`).
Plus `PkCompactionInteTest.DeduplicateWith{,Bitmap64}DeletionVectors`.
### API and Format
Yes — additive public API, plus one new on-disk file kind.
- `include/paimon/append/append_compact_coordinator.h`: two new static entry
points,
`RunAndCommit` and `MaterializeDeletionVectors`, and
`kDefaultCandidateFilesPerRound`.
`Run` keeps its signature; its contract is documented more tightly — on a
data-evolution
table with deletion vectors the returned vector also holds index-only
messages and the
caller must commit the whole vector in one commit.
- `include/paimon/file_store_commit.h`: new pure virtual
`RowIdCheckConflictForMaterializeDeletionVectors`. This is a
source-breaking change for
an out-of-tree implementer of the interface; there is none in this
repository.
- `include/paimon/format/format_writer.h`: new `LastPayloadRange()` with a
`std::nullopt`
default, so the externalizer can point a descriptor at the bytes it just
wrote without
downcasting the writer the format factory handed it.
- `include/paimon/defs.h`: adds `BLOB_COPY_BUFFER_SIZE`,
`BLOB_DESCRIPTOR_SOURCE_TABLE`,
`PK_CLUSTERING_OVERRIDE` and `DATA_EVOLUTION_COMPACTION_REWRITE_ROW_IDS`
(the last three
documenting rejected configurations), and updates the
`DELETION_VECTORS_ENABLED` and
`DELETION_VECTOR_BITMAP64` notes now that a data-evolution table is
compactable and
bitmap64 is supported.
- **Format**: the `.managed.blob` pack is the existing blob file format. New
is the
`.blobref` sidecar, a versioned binary file with magic, version and a
length-prefixed
reference list; `ManagedBlobReferenceFileTest.TestJavaGoldenBytes` pins it
against the
Java bytes. The sidecar is referenced from `DataFileMeta::extra_files`, an
existing field,
so the manifest format is unchanged. Deletion vector index files,
snapshots and manifests
are unchanged.
### Documentation
Yes.
- `docs/source/user_guide/primary_key_table.rst` gains *Managed BLOB
Storage*: how the
payload is externalized and resolved, the `.blobref` sidecar, pack
ownership versus
reference, and the supported surface (top-level scalar BLOB columns only)
with the option
and merge-engine rules.
- `docs/source/user_guide/compaction.rst` gains *Data-Evolution Table
Compaction*, covering
evolved field groups, the deletion-vector migration and the commit check
that verifies it,
*Materializing Deletions* and why it is a separate opt-in, and *Bounded
Rounds and
Committing*. The previous note that such a table is never compacted is
removed.
- `docs/source/user_guide/read.rst` is updated where it stated the same
limitation.
### Generative AI tooling
Generated-by: Claude Opus 5 (1M context)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
