lxy-9602 opened a new pull request, #163:
URL: https://github.com/apache/paimon-cpp/pull/163
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>):
<description> -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: #158
This PR introduces the first phase of Paimon C++ real-time write support for
fixed-bucket append tables.
Applications can attach a `RealtimeContext` with either the default
Arrow-based `MemIndexer` or a custom implementation. Incoming batches are
buffered by partition and bucket. During `PrepareCommitWithProgress`, the
current segment is sealed and exposed through commit readers, while subsequent
writes continue in a new segment. Paimon still owns data-file generation,
rolling, file formats, file indexes, schema conversion, and commit-message
generation.
Each row receives a monotonically increasing `_OFFSET` scoped to its
partition and bucket. Prepare results carry their inclusive offset ranges
through `RealtimeCommitProgress`. `CommitWithProgress` orders concurrent
prepare results, validates that they form a contiguous committed prefix, and
atomically publishes both data files and updated progress. Progress is stored
in a `metadata/{UUID}.offsets` file referenced by the snapshot and is restored
when creating a new writer.
The feature is opt-in through `WriteContextBuilder::WithRealtimeContext`.
The existing write and commit paths remain unchanged when no real-time context
is configured.
The current implementation supports streaming, fixed-bucket append tables.
Primary-key tables, deletion vectors, and data evolution are not included in
this phase.
Follow-up work will add append-table union reads across committed disk data
and real-time memory, reclaim sealed segments after their offsets are covered
by a committed snapshot, and preserve real-time offset information through
compaction.
<!-- What is the purpose of the change -->
### Tests
Added unit tests for:
- default Arrow `MemIndexer` write, seal, and commit-reader behavior;
- physical `_OFFSET` generation during Paimon data-file writing;
- offset serialization, validation, ordering, persistence, and recovery;
- no-op writer memory management;
- public API validation and unsupported table modes.
Added integration tests for:
- append, commit, and read with `_OFFSET`;
- multiple partitions;
- restoring the next offset from a committed snapshot;
- concurrent write and prepare-commit;
- multiple concurrent prepare-commit threads;
- committing unordered prepare results without losing row order.
<!-- List UT and IT cases to verify this change -->
### API and Format
Yes.
New public APIs include `MemIndexer`, `MemIndexerFactory`,
`RealtimeSegmentHandle`, `RealtimeContext`, `RealtimeWriteBatch`, and
`RealtimeCommitProgress`.
`WriteContextBuilder::WithRealtimeContext`,
`FileStoreWrite::PrepareCommitWithProgress`, and
`FileStoreCommit::CommitWithProgress` are added.
For the opt-in real-time path, data files contain an additional `_OFFSET`
`INT64` field. Per-partition-bucket committed offsets are stored in a versioned
metadata file referenced by the snapshot.
<!-- Does this change affect API in include dir or storage format or
protocol -->
### Documentation
<!-- Does this change introduce a new feature -->
### Generative AI tooling
Generated-by: OpenAI Codex (GPT-5)
<!--
If generative AI tooling has been used in the process of authoring this
patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
