lxy-9602 opened a new pull request, #280:
URL: https://github.com/apache/paimon-cpp/pull/280
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>):
<description> -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: #158
This change consolidates the real-time storage and schema handling for
append-only and primary-key tables, and adds real-time read support for
primary-key tables with deletion vectors.
The main changes are:
- Introduce `RealtimeSchemaLayout` to define the schemas used at each
real-time boundary: user input, store write, store commit, table commit, query
- Use `ArrowRealtimeStore` for both append-only and primary-key modes,
removing the dedicated `PrimaryKeyRealtimeStore`.
- Preserve primary-key stored batches as independent sorted readers for
query and commit.
- Make real-time read views retain sealed segments directly so pinned views
remain valid after memory reclamation.
- Add `FileStoreWrite::Seal()` to seal the current in-memory segment without
flushing it to disk. In a follow-up, this interface will support spilling to
disk and reopening the spilled data with a disk reader for query execution.
- Allow `PrepareCommitWithProgress` to consume multiple previously sealed
segments and publish their combined offset range.
For primary-key tables with deletion vectors:
- Allow deletion-vector-backed lookup in real-time mode.
- Keep committed level-0 files in the real-time scan as the durable tail.
- Push value predicates into level > 0 files only.
- Read level-0 files and in-memory data without predicate pushdown.
- Merge high-level files, level-0 files, and in-memory readers in the final
merge-on-read.
- Route disk-only real-time splits containing level-0 files through the same
level-aware reader path.
- Pack eligible non-overlapping high-level files into raw-convertible split
groups to reduce unnecessary merge-reader overhead.
<!-- What is the purpose of the change -->
### Tests
Added or updated coverage for:
- append-only and primary-key real-time schema layouts
- explicit segment sealing and multi-segment commit
- Arrow-backed primary-key commit/query readers
- nested projection and sliced Arrow batches
- pinned read-view lifetime across memory reclamation
- real-time writer offset and sequence handling
- primary-key tables with deletion vectors
- predicate pushdown across high-level files, level-0 files, and memory
- unfiltered reads after compaction, level-0 commits, and in-memory updates
<!-- List UT and IT cases to verify this change -->
### API and Format
API impact:
- Adds `FileStoreWrite::Seal()`.
- Normalizes the primary-key `RealtimeStore` write schema to:
`[_SEQUENCE_NUMBER, _VALUE_KIND, _REALTIME_OFFSET, user fields]`.
- `RealtimeCommitProgress` may represent one or more sealed segments.
There is no change to the persisted table data, manifest, snapshot,
deletion-vector, or offset-file formats.
Custom `RealtimeStoreFactory` implementations should verify that they handle
the normalized primary-key store schema.
<!-- Does this change affect API in include dir or storage format or
protocol -->
### Documentation
Generated-by: OpenAI Codex (GPT-5)
<!-- Does this change introduce a new feature -->
### Generative AI tooling
<!--
If generative AI tooling has been used in the process of authoring this
patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
