lxy-9602 opened a new pull request, #199:
URL: https://github.com/apache/paimon-cpp/pull/199
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>):
<description> -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: #158
This PR extends the real-time append read path introduced in #163 to support
separate scan and
reader creation stages.
The main changes are:
- capture immutable memory views during scan planning and register them in
`RealtimeContext`;
- add version, snapshot id, partition-bucket, committed offset, memory upper
offset, and an opaque
read-view ticket to `RealtimeSplit`;
- allow `ReadContext` to share the same `RealtimeContext` used by the writer
and scanner;
- resolve the ticket when creating a reader and validate that its
partition-bucket and offset range
match the split;
- use `RealtimeReader` to keep the memory view alive while disk and memory
data are being read;
- release abandoned or completed read views through a background release
queue, avoiding large
segment destruction on the query thread;
- preserve an already-created reader across commit, refresh, and
memory-segment reclamation.
A real-time split ticket is currently single-use. Once reader creation
consumes the ticket, retrying
the same reader task requires creating a new scan plan and `RealtimeSplit`.
Tickets are also
process-local and must be resolved by the `RealtimeContext` that created
them.
<!-- What is the purpose of the change -->
### Tests
Added or updated coverage for:
- pinning, resolving, and releasing a read-view ticket;
- expiration and cleanup of abandoned read-view tickets;
- rejection when the same `RealtimeSplit` ticket is reused;
- reading from a reader created before commit and refresh after its memory
segment is reclaimed;
- reading a pinned plan after writer close;
- updated Arrow memory-indexer reader ownership.
<!-- List UT and IT cases to verify this change -->
### API and Format
This PR changes public APIs:
- adds `ReadContextBuilder::WithRealtimeContext`;
- adds read-view ticket lifecycle APIs to `RealtimeContext`;
- adds `RealtimeReader`, which keeps a `MemReadView` alive for the reader
lifetime;
- changes `MemIndexer::CreateQueryReaders` to return `RealtimeReader`
instances.
There is no change to the Paimon table storage format, data-file format,
snapshot format, or commit
protocol.
`RealtimeSplit` carries versioned real-time metadata, but cross-process
split serialization is not
implemented in this PR. Its opaque ticket is currently resolved only inside
the originating process.
<!-- Does this change affect API in include dir or storage format or
protocol -->
### Documentation
<!-- Does this change introduce a new feature -->
### Generative AI tooling
Generated-by: OpenAI Codex (GPT-5)
<!--
If generative AI tooling has been used in the process of authoring this
patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
