YannByron opened a new pull request, #11000:
URL: https://github.com/apache/paimon/pull/11000
### Purpose
A Ray driver can fail after workers have written their files, or after a
table commit succeeds but before its response is recorded. Restarting the
complete ingestion job can then repeat expensive episode processing or append
the same data again. This change adds optional external checkpoints for fixed
episode groups, durable task/batch identities, and an opt-in append recovery
check tied to atomic snapshot publication.
The initial scope is one bucket-unaware append table. It provides a reusable
Python/Ray layer; it does **not** make the existing `PaimonLeRobotWriter` and
its companion tables transactional.
### Usage and identity contract
```python
from pypaimon.ray import write_episodes
progress = write_episodes(
"default.frames",
catalog_options=catalog_options,
episodes=ordered_episode_records,
processor=process_task,
write_id="ingest-2026-11-04",
input_manifest_id="source-manifest-v1",
processing_version="features-v3",
episodes_per_task=20,
commit_every_tasks=5,
checkpoint_location="oss://checkpoint-bucket/pipeline",
checkpoint_options=checkpoint_options,
resume=False,
concurrency=8,
)
```
- Each episode record has a unique nonempty `episode_id` and optional JSON
`input`. `processor(task)` receives `task_id`, `episode_ids`, and the grouped
original `episodes`; it returns an Arrow Table, RecordBatch, or iterable of
either.
- `episodes_per_task=N` groups N consecutive episodes from a fixed ordered
input plan. The tail can be smaller. N, the ordered inputs, source
manifest/version, target identity and processing version must stay unchanged
under one `write_id`.
- Logical task IDs depend on fixed inputs/ranges, not Ray's runtime task ID,
worker, or completion order. A new attempt gets a distinct identity; at most
one prepared result is selected per logical task.
- `commit_every_tasks=M` publishes M complete task results per batch,
followed by a tail. N and M are independent; gathering more results does not
merge their files. M and Ray resources/concurrency can change on resume for
results not already assigned to a sealed batch.
- First use is explicit `resume=False`; an existing checkpoint session is
rejected. Restart with `resume=True` and the same write identity/input plan.
Missing or incompatible checkpoints fail rather than silently starting another
append.
- Without `checkpoint_location`, every invocation processes and appends the
complete input. There is no durable skip/resume or cross-invocation
deduplication guarantee. `resume=True` requires checkpoints.
- Checkpoints must be explicitly configured **outside the target table
path**. `checkpoint_options` supply independent FileIO settings to driver and
workers; table credentials are not implicitly reused.
- Resume must receive the complete original episode list; do not slice it at
`resume_position`. The lower-level `RecoverableWriteSession` supports custom
scheduling through `prepare_task` / `write_task` / `accept_task` /
`accept_task_id`, `commit_ready`, `recover`, `finish`, and `progress`.
### Publication and recovery
Workers persist prepared CommitMessages before acknowledging completion.
Coordinators conditionally create immutable batch plans, then commit the
selected messages with their write/batch identity. Recovery checks that
identity against the same snapshot base used for each atomic commit attempt,
including retries that overlap another coordinator.
| State | Action |
| --- | --- |
| No complete task checkpoint | Recompute that task. |
| Prepared files/checkpoint, not yet committed | Reuse files and submit the
original prepared result. |
| Sealed batch with uncertain outcome | Resolve the original publication
before reusing its members. |
| Commit succeeded but response/receipt was lost | Recover its snapshot
result; do not append again. |
| Durable completion receipt | Skip its tasks. |
Progress returns `committed`, `prepared`, and `pending` task/episode lists.
Business completion is derived from the fixed task-to-episode mapping and
confirmed batch publication, not from the existence of an arbitrary checkpoint
file. `resume_position` is the first hole after the contiguous committed input
prefix, not the maximum completed episode ID. The driver filters work before
scheduling, so applications need not query the data table or issue a remote
checkpoint lookup per episode/frame.
### Supported modes and restrictions
1. **One table, append only.** Full-row bucket-unaware appends, including
supported Data Evolution data, are in scope. This entry point uses the Python
write/commit executors even when the table enables native execution; native
commit recovery is not implemented. No primary-key upserts, overwrites, partial
updates or cross-table atomicity. Existing `write_paimon` and
`PaimonLeRobotWriter` behavior is unchanged; managed LeRobot table groups are
rejected by this recovery entry point. Counting N episode descriptors here is
not intercepting N `save_episode()` calls on that writer. Its
frames/episodes/tasks/info/stats commits need a separate group publication
protocol.
2. **Business identity is supplied by the caller.** Input source versions
and processing/model/configuration versions must be stable. A mutable OSS path
alone is insufficient. Different `write_id` values do not deduplicate
overlapping episodes. Fixed task identity is not a primary-key uniqueness
constraint.
3. **Processor side effects remain at-least-once.** Processing may repeat
before a complete task checkpoint is durable. Writes to other databases, media
encoding, notifications and other external effects are not covered by table
commit idempotence.
4. **Checkpoint lifecycle belongs to the application.** No automatic
checkpoint deletion or TTL, even after commit. All checkpoint records,
including detailed prepared messages, batch plans and completion receipts,
remain for manual cleanup. Retain them for the required recovery/replay window.
Manually delete only after all related writers/coordinators stop and recovery
or replay is no longer needed. A later resume fails if checkpoints were
removed; an intentional fresh write processes the full input again.
5. **Safe storage primitives are required.** Add an explicit
create-if-absent FileIO primitive. Local publication uses a complete temporary
file plus a hard link; OSS uses conditional PUT and requires versioning
disabled and permission to inspect that state. Enabled/suspended versioning and
unsupported providers fail closed. Do not change bucket versioning during a
write. No unsafe rename fallback is used for checkpoint publication.
6. **Preserve unresolved history and files.** Retain snapshot anchors and
intervening history needed to determine uncertain publication. Missing history
fails closed. Keep prepared data files and all sidecars intact and pause
orphan-file cleanup during the recovery window; external maintenance engines do
not automatically honor these checkpoints. Losing task attempts can leave
unselected files for later cleanup.
7. **No structural changes during recovery.** Do not change schema, recreate
or roll back the table, or switch branches while the session can resume. Do not
run legacy filesystem rename-based writers concurrently with recoverable
writes: they can overwrite a winning snapshot across processes. Recoverable
writers use conditional creation and can compete safely with each other. REST
catalogs must enforce atomic snapshot compare-and-set on the server; this
client cannot supply that server-side guarantee.
8. **Cloud/runtime scope.** Every worker needs matching dependencies and
access to the catalog, data and checkpoint stores. Local paths are only
appropriate when all participating processes share them. OSS condition handling
is tested with the SDK against a local HTTP service; live OSS/DLF validation
must not be inferred from those tests.
See `docs/docs/pypaimon/recoverable-writes.md` for the full usage and
recovery examples.
### Tests
Based on Apache `master` at `f9629393b911634ab490a8cc955a17d7659e030e`. The
branch contains only this recovery change; preceding video-frame changes from
the original development branch are excluded.
Python 3.13.12, PyArrow 19.0.1, Vortex 0.70.0. Non-Ray regression and
execution-boundary suite: **416 passed, 16 skipped**. Skips: 14 require a live
OSS test bucket, one requires Windows paths, and one requires pypaimon-rust.
```bash
paimon-python/.venv313/bin/python -m pytest -q -rs \
paimon-python/pypaimon/tests/{recoverable_write,recoverable_write_fault,recoverable_commit,conditional_file_create,oss_atomic_write,file_io,oss_file_io,caching_file_io,resolving_file_io,file_store_commit,table_commit,overwrite_commit_conflict,commit_message_serializer,multimodal_lerobot_writer,data_evolution,public_execution_boundary,execution_layout,shared_execution_import}_test.py
\
paimon-python/pypaimon/tests/rest/rest_token_file_io_test.py \
paimon-python/pypaimon/tests/write/{commit_callback,commit_execution_protocol,backend_boundary}_test.py
```
Ray **2.59.1** (within master's `ray<2.60` requirement): new recovery suite
and existing Ray sink suite **27 passed**. Combined result: **443 passed, 16
skipped**.
```bash
paimon-python/.venv313/bin/python -m pytest -q \
paimon-python/pypaimon/tests/ray_recoverable_write_test.py \
paimon-python/pypaimon/tests/ray_sink_test.py
```
These tests run actual local Ray workers and inject worker failure, driver
failure before publication, and failure after publication before
acknowledgement. Recovery reuses prepared files and skips committed work.
Coverage includes fixed grouping and threshold/tail publication; zero-row
task completion; out-of-order episode progress; immutable plan validation;
changed M on resume; external checkpoint path checks and retention;
prepared-result reuse; commit acknowledgement loss; two-process publication
races; expired history; missing/truncated data files, sidecars and changelogs;
checkpoint corruption; no-checkpoint repeat append; and existing
writer/commit/FileIO/LeRobot regressions. Public recovery imports do not
eagerly load either execution backend.
Changed Python files pass repository-configured flake8. `git diff --check`
passes; documentation Python snippets parse successfully. Live OSS/DLF, REST
server CAS and a multi-machine Ray deployment have not been validated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]