YannByron opened a new pull request, #11000:
URL: https://github.com/apache/paimon/pull/11000

   ### Purpose
   
   A Ray driver can fail after workers have written their files, or after a 
table commit succeeds but before its response is recorded. Restarting the 
complete ingestion job can then repeat expensive episode processing or append 
the same data again. This change adds optional external checkpoints for fixed 
episode groups, durable task/batch identities, and an opt-in append recovery 
check tied to atomic snapshot publication.
   
   The initial scope is one bucket-unaware append table. It provides a reusable 
Python/Ray layer; it does **not** make the existing `PaimonLeRobotWriter` and 
its companion tables transactional.
   
   ### Usage and identity contract
   
   ```python
   from pypaimon.ray import write_episodes
   
   progress = write_episodes(
       "default.frames",
       catalog_options=catalog_options,
       episodes=ordered_episode_records,
       processor=process_task,
       write_id="ingest-2026-11-04",
       input_manifest_id="source-manifest-v1",
       processing_version="features-v3",
       episodes_per_task=20,
       commit_every_tasks=5,
       checkpoint_location="oss://checkpoint-bucket/pipeline",
       checkpoint_options=checkpoint_options,
       resume=False,
       concurrency=8,
   )
   ```
   
   - Each episode record has a unique nonempty `episode_id` and optional JSON 
`input`. `processor(task)` receives `task_id`, `episode_ids`, and the grouped 
original `episodes`; it returns an Arrow Table, RecordBatch, or iterable of 
either.
   - `episodes_per_task=N` groups N consecutive episodes from a fixed ordered 
input plan. The tail can be smaller. N, the ordered inputs, source 
manifest/version, target identity and processing version must stay unchanged 
under one `write_id`.
   - Logical task IDs depend on fixed inputs/ranges, not Ray's runtime task ID, 
worker, or completion order. A new attempt gets a distinct identity; at most 
one prepared result is selected per logical task.
   - `commit_every_tasks=M` publishes M complete task results per batch, 
followed by a tail. N and M are independent; gathering more results does not 
merge their files. M and Ray resources/concurrency can change on resume for 
results not already assigned to a sealed batch.
   - First use is explicit `resume=False`; an existing checkpoint session is 
rejected. Restart with `resume=True` and the same write identity/input plan. 
Missing or incompatible checkpoints fail rather than silently starting another 
append.
   - Without `checkpoint_location`, every invocation processes and appends the 
complete input. There is no durable skip/resume or cross-invocation 
deduplication guarantee. `resume=True` requires checkpoints.
   - Checkpoints must be explicitly configured **outside the target table 
path**. `checkpoint_options` supply independent FileIO settings to driver and 
workers; table credentials are not implicitly reused.
   - Resume must receive the complete original episode list; do not slice it at 
`resume_position`. The lower-level `RecoverableWriteSession` supports custom 
scheduling through `prepare_task` / `write_task` / `accept_task` / 
`accept_task_id`, `commit_ready`, `recover`, `finish`, and `progress`.
   
   ### Publication and recovery
   
   Workers persist prepared CommitMessages before acknowledging completion. 
Coordinators conditionally create immutable batch plans, then commit the 
selected messages with their write/batch identity. Recovery checks that 
identity against the same snapshot base used for each atomic commit attempt, 
including retries that overlap another coordinator.
   
   | State | Action |
   | --- | --- |
   | No complete task checkpoint | Recompute that task. |
   | Prepared files/checkpoint, not yet committed | Reuse files and submit the 
original prepared result. |
   | Sealed batch with uncertain outcome | Resolve the original publication 
before reusing its members. |
   | Commit succeeded but response/receipt was lost | Recover its snapshot 
result; do not append again. |
   | Durable completion receipt | Skip its tasks. |
   
   Progress returns `committed`, `prepared`, and `pending` task/episode lists. 
Business completion is derived from the fixed task-to-episode mapping and 
confirmed batch publication, not from the existence of an arbitrary checkpoint 
file. `resume_position` is the first hole after the contiguous committed input 
prefix, not the maximum completed episode ID. The driver filters work before 
scheduling, so applications need not query the data table or issue a remote 
checkpoint lookup per episode/frame.
   
   ### Supported modes and restrictions
   
   1. **One table, append only.** Full-row bucket-unaware appends, including 
supported Data Evolution data, are in scope. This entry point uses the Python 
write/commit executors even when the table enables native execution; native 
commit recovery is not implemented. No primary-key upserts, overwrites, partial 
updates or cross-table atomicity. Existing `write_paimon` and 
`PaimonLeRobotWriter` behavior is unchanged; managed LeRobot table groups are 
rejected by this recovery entry point. Counting N episode descriptors here is 
not intercepting N `save_episode()` calls on that writer. Its 
frames/episodes/tasks/info/stats commits need a separate group publication 
protocol.
   2. **Business identity is supplied by the caller.** Input source versions 
and processing/model/configuration versions must be stable. A mutable OSS path 
alone is insufficient. Different `write_id` values do not deduplicate 
overlapping episodes. Fixed task identity is not a primary-key uniqueness 
constraint.
   3. **Processor side effects remain at-least-once.** Processing may repeat 
before a complete task checkpoint is durable. Writes to other databases, media 
encoding, notifications and other external effects are not covered by table 
commit idempotence.
   4. **Checkpoint lifecycle belongs to the application.** No automatic 
checkpoint deletion or TTL, even after commit. All checkpoint records, 
including detailed prepared messages, batch plans and completion receipts, 
remain for manual cleanup. Retain them for the required recovery/replay window. 
Manually delete only after all related writers/coordinators stop and recovery 
or replay is no longer needed. A later resume fails if checkpoints were 
removed; an intentional fresh write processes the full input again.
   5. **Safe storage primitives are required.** Add an explicit 
create-if-absent FileIO primitive. Local publication uses a complete temporary 
file plus a hard link; OSS uses conditional PUT and requires versioning 
disabled and permission to inspect that state. Enabled/suspended versioning and 
unsupported providers fail closed. Do not change bucket versioning during a 
write. No unsafe rename fallback is used for checkpoint publication.
   6. **Preserve unresolved history and files.** Retain snapshot anchors and 
intervening history needed to determine uncertain publication. Missing history 
fails closed. Keep prepared data files and all sidecars intact and pause 
orphan-file cleanup during the recovery window; external maintenance engines do 
not automatically honor these checkpoints. Losing task attempts can leave 
unselected files for later cleanup.
   7. **No structural changes during recovery.** Do not change schema, recreate 
or roll back the table, or switch branches while the session can resume. Do not 
run legacy filesystem rename-based writers concurrently with recoverable 
writes: they can overwrite a winning snapshot across processes. Recoverable 
writers use conditional creation and can compete safely with each other. REST 
catalogs must enforce atomic snapshot compare-and-set on the server; this 
client cannot supply that server-side guarantee.
   8. **Cloud/runtime scope.** Every worker needs matching dependencies and 
access to the catalog, data and checkpoint stores. Local paths are only 
appropriate when all participating processes share them. OSS condition handling 
is tested with the SDK against a local HTTP service; live OSS/DLF validation 
must not be inferred from those tests.
   
   See `docs/docs/pypaimon/recoverable-writes.md` for the full usage and 
recovery examples.
   
   ### Tests
   
   Based on Apache `master` at `f9629393b911634ab490a8cc955a17d7659e030e`. The 
branch contains only this recovery change; preceding video-frame changes from 
the original development branch are excluded.
   
   Python 3.13.12, PyArrow 19.0.1, Vortex 0.70.0. Non-Ray regression and 
execution-boundary suite: **416 passed, 16 skipped**. Skips: 14 require a live 
OSS test bucket, one requires Windows paths, and one requires pypaimon-rust.
   
   ```bash
   paimon-python/.venv313/bin/python -m pytest -q -rs \
     
paimon-python/pypaimon/tests/{recoverable_write,recoverable_write_fault,recoverable_commit,conditional_file_create,oss_atomic_write,file_io,oss_file_io,caching_file_io,resolving_file_io,file_store_commit,table_commit,overwrite_commit_conflict,commit_message_serializer,multimodal_lerobot_writer,data_evolution,public_execution_boundary,execution_layout,shared_execution_import}_test.py
 \
     paimon-python/pypaimon/tests/rest/rest_token_file_io_test.py \
     
paimon-python/pypaimon/tests/write/{commit_callback,commit_execution_protocol,backend_boundary}_test.py
   ```
   
   Ray **2.59.1** (within master's `ray<2.60` requirement): new recovery suite 
and existing Ray sink suite **27 passed**. Combined result: **443 passed, 16 
skipped**.
   
   ```bash
   paimon-python/.venv313/bin/python -m pytest -q \
     paimon-python/pypaimon/tests/ray_recoverable_write_test.py \
     paimon-python/pypaimon/tests/ray_sink_test.py
   ```
   
   These tests run actual local Ray workers and inject worker failure, driver 
failure before publication, and failure after publication before 
acknowledgement. Recovery reuses prepared files and skips committed work.
   
   Coverage includes fixed grouping and threshold/tail publication; zero-row 
task completion; out-of-order episode progress; immutable plan validation; 
changed M on resume; external checkpoint path checks and retention; 
prepared-result reuse; commit acknowledgement loss; two-process publication 
races; expired history; missing/truncated data files, sidecars and changelogs; 
checkpoint corruption; no-checkpoint repeat append; and existing 
writer/commit/FileIO/LeRobot regressions. Public recovery imports do not 
eagerly load either execution backend.
   
   Changed Python files pass repository-configured flake8. `git diff --check` 
passes; documentation Python snippets parse successfully. Live OSS/DLF, REST 
server CAS and a multi-machine Ray deployment have not been validated.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to