Stephen0421 commented on PR #9099: URL: https://github.com/apache/paimon/pull/9099#issuecomment-5238841610
Hi @JingsongLi — following your suggestion to split #9099 into smaller PRs, here is the updated stacked plan. **PR1 is open for review:** https://github.com/apache/paimon/pull/9148 **Stacking:** `master` ← **PR1 (#9148)** ← PR2 ← PR3 ← PR4 · PR5 branches from PR2 --- ### PR1 — BLOB descriptor parsing & read-path compatibility *(#9148, ready for review)* **Scope:** Shared foundation for all managed-BLOB work. No PK-specific logic. **Changes:** - `blob.py`: split heuristic `from_bytes` (v2 magic only) vs explicit `from_descriptor_bytes` (v1 strict length + v2 deserialize) - `BlobInlineConvertReader`: descriptor fields use `from_descriptor_bytes` - Row path: `descriptor_field_indices` for `blob-as-descriptor` tables (`OffsetRow.get_blob`, read-chain metadata) - `core_options`: legacy `blob.stored-descriptor-fields`; blank `blob-descriptor-field` falls back to legacy - `blob_format_writer`: `EOFError` on truncated external blob copy - `UriReaderFactory` lifecycle + FileIO `close()` wiring **Tests:** `blob_test`, `uri_reader_factory_test` --- ### PR2 — Managed BLOB lifecycle & staged commit *(planned)* **Scope:** Reference-file tracking and commit/abort semantics. No PK write yet. **Key files:** `managed_blob_reference_collector`, `managed_blob_reference_file`, `file_store_write`, `file_store_commit`, `commit_message`, `table_commit`, `table_write` / `write_builder` **Tests:** `managed_blob_lifecycle_test`, `managed_blob_reference_file_test`, `test_file_store_write`, relevant `table_write_test` --- ### PR3 — PK managed BLOB write *(planned)* **Scope:** PK merge/write: externalize inline blobs, wire PR2 lifecycle. **Key files:** `primary_key_blob_externalizer`, `key_value_data_writer`, `data_writer`, `data_vector_writer`, `dedicated_format_writer`, `blob_writer`, `row_key_extractor`, `table_update_by_row_id`, `ray_datasink` **Tests:** `primary_key_blob_externalizer_test`, `managed_blob_write_ownership_test`, `primary_key_blob_inline_compat_test`, `test_write_merge_buffer` --- ### PR4 — PK managed BLOB read & view *(planned)* **Scope:** PK-specific read on top of PR1 parsing. *(PR1 already covers generic v1/v2 descriptor-byte parsing for batch + row paths.)* **Key files:** `managed_blob_convert_record_reader`, `blob_descriptor_reader_factory`, `blob_view_lookup`, `read_builder`, `table_read`, `split_read` (PK wiring), `schema_manager` / `data_types` **Tests:** `blob_table_test`, `primary_key_blob_e2e_test`, `test_primary_key_blob_read_contract`, `deferred_blob_resolve_test`, `catalog_environment_test`, `schema_manager_test` --- ### PR5 — Dynamic bucket HASH commit callback *(planned)* **Scope:** Independent of PR3/PR4. Refresh `base_snapshot_id` after commit for HASH-indexed dynamic-bucket tables. **Key files:** `hash_index_commit_callback`, `dynamic_bucket`, `table_write` callback wiring **Tests:** `dynamic_bucket_test` --- **PR1 review notes (already addressed in #9148):** - `from_bytes` stays v2-only heuristic; v1 only via `from_descriptor_bytes` in descriptor-field context - Descriptor-field malformed bytes → `ValueError` (fail-fast), not silent `BlobData` - `UriReaderFactory.clear_cache()` resets LRU instance (no double-close via `popitem`); closes FileIOs outside write lock PR2–PR5 will be opened sequentially on top of #9148. This PR (#9099) will remain Draft and link to each stacked PR as they land. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
