henriquejsza opened a new pull request, #10898:
URL: https://github.com/apache/paimon/pull/10898
### Purpose
Closes #10884.
pypaimon only checked `scan.fallback-branch` in the catalog options, where
it is rejected, and never read it from the table. A batch read therefore
left out the partitions that exist only in the fallback branch, while
Spark's `SELECT ... FROM paimon.db.t` on the same table returns them.
`FileStoreTable.new_read_builder` now returns a `FallbackReadBuilder` when
the table option `scan.fallback-branch` is set and the current branch is not
the fallback branch itself. Mirroring Java's `FallbackReadFileStoreTable`,
partitions that exist in the main branch are read from main, and partitions
that exist only in the fallback branch are read from the fallback branch.
Filter, projection, read type and limit are applied to both branch scans,
and the fallback branch must have the same row type as main (same check Java
performs). All branches of a table share the same physical data files (only
snapshot, tag and schema metadata are branch scoped), so the merged splits
are read with a single `TableRead`.
### Tests
- Added `pypaimon/tests/batch_read_fallback_branch_test.py`:
- `test_batch_read_includes_fallback_only_partitions` reproduces the issue
(a partition missing from main is read from the fallback branch).
- `test_main_partition_shadows_fallback` asserts a partition present in
main
is read from main, not the fallback.
- `test_no_fallback_option_reads_main_only` asserts the read stays on main
when the option is unset.
- `flake8 --config=./dev/cfg.ini` passes on the changed files, and the
license
header check passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
