RyanLin5967 opened a new issue, #10884: URL: https://github.com/apache/paimon/issues/10884
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version pypaimon 2.1.0, and master, which prints the same. ### Compute Engine Python API (pypaimon 2.1.0, Python 3.12.16, pyarrow 19.0.1) on a local filesystem catalog. Spark 3.5.6 with paimon-spark-3.5_2.12 2.1.0 for the comparison. ### Minimal reproduce step Currently, pypaimon checks `scan.fallback-branch` only in the catalog options and never reads it from the table, so a batch read leaves out the partitions that exist only in the fallback branch. Spark's `SELECT k, p FROM paimon.db.t` on the same table returns `(1, 'a')` and `(2, 'b')`. ```python import pyarrow as pa from pypaimon import CatalogFactory, Schema from pypaimon.schema.schema_change import SchemaChange cat = CatalogFactory.create({"warehouse": "wh"}) cat.create_database("db", True) sch = pa.schema([("k", pa.int64()), ("p", pa.string())]) cat.create_table("db.t", Schema.from_pyarrow_schema(sch, partition_keys=["p"]), False) cat.create_branch("db.t", "b") def write(name, k, p): wb = cat.get_table(name).new_batch_write_builder() w, c = wb.new_write(), wb.new_commit() w.write_arrow(pa.table({"k": [k], "p": [p]}, schema=sch)) c.commit(w.prepare_commit()) write("db.t", 1, "a") write("db.t$branch_b", 2, "b") cat.alter_table("db.t", [SchemaChange.set_option("scan.fallback-branch", "b")]) rb = cat.get_table("db.t").new_read_builder() print(rb.new_read().to_arrow(rb.new_scan().plan().splits()).to_pylist()) # [{'k': 1, 'p': 'a'}], should also have {'k': 2, 'p': 'b'} # partition p=b does not exist on main, so it should be read from the fallback branch b ``` ### What doesn't meet your expectations? The read returns `(1, 'a')` from main and `(2, 'b')` from the fallback branch `b`, as Spark does. ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
