limatakamaidotcom-github opened a new issue, #18163:
URL: https://github.com/apache/iceberg/issues/18163
### Query engine
Spark
### Question
Hi Team,
When `spark.sql.catalog.spark_catalog.scan-planning-mode=server`, by
default, RESTTableScan's PlanTableScanRequest has no snapshotId set and is up
to the catalog server to pick up the latest snapshotId. In theory, between
Spark’s datasource schema binding on client side and remote table scan on the
server side, table schema can change and new data can be ingested into the
table. The server can return files written with new schema while the client
tries to read them with old schema. Is there a potential issue regarding schema
mismatch as the result of different snapshotIds are used by the client and the
server?
The source of the behavior:
Compared with the local mode, DataTableScan.doPlanFiles() invokes the public
version of SnapshotScan.snapshot(), which picks snapshotId via
table().currentSnapshot() if snapshotId is not provided in the context, so
schema binding and table scanning are based on the same snapshotId in the Table
object.
public Snapshot snapshot() {
return snapshotId() != null ? table().snapshot(snapshotId()) :
table().currentSnapshot();
}
In contrast, RESTTableScan.planFiles() invokes the protected version of
SnapshotScan.snapshot(), which returns null if not provided in the context.
protected Long snapshotId() {
return context().snapshotId();
}
I wonder if the diverged behavior is intentional? If yes, for what reason?
Thanks,
Limin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
