NoahKusaba commented on PR #2862: URL: https://github.com/apache/iceberg-rust/pull/2862#issuecomment-5062798811
> Something seems a little off here to me. Read/write skew seems like a footgun. A single registered table that reads snapshot A but writes to HEAD is a confusing object. `INSERT INTO t; SELECT * FROM t` won't show the row you just inserted. That's a strange contract to hand a user in my opinion. > > I also think schema isn't being handled here correctly for pinned snapshots, we should resolve against the snapshot's schema rather than current, the static table provider handles this correctly. insert-read problem: Sorry for that, I understand what you are saying on careful review. I was so tunnel visioned on the ballista integration, that I had missed actual users directly calling snapshot_id. The way I have it wired in my Ballista integration is that snapshot_id is resolved by the scheduler when it encodes the physical plan to executors, so a read -> insert -> read would properly show the new data as the snapshot_id would be reset to head by the scheduler as queries sequentially are made. I think it makes sense to add a write guard, if the snapshot_id is pinned (not None), to constrain confusing behavior? I added it in a new commit, let me know how you think this should be handled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
