Guosmilesmile commented on PR #17280:
URL: https://github.com/apache/iceberg/pull/17280#issuecomment-5694716950
I agree with Peter.
I think we can focus on the heap-based approach first and split it into
three stages:
1. **Memory-only full cache**
2. **Periodic full reload**
3. **Incremental reload via snapshot deltas**
For large tables, we can wait until the second index is available and then
compare different approaches, such as RocksDB, FLIP-462 slicing, and async
lookup.
This should allow us to move forward with the basic lookup join without
being blocked by the large-table optimization.
If everyone agrees with this direction, I’ll start splitting the PR into
smaller ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
