Guosmilesmile commented on PR #17280:
URL: https://github.com/apache/iceberg/pull/17280#issuecomment-5693155806
@talatuyarer Thanks a lot for the detailed feedback.
My initial thought was to first align on the overall direction, and then
break the implementation into smaller PRs for further review.
For the roadmap, I was thinking about:
1. **Memory-only full cache**
2. **Reload support**
- Whether we can directly reuse Flink's `CacheReloadTrigger` still needs
to be investigated, since `FullCachingLookupProvider` currently only supports
the legacy `InputFormat`.
3. **A disk-backed backend**
- A disk-spilling store or RocksDB
4. **Catch-up support for append-only tables**
5. **FLIP-462 slicing for subtask-specific data distribution**
**TBD:**
1. **Async lookup**
For 3, I am still considering the best approach. I think we eventually need
a backend that does not require the entire lookup table to fit in memory.
FLIP-462 helps distribute data across subtasks, but does not solve this problem.
At this stage, we don't have strong requirements for a dedicated
disk-spilling store, so I am more inclined to reuse RocksDB. Paimon has
replaced its RocksDB-based lookup state with `LocalKvDb`, but I don't think
this means disk-backed lookup state is no longer needed. For our relatively
simple lookup join use case, RocksDB should be sufficient, and we may also be
able to reuse the RocksDB integration already available in Flink. Building
another disk-spilling store from scratch would add significant implementation
and maintenance costs.
For **async lookup**, since the lookup data is already loaded in memory or
held by the backend, the lookup itself should be quite fast. So I think we can
evaluate whether async lookup is actually necessary for this use case later.
I would really appreciate more feedback from everyone, especially on 3 and
async lookup. @pvary @talatuyarer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
