pvary commented on PR #17280: URL: https://github.com/apache/iceberg/pull/17280#issuecomment-5693932726
Agree on splitting the PR. A few concerns on the roadmap: - Per-key scans: Iceberg pays planning and manifest evaluation per lookup, so the range where this wins is narrow. Do we have any numbers? - Full cache per subtask: memory scales with parallelism, not table size. Every subtask also re-reads the whole table on every reload. - [FLIP-462](https://cwiki.apache.org/confluence/spaces/FLINK/pages/309496789/FLIP-462+Support+Custom+Data+Distribution+for+Input+Stream+of+Lookup+Join): The API is one-directional: the connector hands the planner a partitioner, and nothing tells the lookup function which slice it owns, or whether the shuffle was applied at all. That would need split assignment for lookup subtasks, which is a Flink-side change, not an Iceberg follow-up. - All options treat the table scan as the lookup mechanism. If V4 indexes land, per-key scans could become a bit better. IMHO we need to focus on a path which gives the best value on the shortest path. My current thinking is: 1. **Memory-only full cache**: heap store with projection/filter pushdown and snapshot pinning, loaded once, documented as targeting dimension tables that fit comfortably in heap. 2. **Periodic full reload**: refresh on an interval with atomic swap and an explicit failure policy that fails rather than silently serving stale data. 3. **Incremental reload via snapshot deltas**: use Iceberg's incremental scan between the pinned and current snapshot so reload cost scales with the change set instead of the whole table. Then reassess — RocksDB, FLIP-462 slicing, and async lookup all target the doesn't-fit-in-heap case, which V4 indexes may solve better than another layer of cache. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
