GitHub user jerryshao edited a comment on the discussion: Proposal for Integrating Redis Distributed Cache alongside Caffeine for Enhanced Scalability and Consistency
Hi @lzh010817, thanks a lot for your proposal. IMO, I think a distributed cache is quite useful. We now only have the local cache, which will introduce the inconsistency problem when deploying multiple Gravitino nodes as a federation. But the problem you mentioned about the deployment complexity also exists. I would suggest if you can investigate more about different cache solutions, and we can discuss which one is the best fit for. Currently, I can think of 3 options: 1. Using local cache + journals. Journals will record the operations and sync to the different systems consistently. Each node that received the journal can replay and update the cache. 2. Using the distributed cache as you proposed. But using a distributed cache still suffers from the transactional problem; for example, if one node fails to update the cache, then other nodes will get the old data. 3. Using some DB-level cache. Since Gravitino fetches data from DB, if there are some middlewares that can transparently cache the data from DB, then it is simple and easy to leverage. Also loop in @unknowntpo . @unknowntpo has some initial investigations, maybe we can discuss more here. GitHub link: https://github.com/apache/gravitino/discussions/8480#discussioncomment-14362159 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
