GitHub user jerryshao added a comment to the discussion: Proposal for Integrating Redis Distributed Cache alongside Caffeine for Enhanced Scalability and Consistency
Hi @lzh010817 thanks a lot for your proposal. IMO, I think a distributed cache is quite useful. We now only have the local cache, which will introduce the inconsistency problem when deploying multiple Gravitino node as a federation. But the problem you mentioned about the deployment complexity also exists. I would suggest if you can investigate more about different cache solutions, and we can discuss which one is best fit for. Currently, I can think of 3 options: 1. Using local cache + journals. Journals will record the operations and sync to the different system consistently. Each node received the journal can replay and update the cache. 2. Using distributed cache as you proposed. But using distrubted cache still suffer from the tansactional problem, for example, if one node failed to update the cache, then other nodes will get the old data. 3. Using some DB level cache. Since Gravitino is fetch data from DB, if there are some middlewares that can transparent cache the data from DB, then it is simple and easy to leverage. Also loop in @unknowntpo . @unknowntpo has some initial investigations, may be we can discuss more here. GitHub link: https://github.com/apache/gravitino/discussions/8480#discussioncomment-14362159 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
