Thanks folks. I updated my draft PR and marked it ready fo review at https://github.com/apache/polaris/pull/2508 . After earlier comments by Dennis, I included implementations in the InMemoryEntityCache that match the PolarisMetaStoreManager APIs. This allows callers to use either the EntityCache or the PolarisMetaStoreManager APIs and get the same result. My hope is to one day hide the EntityCache entirely so that callers never have to deal with the cache directly, but that we can make all persistence calls through the one persistence interface. PTAL and let me know what you think.
Mike On Tue, Sep 23, 2025 at 5:21 PM Yufei Gu <[email protected]> wrote: > Hi Mike, big +1 on the direction. Looking forward to the PR. > > Yufei > > > On Mon, Sep 22, 2025 at 10:26 AM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi Michael, > > > > This sounds like a reasonable proposal to me. Looking forward to the PR > :) > > > > Cheers, > > Dmitri. > > > > On Fri, Sep 19, 2025 at 2:26 PM Michael Collado <[email protected]> > > wrote: > > > > > Hey folks > > > > > > Building on the loadEntities API added in > > > https://github.com/apache/polaris/pull/2290 , I'd like to propose two > > new > > > APIs to support batch loading of entities in the > PolarisMetaStoreManager. > > > The new loadEntities API is great for avoiding the N+1 queries problem > > > during lists and I'd like to extend that behavior for batch retrieving > > > entities outside of list operations. > > > > > > The two APIs would be > > > > > > ResolvedEntitiesResult loadResolvedEntities( > > > @Nonnull PolarisCallContext callCtx, > > > @Nonnull List<EntityNameLookupRecord> entityLookupRecords); > > > > > > and > > > > > > ResolvedEntitiesResult loadResolvedEntities( > > > @Nonnull PolarisCallContext callCtx, > > > @Nonnull PolarisEntityType entityType, > > > @Nonnull List<PolarisEntityId> entityIds); > > > > > > The first API is useful for batch loading entities after a list > > operation, > > > but may only require loading a subset of entities (e.g., because a > subset > > > is already in the in-memory cache). The latter supports a batch version > > of > > > the loadEntity - also useful for loading entities for the cache without > > the > > > need to refresh one by one (consider the Resolver, which does this). > > > > > > Notably, both APIs return ResolvedEntitiesResult, rather than > > > EntitiesResult because I'd like to resolve the discrepancy between the > > > EntityCache API and the PolarisMetaStoreManager API. Right now, it's > > > important to directly interact with the cache whereas, if we can > > > consolidate the API, it could be possible to support a cache-backed > > > PolarisMetaStoreManager implementation that manages the in-memory cache > > > invisibly. > > > > > > I already have a draft PR with some of the work done, but if everyone > > > agrees on the additions, I can have a PR ready for review next week > some > > > time. Let me know your thoughts. > > > > > > Mike > > > > > >
