Hi Mike, big +1 on the direction. Looking forward to the PR. Yufei
On Mon, Sep 22, 2025 at 10:26 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Michael, > > This sounds like a reasonable proposal to me. Looking forward to the PR :) > > Cheers, > Dmitri. > > On Fri, Sep 19, 2025 at 2:26 PM Michael Collado <[email protected]> > wrote: > > > Hey folks > > > > Building on the loadEntities API added in > > https://github.com/apache/polaris/pull/2290 , I'd like to propose two > new > > APIs to support batch loading of entities in the PolarisMetaStoreManager. > > The new loadEntities API is great for avoiding the N+1 queries problem > > during lists and I'd like to extend that behavior for batch retrieving > > entities outside of list operations. > > > > The two APIs would be > > > > ResolvedEntitiesResult loadResolvedEntities( > > @Nonnull PolarisCallContext callCtx, > > @Nonnull List<EntityNameLookupRecord> entityLookupRecords); > > > > and > > > > ResolvedEntitiesResult loadResolvedEntities( > > @Nonnull PolarisCallContext callCtx, > > @Nonnull PolarisEntityType entityType, > > @Nonnull List<PolarisEntityId> entityIds); > > > > The first API is useful for batch loading entities after a list > operation, > > but may only require loading a subset of entities (e.g., because a subset > > is already in the in-memory cache). The latter supports a batch version > of > > the loadEntity - also useful for loading entities for the cache without > the > > need to refresh one by one (consider the Resolver, which does this). > > > > Notably, both APIs return ResolvedEntitiesResult, rather than > > EntitiesResult because I'd like to resolve the discrepancy between the > > EntityCache API and the PolarisMetaStoreManager API. Right now, it's > > important to directly interact with the cache whereas, if we can > > consolidate the API, it could be possible to support a cache-backed > > PolarisMetaStoreManager implementation that manages the in-memory cache > > invisibly. > > > > I already have a draft PR with some of the work done, but if everyone > > agrees on the additions, I can have a PR ready for review next week some > > time. Let me know your thoughts. > > > > Mike > > >
