Hey folks
Building on the loadEntities API added in
https://github.com/apache/polaris/pull/2290 , I'd like to propose two new
APIs to support batch loading of entities in the PolarisMetaStoreManager.
The new loadEntities API is great for avoiding the N+1 queries problem
during lists and I'd like to extend that behavior for batch retrieving
entities outside of list operations.
The two APIs would be
ResolvedEntitiesResult loadResolvedEntities(
@Nonnull PolarisCallContext callCtx,
@Nonnull List<EntityNameLookupRecord> entityLookupRecords);
and
ResolvedEntitiesResult loadResolvedEntities(
@Nonnull PolarisCallContext callCtx,
@Nonnull PolarisEntityType entityType,
@Nonnull List<PolarisEntityId> entityIds);
The first API is useful for batch loading entities after a list operation,
but may only require loading a subset of entities (e.g., because a subset
is already in the in-memory cache). The latter supports a batch version of
the loadEntity - also useful for loading entities for the cache without the
need to refresh one by one (consider the Resolver, which does this).
Notably, both APIs return ResolvedEntitiesResult, rather than
EntitiesResult because I'd like to resolve the discrepancy between the
EntityCache API and the PolarisMetaStoreManager API. Right now, it's
important to directly interact with the cache whereas, if we can
consolidate the API, it could be possible to support a cache-backed
PolarisMetaStoreManager implementation that manages the in-memory cache
invisibly.
I already have a draft PR with some of the work done, but if everyone
agrees on the additions, I can have a PR ready for review next week some
time. Let me know your thoughts.
Mike
