collado-mike commented on PR #2290: URL: https://github.com/apache/polaris/pull/2290#issuecomment-3193418580
> The more I mull it over the the more it seems like the previous additions of `filter` and `transformer` to `BasePersistence` were a mistake in hindsight, especially as long as it was used simply as syntactic sugar rather than defining any true pushdown semantic, _except_ for the filtering in `loadTasks` (which we want to rework anyways), the filter/transformer arguments were specifically added to support the `loadTasks` API 😅. Personally, I see a lot of potential usefulness in the ability to do pushdown filtering especially with regard to pagination (If I specify a page of 10 tables, don't give me back 8 and then tell me there are more pages because you did the filter after the list). But you're totally right regarding the serialization problem. Without a filter/transform argument, we'd have to add a new method for every list* method that needed to support a new filter predicate. Admittedly, the in-memory filter isn't as efficient as it would be if we, say, expressed the predicate in SQL and pushed down into the actual database call, but it's still useful for iterating over statement results and terminating the results iteration after the correct number of results have passed the filter. > It's technically possible to add basic `loadEntities` that preserves this invariant for now without the `filter`/`transformer` arguments, while still providing the benefit of the more fundamental functionality of fetching entire entities all at once. I think we need to add this anyway. I've been planning on some proposed changes for fetching principal roles during authentication and improving our list iteration. Both `loadEntitiesByName` and `loadEntitiesById` seem like they solve the N+1 problem and also avoid trying to serialize lambdas over the wire. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@polaris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org