collado-mike commented on PR #2290:
URL: https://github.com/apache/polaris/pull/2290#issuecomment-3193418580

   > The more I mull it over the the more it seems like the previous additions 
of `filter` and `transformer` to `BasePersistence` were a mistake in hindsight, 
especially as long as it was used simply as syntactic sugar rather than 
defining any true pushdown semantic, _except_ for the filtering in `loadTasks` 
(which we want to rework anyways),
   
   the filter/transformer arguments were specifically added to support the 
`loadTasks` API 😅. Personally, I see a lot of potential usefulness in the 
ability to do pushdown filtering especially with regard to pagination (If I 
specify a page of 10 tables, don't give me back 8 and then tell me there are 
more pages because you did the filter after the list). But you're totally right 
regarding the serialization problem. Without a filter/transform argument, we'd 
have to add a new method for every list* method that needed to support a new 
filter predicate. Admittedly, the in-memory filter isn't as efficient as it 
would be if we, say, expressed the predicate in SQL and pushed down into the 
actual database call, but it's still useful for iterating over statement 
results and terminating the results iteration after the correct number of 
results have passed the filter.
   
   > It's technically possible to add basic `loadEntities` that preserves this 
invariant for now without the `filter`/`transformer` arguments, while still 
providing the benefit of the more fundamental functionality of fetching entire 
entities all at once.
   
   I think we need to add this anyway. I've been planning on some proposed 
changes for fetching principal roles during authentication and improving our 
list iteration. Both `loadEntitiesByName` and `loadEntitiesById` seem like they 
solve the N+1 problem  and also avoid trying to serialize lambdas over the wire.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@polaris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Reply via email to