LantaoJin commented on a change in pull request #24982: [SPARK-28181][CORE] Add
a filter interface to KVStore to speed up the entities retrieve
URL: https://github.com/apache/spark/pull/24982#discussion_r298646144
##########
File path:
common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java
##########
@@ -106,6 +106,12 @@ public void delete(Class<?> type, Object naturalKey) {
return list != null ? list.view() : emptyView();
}
+ @Override
+ public <T> KVStoreView<T> viewWithCondition(Class<T> type, Predicate<T>
condition) {
+ InstanceList<T> list = inMemoryLists.get(type);
Review comment:
There are low level APIs, the different between their efficiencies doesn’t
depend on the API self, it depends on the invocation layer. For example the
logic of invocation layer uses `view` to get all items and creates some kinds
of objects for them, then it executes multiple heavy logic, finally it uses a
filter to get less result. So using `viewWithCondiction` is very like SQL
prediction pushdown optimization rule. Even the cost of the filter operation
itself always fixed. But filter out as earlier as possible still could save
memory and time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
