dongxiaoman commented on a change in pull request #7243:
URL: https://github.com/apache/pinot/pull/7243#discussion_r681353371
##########
File path:
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/PartitionSegmentPruner.java
##########
@@ -278,5 +287,37 @@ private boolean isPartitionMatch(FilterQueryTree
filterQueryTree, PartitionInfo
_partitionFunction = partitionFunction;
_partitions = partitions;
}
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (o == null || getClass() != o.getClass()) return false;
+ PartitionInfo that = (PartitionInfo) o;
+ return Objects.equals(_partitionFunction, that._partitionFunction) &&
+ Objects.equals(_partitions, that._partitions);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(_partitionFunction, _partitions);
+ }
+ }
+
+ /**
+ * This class is created to speed up partition number calls. Sometimes with
lots of segments (like 40k+)
+ * Computing partition values may be CPU intensive, caching the computation
result may help
+ */
+ private static class CachedPartitionFunction {
+ final Map<PartitionFunction, Map<String, Integer>> cache = new HashMap<>();
+
+ public int getPartition(PartitionFunction func, String value) {
Review comment:
For a partition function, it should be a stable fixed map: for each
specific function f(x) and a string input x, we should always get the same
partition for that immutable string x.
The only major concern could be the cached map size; if we have too many
values in the query we could end up having OOM case.
Let me change to use Guava's CacheBuilder if possible
https://guava.dev/releases/17.0/api/docs/com/google/common/cache/CacheBuilder.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
