emissionnebula opened a new pull request, #13437: URL: https://github.com/apache/kafka/pull/13437
Currently, StandardAuthorizer uses a R/W lock for maintaining the consistency of data. For the clusters with very high traffic, we will typically see an increase in latencies whenever a write operation comes. The intent of this PR is to get rid of the R/W lock with the help of immutable or persistent collections. Basically, new object references are used to hold the intermediate state of the write operation. After the completion of the operation, the main reference to the cache is changed to point to the new object. Also, for the read operation, the code is changed such that all accesses to the cache for a single read operation are done to a particular cache object only. In another [PR](https://github.com/apache/kafka/pull/13280), new persistent libraries ([Paguro](https://github.com/GlenKPeterson/Paguro/) and [Vavr](https://github.com/vavr-io/vavr)) have been evaluated to improve the performance of Kraft metadata image change performance. That PR favors Vavr over Paguro. In the description below, you can find the performance of various libraries at the time of both read and write. Read performance is checked with the existing `AuthorizerBenchmark`. For write performance, a new `AuthorizerUpdateBenchmark` has been added which evaluates the performance of the `addAcl` operation. # Library Comparison (TLDR) | | PCollections | Paguro | Vavr | |--------|--------|--------|--------| | Implements Java native collection interfaces | ✅ | ✅ | ❗ (Provides adaptors to convert to Java, which in case of TreeSet are not efficient) | | Read Performance | ❌ (10000x of AclAuthorizer) | ✅ (Similar to AclAuthorizer) | ❌ (1000x of AclAuthorizer) | | Write Performance | ⚠️ (10x of Paguro) | ✅ (best) | ✅ (4x of Paguro) | | Library Completeness | ✅ (HashMap, HashSet, TreeMap, TreeSet, OrderList, OrderedSet, OrderedMap, Stack, Queue) | ⚠️ (HashMap, HashSet, TreeMap, TreeSet, List) | ✅ (most complete - Array, List, Vector, HashMap, HashSet, LinkedHashMap, LinkedHashSet, PriorityQueue, TreeMap, TreeSet) | # Conclusion Since the read time performance is better for pcollections compared to Vavr or Paguro, for StandardAuthorizer it makes sense to go ahead with it. But for Kraft metadata image change patch, `pcollections` was [not considered](https://github.com/apache/kafka/pull/13280#issuecomment-1442082585) because of performance concerns explained on this [page](https://github.com/GlenKPeterson/Paguro/wiki/UncleJim-vs.-PCollections). So now there are two ways forward - 1. Check the performance of Kafka metadata image change with pcollections and see if we are also noticing any difference compared to Vavr. 2. Include two separate libraries in Kafka - `pcollections` for StandardAuthorizer and Vavr for Kraft metadata image change. # Appendix: Detailed Library Comparison ## Current Performance ### AclAuthorizer Read ``` Benchmark (aclCount) (denyPercentage) (resourceCount) Mode Cnt Score Error Units AuthorizerBenchmark.testAclsIterator 50 20 50000 avgt 2 186.399 ms/op AuthorizerBenchmark.testAuthorizeByResourceType 50 20 50000 avgt 2 0.005 ms/op AuthorizerBenchmark.testAuthorizer 50 20 50000 avgt 2 0.244 ms/op AuthorizerBenchmark.testUpdateCache 50 20 50000 avgt 2 502.015 ms/op JMH benchmarks done ``` ### AclAuthorizer Read ``` Benchmark (aclCount) (denyPercentage) (resourceCount) Mode Cnt Score Error Units AuthorizerBenchmark.testAclsIterator 50 20 50000 avgt 2 218.633 ms/op AuthorizerBenchmark.testAuthorizeByResourceType 50 20 50000 avgt 2 486.814 ms/op AuthorizerBenchmark.testAuthorizer 50 20 50000 avgt 2 0.750 ms/op AuthorizerBenchmark.testUpdateCache 50 20 50000 avgt 2 ≈ 10⁻⁶ ms/op JMH benchmarks done ``` ## PCollections ### Read ``` Benchmark (aclCount) (denyPercentage) (resourceCount) Mode Cnt Score Error Units AuthorizerBenchmark.testAclsIterator 50 20 50000 avgt 2 208.878 ms/op AuthorizerBenchmark.testAuthorizeByResourceType 50 20 50000 avgt 2 509.253 ms/op AuthorizerBenchmark.testAuthorizer 50 20 50000 avgt 2 0.785 ms/op AuthorizerBenchmark.testUpdateCache 50 20 50000 avgt 2 ≈ 10⁻⁶ ms/op JMH benchmarks done ``` ### Write ``` Benchmark Mode Cnt Score Error Units AuthorizerUpdateBenchmark.testAddAcl avgt 4 9.751 ± 2.633 ms/op JMH benchmarks done ``` ## Paguro ⚠️ Note - There are two bugs in the TreeSet code of Paguro. Following results are computed after fixing those bugs in local. ### Read ``` Benchmark (aclCount) (denyPercentage) (resourceCount) Mode Cnt Score Error Units AuthorizerBenchmark.testAclsIterator 50 20 50000 avgt 2 289.411 ms/op AuthorizerBenchmark.testAuthorizeByResourceType 50 20 50000 avgt 2 1048.940 ms/op AuthorizerBenchmark.testAuthorizer 50 20 50000 avgt 2 3751.478 ms/op AuthorizerBenchmark.testUpdateCache 50 20 50000 avgt 2 ≈ 10⁻⁶ ms/op JMH benchmarks done ``` ⚠️ - Read performance of Paguro is worst than pcollections because of inefficient implementation of the `tailSet()` function. In Paguro, it is an O(n) operation however in `pcollections` it is an O(log(n)) operation. ### Write ```Benchmark Mode Cnt Score Error Units AuthorizerUpdateBenchmark.testAddAcl avgt 4 0.509 ± 1.108 ms/op JMH benchmarks done ``` ## Vavr ### Read ``` Vavr Benchmark (aclCount) (denyPercentage) (resourceCount) Mode Cnt Score Error Units AuthorizerBenchmark.testAclsIterator 50 20 50000 avgt 2 277.919 ms/op AuthorizerBenchmark.testAuthorizeByResourceType 50 20 50000 avgt 2 3406.584 ms/op AuthorizerBenchmark.testAuthorizer 50 20 50000 avgt 2 3465.406 ms/op AuthorizerBenchmark.testUpdateCache 50 20 50000 avgt 2 ≈ 10⁻⁶ ms/op JMH benchmarks done ``` ⚠️ - Vavr doesn't implement Java's SortedSet interface and it doesn't expose any API equivalent to [SortedSet.tailSet()](https://docs.oracle.com/javase/8/docs/api/java/util/SortedSet.html#tailSet-E-). So, it is required to convert the vavr's TreeSet into Java equivalent, which is an O(n) operation, and then call `tailSet` over it. ### Write ``` Benchmark Mode Cnt Score Error Units AuthorizerUpdateBenchmark.testAddAcl avgt 4 1.515 ± 0.995 ms/op JMH benchmarks done ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
