[ https://issues.apache.org/jira/browse/CASSANDRA-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14069862#comment-14069862 ]
Vijay commented on CASSANDRA-7438: ---------------------------------- Attached patch makes the off heap/Serialization Cache configurable. (the default is still SerializationCache). Regarding performance, the performance of the new cache is obviously better when the JNI overhead is less than the GC overhead, but for the smaller sized caches which can fit in memory the performance is little lower which is understandable (but both of them out perform pagecache performance by a large number). Here are the numbers. OffheapCacheProvider ==================== Running READ with 1200 threads for 10000000 iterations ops , op/s, key/s, mean, med, .95, .99, .999, max, time, stderr 2030355 , 2029531, 2029531, 3.1, 3.1, 5.4, 5.7, 61.8, 3014.5, 1.0, 0.00000 2395480 , 202845, 202845, 5.8, 5.4, 5.8, 20.2, 522.4, 545.9, 2.8, 0.00000 2638600 , 221368, 221368, 5.4, 5.3, 5.8, 16.3, 78.8, 131.5, 3.9, 0.57860 2891705 , 221976, 221976, 5.4, 5.3, 5.6, 6.2, 15.2, 19.2, 5.0, 0.60478 3147747 , 222527, 222527, 5.4, 5.3, 5.6, 6.1, 15.4, 18.2, 6.2, 0.58659 3394999 , 221527, 221527, 5.4, 5.3, 5.6, 6.6, 15.9, 19.4, 7.3, 0.55884 3663559 , 226114, 226114, 5.3, 5.2, 5.6, 15.0, 84.4, 110.7, 8.5, 0.52924 3911154 , 223831, 223831, 5.4, 5.3, 5.6, 6.1, 15.6, 20.0, 9.6, 0.50018 4152946 , 223246, 223246, 5.4, 5.3, 5.6, 6.1, 15.7, 18.8, 10.7, 0.47323 4403162 , 228532, 228532, 5.2, 5.2, 5.6, 23.2, 107.4, 121.4, 11.8, 0.44856 4641021 , 225196, 225196, 5.3, 5.2, 5.6, 5.9, 15.3, 18.4, 12.8, 0.42557 4889523 , 222826, 222826, 5.4, 5.3, 5.6, 6.3, 16.2, 22.0, 13.9, 0.40476 5124891 , 223203, 223203, 5.4, 5.3, 5.6, 5.8, 6.2, 14.8, 15.0, 0.38602 5375262 , 221222, 221222, 5.4, 5.2, 5.6, 18.4, 94.2, 115.1, 16.1, 0.36899 5616470 , 224022, 224022, 5.4, 5.3, 5.6, 5.9, 14.3, 17.8, 17.2, 0.35349 5866825 , 223000, 223000, 5.4, 5.3, 5.6, 6.1, 15.5, 18.2, 18.3, 0.33882 6125601 , 225757, 225757, 5.2, 5.3, 5.6, 9.6, 49.4, 72.0, 19.5, 0.32535 6348030 , 192703, 192703, 6.3, 5.3, 9.3, 14.4, 77.1, 91.5, 20.6, 0.31282 6483574 , 128520, 128520, 9.3, 8.4, 10.9, 19.5, 88.7, 99.0, 21.7, 0.30329 6626176 , 137199, 137199, 8.7, 8.4, 10.6, 14.0, 32.7, 40.9, 22.7, 0.29771 6768401 , 136860, 136860, 8.8, 8.4, 10.3, 14.1, 35.1, 40.8, 23.8, 0.29213 6911785 , 138204, 138204, 8.7, 8.3, 10.2, 13.7, 34.1, 37.8, 24.8, 0.28669 7055951 , 138633, 138633, 8.7, 8.3, 10.5, 32.0, 40.5, 46.9, 25.8, 0.28130 7199084 , 137731, 137731, 8.7, 8.4, 10.2, 14.0, 33.4, 40.9, 26.9, 0.27623 7338032 , 133201, 133201, 9.0, 8.4, 10.9, 34.0, 39.4, 43.8, 27.9, 0.27116 7480439 , 137059, 137059, 8.8, 8.4, 10.2, 13.9, 35.9, 39.5, 29.0, 0.26663 7647810 , 161209, 161209, 7.5, 7.8, 9.6, 13.4, 33.9, 77.9, 30.0, 0.26185 7898882 , 226498, 226498, 5.3, 5.2, 5.6, 19.7, 108.5, 119.3, 31.1, 0.25629 8136305 , 223840, 223840, 5.4, 5.3, 5.6, 5.9, 17.3, 23.2, 32.2, 0.24838 8372076 , 223790, 223790, 5.4, 5.3, 5.6, 6.0, 15.2, 20.0, 33.2, 0.24095 8633758 , 232914, 232914, 5.1, 5.2, 5.6, 17.5, 138.4, 182.0, 34.4, 0.23397 8869214 , 222243, 222243, 5.4, 5.3, 5.6, 6.0, 15.2, 17.9, 35.4, 0.22717 9121652 , 223037, 223037, 5.4, 5.3, 5.6, 5.9, 15.4, 18.8, 36.5, 0.22105 9360286 , 225070, 225070, 5.3, 5.3, 5.6, 14.8, 82.7, 92.1, 37.6, 0.21524 9609676 , 224089, 224089, 5.4, 5.3, 5.6, 5.8, 6.2, 14.3, 38.7, 0.20967 9848551 , 222123, 222123, 5.4, 5.3, 5.6, 5.9, 24.2, 27.2, 39.8, 0.20440 10000000 , 229511, 229511, 5.0, 5.2, 5.8, 60.0, 74.3, 132.0, 40.5, 0.19935 Results: real op rate : 247211 adjusted op rate stderr : 0 key rate : 247211 latency mean : 5.4 latency median : 3.5 latency 95th percentile : 5.5 latency 99th percentile : 6.1 latency 99.9th percentile : 83.4 latency max : 3014.5 Total operation time : 00:00:40 END SerializingCacheProvider ======================== Running READ with 1200 threads for 10000000 iterations ops , op/s, key/s, mean, med, .95, .99, .999, max, time, stderr 2517463 , 2496430, 2496430, 1.5, 0.2, 4.1, 11.5, 94.7, 6217.0, 1.0, 0.00000 6690312 , 266503, 266503, 4.7, 4.6, 5.2, 15.6, 408.0, 21106.5, 16.7, 0.00000 6974745 , 233722, 233722, 5.1, 5.0, 5.3, 6.5, 15.5, 19.0, 17.9, 0.57070 7275403 , 277156, 277156, 3.9, 4.0, 5.4, 27.6, 62.8, 659.4, 19.0, 0.61144 7770460 , 271264, 271264, 4.7, 5.0, 5.3, 20.2, 426.0, 1037.5, 20.8, 0.59139 8030818 , 233497, 233497, 5.1, 5.0, 5.3, 6.4, 16.5, 20.9, 21.9, 0.56335 8299037 , 232178, 232178, 5.2, 5.1, 5.3, 5.8, 15.8, 21.9, 23.1, 0.54048 8556018 , 231981, 231981, 5.2, 5.1, 5.3, 5.9, 16.7, 20.3, 24.2, 0.51744 8811639 , 234401, 234401, 5.1, 5.0, 5.3, 5.8, 16.2, 19.2, 25.3, 0.49459 9245382 , 262824, 262824, 4.6, 4.9, 5.3, 20.5, 290.9, 546.4, 26.9, 0.47255 9546906 , 235163, 235163, 5.1, 5.0, 5.3, 5.8, 16.1, 21.5, 28.2, 0.44887 9819817 , 235254, 235254, 5.1, 5.0, 5.3, 5.7, 15.5, 19.7, 29.4, 0.43014 10000000 , 233793, 233793, 5.0, 5.0, 5.3, 15.4, 44.0, 51.2, 30.1, 0.41249 Results: real op rate : 331952 adjusted op rate stderr : 0 key rate : 331952 latency mean : 4.0 latency median : 0.7 latency 95th percentile : 5.2 latency 99th percentile : 14.9 latency 99.9th percentile : 294.2 latency max : 21106.5 Total operation time : 00:00:30 END I think we can commit this changes, all known optimizations are done, Thanks! > Serializing Row cache alternative (Fully off heap) > -------------------------------------------------- > > Key: CASSANDRA-7438 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7438 > Project: Cassandra > Issue Type: Improvement > Components: Core > Environment: Linux > Reporter: Vijay > Assignee: Vijay > Labels: performance > Fix For: 3.0 > > > Currently SerializingCache is partially off heap, keys are still stored in > JVM heap as BB, > * There is a higher GC costs for a reasonably big cache. > * Some users have used the row cache efficiently in production for better > results, but this requires careful tunning. > * Overhead in Memory for the cache entries are relatively high. > So the proposal for this ticket is to move the LRU cache logic completely off > heap and use JNI to interact with cache. We might want to ensure that the new > implementation match the existing API's (ICache), and the implementation > needs to have safe memory access, low overhead in memory and less memcpy's > (As much as possible). > We might also want to make this cache configurable. -- This message was sent by Atlassian JIRA (v6.2#6252)