[
https://issues.apache.org/jira/browse/OAK-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17890839#comment-17890839
]
Alejandro Moratinos edited comment on OAK-11171 at 10/23/24 6:25 AM:
---------------------------------------------------------------------
Created a new benchmark test to test external user login added to local and
external groups using cached membership provider.
Executed different set of benchmarks with different set of parameters. All test
were executed with 30s of runtime and 15s of warmup with different concurrency
threads setup. Cache expiration is set with a large value to ensure values are
kept through benchmark execution.
300 users, 10 external groups, 10 local groups
{code:java}
## No cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar
target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50 \
--numberOfGroups 10 --numberOfLocalGroups 10 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max
N mean
Oak-Segment-Tar 1 4 4 4 5 33
6770 4
Oak-Segment-Tar 5 10 16 18 20 61
8431 18
Oak-Segment-Tar 10 25 36 38 41 81
7825 38
Oak-Segment-Tar 20 44 72 75 79 183
7747 78
Oak-Segment-Tar 50 46 172 184 330 808
7311 206
Unique users 300 out of total 300. Groups 10. Err 0. Seed 1729165820373
## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar
target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50 \
--numberOfGroups 10 --numberOfLocalGroups 10 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max
N mean
Oak-Segment-Tar 1 1 2 2 2 27
14615 2
Oak-Segment-Tar 5 3 7 8 10 86
18042 8
Oak-Segment-Tar 10 8 13 15 18 49
19051 16
Oak-Segment-Tar 20 17 29 32 37 194
17872 34
Oak-Segment-Tar 50 13 77 81 156 457
16045 94
Unique users 300 out of total 300. Groups 10. Err 0. Seed 1729234661753{code}
300 users, 100 groups and 10 local groups
{code:java}
## No Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar
target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50 \
--numberOfGroups 100 --numberOfLocalGroups 10 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max
N mean
Oak-Segment-Tar 1 41 43 44 47 102
671 45
Oak-Segment-Tar 5 125 184 192 201 267
779 193
Oak-Segment-Tar 10 233 412 446 480 625
676 446
Oak-Segment-Tar 20 428 872 960 1037 1839
633 957
Oak-Segment-Tar 50 798 2122 2450 2718 5039
629 2473
Unique users 300 out of total 300. Groups 100. Err 0. Seed 1729167541161
## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar
target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50 \
--numberOfGroups 100 --numberOfLocalGroups 10 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max
N mean
Oak-Segment-Tar 1 12 13 13 15 46
2103 14
Oak-Segment-Tar 5 38 56 64 72 148
2325 65
Oak-Segment-Tar 10 60 118 129 143 373
2268 133
Oak-Segment-Tar 20 109 246 272 302 773
2120 284
Oak-Segment-Tar 50 326 648 699 1334 2716
1899 797
Unique users 300 out of total 300. Groups 100. Err 0. Seed 1729170222022{code}
100 users, 200 groups and 100 local groups
{code:java}
## No Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar
target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 200 --numberOfLocalGroups 100 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max
N mean
Oak-Segment-Tar 1 358 367 375 385 409
80 375
Oak-Segment-Tar 5 2115 2370 2420 2463 2491
65 2412
Oak-Segment-Tar 10 5028 5395 5640 5805 5984
60 5611
Oak-Segment-Tar 20 10090 10304 11285 12178 12757
60 11262
Oak-Segment-Tar 50 14954 15543 28794 33096 34039
74 25654
Unique users 97 out of total 100. Groups 200. Err 0. Seed 1729235215305
## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar
target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 200 --numberOfLocalGroups 100 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max
N mean
Oak-Segment-Tar 1 25 26 28 461 573
158 190
Oak-Segment-Tar 5 63 117 128 147 2393
942 160
Oak-Segment-Tar 10 137 234 255 289 790
1107 272
Oak-Segment-Tar 20 233 468 536 976 2444
998 606
Oak-Segment-Tar 50 643 1317 1451 3534 7357
759 2012
Unique users 100 out of total 100. Groups 200. Err 0. Seed 1729235457138{code}
100 users, 1200 groups and 100 local groups
{code:java}
## No Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar
target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 1200 --numberOfLocalGroups 100 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max
N mean
Oak-Segment-Tar 1 3123 3123 3134 3173 3185
13 3139
Oak-Segment-Tar 5 14616 14675 15289 15698 15744
15 15246
Oak-Segment-Tar 10 27534 32217 34787 37053 38323
20 34477
Oak-Segment-Tar 20 41631 45477 66119 69066 69123
20 62596
Oak-Segment-Tar 50 107535 114668 154058 175837 176512
50 150076
Unique users 69 out of total 100. Groups 1200. Err 0. Seed 1729171093374
## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar
target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 1200 --numberOfLocalGroups 100 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max
N mean
Oak-Segment-Tar 1 170 1067 2477 2753 2785
13 2354
Oak-Segment-Tar 5 736 778 14244 15009 15176
16 9877
Oak-Segment-Tar 10 1687 1764 34183 35392 35404
13 26713
Oak-Segment-Tar 20 3116 3353 59325 67689 71273
33 40481
Oak-Segment-Tar 50 8123 9187 39456 108810 117207
73 54874
Unique users 78 out of total 100. Groups 1200. Err 0. Seed 1729236152455{code}
To test a configuration with short cache time which is closest to reality
executed a test with 30s cache expiration time
{code:java}
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -Druntime=40 -jar
target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 1200 --numberOfLocalGroups 100 --cacheExpiration 30000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max
N mean
Oak-Segment-Tar 1 182 2050 2528 2555 2589
17 2395
Oak-Segment-Tar 5 685 694 14276 14692 14834
17 12682
Oak-Segment-Tar 10 2009 2186 32604 34728 36387
25 26588
Oak-Segment-Tar 20 3008 3619 62627 67037 67603
24 52872
Oak-Segment-Tar 50 7188 10529 138560 167500 168827
56 124069
Unique users 72 out of total 100. Groups 1200. Err 0. Seed 1729171927509 {code}
In general the results obtained with the cache version outperforms the
non-cached version seeing huge reduction in execution times.
was (Author: JIRAUSER307002):
Created a new benchmark test to test external user login added to local and
external groups using cached membership provider.
Executed different set of benchmarks with different set of parameters. All test
were executed with 30s of runtime and 15s of warmup with different concurrency
threads setup. Cache expiration is set with a large value to ensure values are
kept through benchmark execution.
## 300 users, 10 external groups, 10 local groups
{code:java}
## No cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar
target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50 \
--numberOfGroups 10 --numberOfLocalGroups 10 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max
N mean
Oak-Segment-Tar 1 4 4 4 5 33
6770 4
Oak-Segment-Tar 5 10 16 18 20 61
8431 18
Oak-Segment-Tar 10 25 36 38 41 81
7825 38
Oak-Segment-Tar 20 44 72 75 79 183
7747 78
Oak-Segment-Tar 50 46 172 184 330 808
7311 206
Unique users 300 out of total 300. Groups 10. Err 0. Seed 1729165820373
## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar
target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50 \
--numberOfGroups 10 --numberOfLocalGroups 10 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max
N mean
Oak-Segment-Tar 1 1 2 2 2 27
14615 2
Oak-Segment-Tar 5 3 7 8 10 86
18042 8
Oak-Segment-Tar 10 8 13 15 18 49
19051 16
Oak-Segment-Tar 20 17 29 32 37 194
17872 34
Oak-Segment-Tar 50 13 77 81 156 457
16045 94
Unique users 300 out of total 300. Groups 10. Err 0. Seed 1729234661753{code}
### 300 users, 100 groups and 10 local groups
{code:java}
## No Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar
target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50 \
--numberOfGroups 100 --numberOfLocalGroups 10 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max
N mean
Oak-Segment-Tar 1 41 43 44 47 102
671 45
Oak-Segment-Tar 5 125 184 192 201 267
779 193
Oak-Segment-Tar 10 233 412 446 480 625
676 446
Oak-Segment-Tar 20 428 872 960 1037 1839
633 957
Oak-Segment-Tar 50 798 2122 2450 2718 5039
629 2473
Unique users 300 out of total 300. Groups 100. Err 0. Seed 1729167541161
## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar
target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 300 --concurrency 1,5,10,20,50 \
--numberOfGroups 100 --numberOfLocalGroups 10 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max
N mean
Oak-Segment-Tar 1 12 13 13 15 46
2103 14
Oak-Segment-Tar 5 38 56 64 72 148
2325 65
Oak-Segment-Tar 10 60 118 129 143 373
2268 133
Oak-Segment-Tar 20 109 246 272 302 773
2120 284
Oak-Segment-Tar 50 326 648 699 1334 2716
1899 797
Unique users 300 out of total 300. Groups 100. Err 0. Seed 1729170222022{code}
## 100 users, 200 groups and 100 local groups
{code:java}
## No Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar
target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 200 --numberOfLocalGroups 100 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max
N mean
Oak-Segment-Tar 1 358 367 375 385 409
80 375
Oak-Segment-Tar 5 2115 2370 2420 2463 2491
65 2412
Oak-Segment-Tar 10 5028 5395 5640 5805 5984
60 5611
Oak-Segment-Tar 20 10090 10304 11285 12178 12757
60 11262
Oak-Segment-Tar 50 14954 15543 28794 33096 34039
74 25654
Unique users 97 out of total 100. Groups 200. Err 0. Seed 1729235215305
## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar
target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 200 --numberOfLocalGroups 100 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max
N mean
Oak-Segment-Tar 1 25 26 28 461 573
158 190
Oak-Segment-Tar 5 63 117 128 147 2393
942 160
Oak-Segment-Tar 10 137 234 255 289 790
1107 272
Oak-Segment-Tar 20 233 468 536 976 2444
998 606
Oak-Segment-Tar 50 643 1317 1451 3534 7357
759 2012
Unique users 100 out of total 100. Groups 200. Err 0. Seed 1729235457138{code}
## 100 users, 1200 groups and 100 local groups
{code:java}
## No Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar
target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 1200 --numberOfLocalGroups 100 --cacheExpiration 0 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max
N mean
Oak-Segment-Tar 1 3123 3123 3134 3173 3185
13 3139
Oak-Segment-Tar 5 14616 14675 15289 15698 15744
15 15246
Oak-Segment-Tar 10 27534 32217 34787 37053 38323
20 34477
Oak-Segment-Tar 20 41631 45477 66119 69066 69123
20 62596
Oak-Segment-Tar 50 107535 114668 154058 175837 176512
50 150076
Unique users 69 out of total 100. Groups 1200. Err 0. Seed 1729171093374
## Cache
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -jar
target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 1200 --numberOfLocalGroups 100 --cacheExpiration 10000000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max
N mean
Oak-Segment-Tar 1 170 1067 2477 2753 2785
13 2354
Oak-Segment-Tar 5 736 778 14244 15009 15176
16 9877
Oak-Segment-Tar 10 1687 1764 34183 35392 35404
13 26713
Oak-Segment-Tar 20 3116 3353 59325 67689 71273
33 40481
Oak-Segment-Tar 50 8123 9187 39456 108810 117207
73 54874
Unique users 78 out of total 100. Groups 1200. Err 0. Seed 1729236152455{code}
To test a configuration with short cache time which is closest to reality
executed a test with 30s cache expiration time
{code:java}
java -Xmx2048m -Dprofile=false -Druntime=30 -Dwarmup=15 -Druntime=40 -jar
target/oak-benchmarks-1.71-SNAPSHOT.jar benchmark \
--dynamicMembership true --numberOfUsers 100 --concurrency 1,5,10,20,50 \
--numberOfGroups 1200 --numberOfLocalGroups 100 --cacheExpiration 30000 \
--report false CachedMembershipLoginTest Oak-Segment-Tar
Apache Jackrabbit Oak 1.71-SNAPSHOT
# CachedMembershipLoginTest C min 10% 50% 90% max
N mean
Oak-Segment-Tar 1 182 2050 2528 2555 2589
17 2395
Oak-Segment-Tar 5 685 694 14276 14692 14834
17 12682
Oak-Segment-Tar 10 2009 2186 32604 34728 36387
25 26588
Oak-Segment-Tar 20 3008 3619 62627 67037 67603
24 52872
Oak-Segment-Tar 50 7188 10529 138560 167500 168827
56 124069
Unique users 72 out of total 100. Groups 1200. Err 0. Seed 1729171927509 {code}
In general the results obtained with the cache version outperforms the
non-cached version seeing huge reduction in execution times.
> Create benchmark
> ----------------
>
> Key: OAK-11171
> URL: https://issues.apache.org/jira/browse/OAK-11171
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: security
> Reporter: Alejandro Moratinos
> Assignee: Alejandro Moratinos
> Priority: Major
>
> Create or modify actual benchmarks to test external group caching feature
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
