[
https://issues.apache.org/jira/browse/OAK-12290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joerg Hoh updated OAK-12290:
----------------------------
Description:
I investigated a performance bottleneck on an environment running with Oak
2.2.0, and I have come across a series of instances where freshly introduced
CaffeineCache is involved:
I see a massive series of calls like this (566 instances out of 580, where the
search term Caffeine is present in the threaddump):
{noformat}
java.lang.Thread.State: RUNNABLE
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.afterRead(BoundedLocalCache.java:1286)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.getIfPresent(BoundedLocalCache.java:2206)
at
com.github.benmanes.caffeine.cache.LocalManualCache.getIfPresent(LocalManualCache.java:56)
at
org.apache.jackrabbit.oak.cache.impl.caffeine.CaffeineCacheAdapter.getIfPresent(CaffeineCacheAdapter.java:43)
at
org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.recordHit(SegmentCache.java:247)
at
org.apache.jackrabbit.oak.segment.file.AbstractFileStore$2$$Lambda/0x0000000800caa900.accept(Unknown
Source)
at
org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:164)
at org.apache.jackrabbit.oak.segment.Record.getSegment(Record.java:71)
at
org.apache.jackrabbit.oak.segment.MapRecord.getEntry(MapRecord.java:182)
at
org.apache.jackrabbit.oak.segment.SegmentNodeState.getChildNode(SegmentNodeState.java:450)
at
org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.<init>(MemoryNodeBuilder.java:141)
{noformat}
and
{noformat}
java.lang.Thread.State: RUNNABLE
at com.github.benmanes.caffeine.cache.Node.inWindow(Node.java:162)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.onAccess(BoundedLocalCache.java:1782)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda/0x0000000800a4c290.accept(Unknown
Source)
at
com.github.benmanes.caffeine.cache.BoundedBuffer$RingBuffer.drainTo(BoundedBuffer.java:101)
at
com.github.benmanes.caffeine.cache.StripedBuffer.drainTo(StripedBuffer.java:228)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.drainReadBuffer(BoundedLocalCache.java:1769)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.maintenance(BoundedLocalCache.java:1711)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.performCleanUp(BoundedLocalCache.java:1660)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache$PerformCleanupTask.run(BoundedLocalCache.java:3886)
at
org.apache.jackrabbit.oak.cache.api.CacheBuilder$$Lambda/0x0000000800a115c8.execute(Unknown
Source)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.scheduleDrainBuffers(BoundedLocalCache.java:1633)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.afterRead(BoundedLocalCache.java:1288)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.getIfPresent(BoundedLocalCache.java:2206)
at
com.github.benmanes.caffeine.cache.LocalManualCache.getIfPresent(LocalManualCache.java:56)
at
org.apache.jackrabbit.oak.cache.impl.caffeine.CaffeineCacheAdapter.getIfPresent(CaffeineCacheAdapter.java:43)
at
org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.recordHit(SegmentCache.java:247)
at
org.apache.jackrabbit.oak.segment.file.AbstractFileStore$2$$Lambda/0x0000000800caa900.accept(Unknown
Source)
at
org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:164)
at org.apache.jackrabbit.oak.segment.Record.getSegment(Record.java:71)
{noformat}
{noformat:title=SegmentCache.java}
@Override
public void recordHit(@NotNull SegmentId id) {
stats.hitCount.incrementAndGet();
if (id.isDataSegmentId() &&
FT_OAK_12214_PROPAGATE_L1_HITS_TO_L2_ENABLED.get()) {
cache.getIfPresent(id);
}
}
{noformat}
I think that the FT {{FT_OAK_12214_PROPAGATE_L1_HITS_TO_L2_ENABLED}} is
responsible for that, and it creates unnecessary latency (see OAK-12214); while
I think that the idea is good, there should be a different way to increment the
counter.
was:
I investigated a performance bottleneck on an environment running with Oak
2.2.0, and I have come across a series of instances where freshly introduced
CaffeineCache is involved.
I see a massive series of calls like this:
{noformat}
java.lang.Thread.State: RUNNABLE
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.afterRead(BoundedLocalCache.java:1286)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.getIfPresent(BoundedLocalCache.java:2206)
at
com.github.benmanes.caffeine.cache.LocalManualCache.getIfPresent(LocalManualCache.java:56)
at
org.apache.jackrabbit.oak.cache.impl.caffeine.CaffeineCacheAdapter.getIfPresent(CaffeineCacheAdapter.java:43)
at
org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.recordHit(SegmentCache.java:247)
at
org.apache.jackrabbit.oak.segment.file.AbstractFileStore$2$$Lambda/0x0000000800caa900.accept(Unknown
Source)
at
org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:164)
at org.apache.jackrabbit.oak.segment.Record.getSegment(Record.java:71)
at
org.apache.jackrabbit.oak.segment.MapRecord.getEntry(MapRecord.java:182)
at
org.apache.jackrabbit.oak.segment.SegmentNodeState.getChildNode(SegmentNodeState.java:450)
at
org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.<init>(MemoryNodeBuilder.java:141)
{noformat}
and
{noformat}
java.lang.Thread.State: RUNNABLE
at com.github.benmanes.caffeine.cache.Node.inWindow(Node.java:162)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.onAccess(BoundedLocalCache.java:1782)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda/0x0000000800a4c290.accept(Unknown
Source)
at
com.github.benmanes.caffeine.cache.BoundedBuffer$RingBuffer.drainTo(BoundedBuffer.java:101)
at
com.github.benmanes.caffeine.cache.StripedBuffer.drainTo(StripedBuffer.java:228)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.drainReadBuffer(BoundedLocalCache.java:1769)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.maintenance(BoundedLocalCache.java:1711)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.performCleanUp(BoundedLocalCache.java:1660)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache$PerformCleanupTask.run(BoundedLocalCache.java:3886)
at
org.apache.jackrabbit.oak.cache.api.CacheBuilder$$Lambda/0x0000000800a115c8.execute(Unknown
Source)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.scheduleDrainBuffers(BoundedLocalCache.java:1633)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.afterRead(BoundedLocalCache.java:1288)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.getIfPresent(BoundedLocalCache.java:2206)
at
com.github.benmanes.caffeine.cache.LocalManualCache.getIfPresent(LocalManualCache.java:56)
at
org.apache.jackrabbit.oak.cache.impl.caffeine.CaffeineCacheAdapter.getIfPresent(CaffeineCacheAdapter.java:43)
at
org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.recordHit(SegmentCache.java:247)
at
org.apache.jackrabbit.oak.segment.file.AbstractFileStore$2$$Lambda/0x0000000800caa900.accept(Unknown
Source)
at
org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:164)
at org.apache.jackrabbit.oak.segment.Record.getSegment(Record.java:71)
{noformat}
{noformat:title=SegmentCache.java}
@Override
public void recordHit(@NotNull SegmentId id) {
stats.hitCount.incrementAndGet();
if (id.isDataSegmentId() &&
FT_OAK_12214_PROPAGATE_L1_HITS_TO_L2_ENABLED.get()) {
cache.getIfPresent(id);
}
}
{noformat}
I think that the FT {{FT_OAK_12214_PROPAGATE_L1_HITS_TO_L2_ENABLED}} is
responsible for that, and it creates unnecessary latency (see OAK-12214); while
I think that the idea is good, there should be a different way to increment the
counter.
> Lock Contention in SegmentCache
> -------------------------------
>
> Key: OAK-12290
> URL: https://issues.apache.org/jira/browse/OAK-12290
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
> Affects Versions: 2.2.0
> Reporter: Joerg Hoh
> Priority: Major
>
> I investigated a performance bottleneck on an environment running with Oak
> 2.2.0, and I have come across a series of instances where freshly introduced
> CaffeineCache is involved:
> I see a massive series of calls like this (566 instances out of 580, where
> the search term Caffeine is present in the threaddump):
> {noformat}
> java.lang.Thread.State: RUNNABLE
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.afterRead(BoundedLocalCache.java:1286)
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.getIfPresent(BoundedLocalCache.java:2206)
> at
> com.github.benmanes.caffeine.cache.LocalManualCache.getIfPresent(LocalManualCache.java:56)
> at
> org.apache.jackrabbit.oak.cache.impl.caffeine.CaffeineCacheAdapter.getIfPresent(CaffeineCacheAdapter.java:43)
> at
> org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.recordHit(SegmentCache.java:247)
> at
> org.apache.jackrabbit.oak.segment.file.AbstractFileStore$2$$Lambda/0x0000000800caa900.accept(Unknown
> Source)
> at
> org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:164)
> at org.apache.jackrabbit.oak.segment.Record.getSegment(Record.java:71)
> at
> org.apache.jackrabbit.oak.segment.MapRecord.getEntry(MapRecord.java:182)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeState.getChildNode(SegmentNodeState.java:450)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.<init>(MemoryNodeBuilder.java:141)
> {noformat}
> and
> {noformat}
> java.lang.Thread.State: RUNNABLE
> at com.github.benmanes.caffeine.cache.Node.inWindow(Node.java:162)
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.onAccess(BoundedLocalCache.java:1782)
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda/0x0000000800a4c290.accept(Unknown
> Source)
> at
> com.github.benmanes.caffeine.cache.BoundedBuffer$RingBuffer.drainTo(BoundedBuffer.java:101)
> at
> com.github.benmanes.caffeine.cache.StripedBuffer.drainTo(StripedBuffer.java:228)
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.drainReadBuffer(BoundedLocalCache.java:1769)
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.maintenance(BoundedLocalCache.java:1711)
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.performCleanUp(BoundedLocalCache.java:1660)
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache$PerformCleanupTask.run(BoundedLocalCache.java:3886)
> at
> org.apache.jackrabbit.oak.cache.api.CacheBuilder$$Lambda/0x0000000800a115c8.execute(Unknown
> Source)
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.scheduleDrainBuffers(BoundedLocalCache.java:1633)
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.afterRead(BoundedLocalCache.java:1288)
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.getIfPresent(BoundedLocalCache.java:2206)
> at
> com.github.benmanes.caffeine.cache.LocalManualCache.getIfPresent(LocalManualCache.java:56)
> at
> org.apache.jackrabbit.oak.cache.impl.caffeine.CaffeineCacheAdapter.getIfPresent(CaffeineCacheAdapter.java:43)
> at
> org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.recordHit(SegmentCache.java:247)
> at
> org.apache.jackrabbit.oak.segment.file.AbstractFileStore$2$$Lambda/0x0000000800caa900.accept(Unknown
> Source)
> at
> org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:164)
> at org.apache.jackrabbit.oak.segment.Record.getSegment(Record.java:71)
> {noformat}
> {noformat:title=SegmentCache.java}
> @Override
> public void recordHit(@NotNull SegmentId id) {
> stats.hitCount.incrementAndGet();
> if (id.isDataSegmentId() &&
> FT_OAK_12214_PROPAGATE_L1_HITS_TO_L2_ENABLED.get()) {
> cache.getIfPresent(id);
> }
> }
> {noformat}
> I think that the FT {{FT_OAK_12214_PROPAGATE_L1_HITS_TO_L2_ENABLED}} is
> responsible for that, and it creates unnecessary latency (see OAK-12214);
> while I think that the idea is good, there should be a different way to
> increment the counter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
