Tarball Compaction Performance Issue
Hi I already posted a question on Jackrabbit Users but did not get a response so far. That said we changed the approach and ran into new issues with the TarBall Compaction. Our Segment Store on an AEM 6.5.6 (oak.core v. 1.22.4) is about 700GB and was not compacted for many years. We tested that we can run the compaction with 1.62.0 without any side-effects and so we started it this way with JDK 11: java \ -Dtar.memoryMapped=true \ -Doak.compaction.eagerFlush=true \ -Dlogback.configurationFile=logback-compaction.xml \ -jar oak-run-1.62.0.jar \ —compactor=parallel \ —trheads=16 \ This started pretty well until about 15% compared and then came to a crawl where only one process is actually running. Thread Dump: "pool-2-thread-2" #23 prio=5 os_prio=0 cpu=1837679.95ms elapsed=61999.36s tid=0x7ecf0d913800 nid=0x1835e6 waiting on condition [0x7ecec0afd000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.22/Native Method) - parking to wait for <0x000451000178> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.22/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.22/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.22/LinkedBlockingQueue.java:433) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.22/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.22/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.22/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.22/Thread.java:834) "pool-2-thread-3" #24 prio=5 os_prio=0 cpu=34785318.09ms elapsed=61999.29s tid=0x7ecf0d914800 nid=0x1835e8 runnable [0x7ece7bffd000] java.lang.Thread.State: RUNNABLE at java.lang.ThreadLocal.get(java.base@11.0.22/ThreadLocal.java:163) at java.lang.StringCoding.decodeUTF8(java.base@11.0.22/StringCoding.java:723) at java.lang.StringCoding.decode(java.base@11.0.22/StringCoding.java:257) at java.lang.String.(java.base@11.0.22/String.java:507) at java.lang.String.(java.base@11.0.22/String.java:561) at org.apache.jackrabbit.oak.segment.data.SegmentDataV12.getSignature(SegmentDataV12.java:88) at org.apache.jackrabbit.oak.segment.Segment.(Segment.java:201) at org.apache.jackrabbit.oak.segment.file.AbstractFileStore.readSegmentUncached(AbstractFileStore.java:300) at org.apache.jackrabbit.oak.segment.file.FileStore.lambda$readSegment$10(FileStore.java:512) at org.apache.jackrabbit.oak.segment.file.FileStore$$Lambda$95/0x0008001ff840.call(Unknown Source) at org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.lambda$getSegment$0(SegmentCache.java:163) at org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache$$Lambda$96/0x0008001ffc40.call(Unknown Source) at org.apache.jackrabbit.guava.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4938) at org.apache.jackrabbit.guava.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3576) at org.apache.jackrabbit.guava.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2318) at org.apache.jackrabbit.guava.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2191) - locked <0x0006f028d408> (a org.apache.jackrabbit.guava.common.cache.LocalCache$StrongAccessEntry) at org.apache.jackrabbit.guava.common.cache.LocalCache$Segment.get(LocalCache.java:2081) at org.apache.jackrabbit.guava.common.cache.LocalCache.get(LocalCache.java:4019) at org.apache.jackrabbit.guava.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4933) at org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.getSegment(SegmentCache.java:160) at org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:512) at org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:153) - locked <0x0006f028d218> (a org.apache.jackrabbit.oak.segment.SegmentId) at org.apache.jackrabbit.oak.segment.CachingSegmentReader$1.apply(CachingSegmentReader.java:105) at org.apache.jackrabbit.oak.segment.CachingSegmentReader$1.apply(CachingSegmentReader.java:101) at org.apache.jackrabbit.oak.segment.ReaderCache.get(ReaderCache.java:117) at org.apache.jackrabbit.oak.segment.CachingSegmentReader.readString(CachingSegmentReader.java:101) at org.apache.jackrabbit.oak.segment.MapRecord.getEntries(MapRecord.java:400) at org.apache.jackrabbit.oak.segment.MapRecord$2.iterator(MapRecord.java:384)
[PR] OAK-10833 - Consistency check reports success for repository with SNF… [jackrabbit-oak]
dulceanu opened a new pull request, #1487: URL: https://github.com/apache/jackrabbit-oak/pull/1487 …E in checkpoints -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@jackrabbit.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Issue/oak 10621 [jackrabbit-oak]
reschke commented on PR #1286: URL: https://github.com/apache/jackrabbit-oak/pull/1286#issuecomment-2129369108 @mbaedke - could you please rebase the patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@jackrabbit.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Issues/oak 10781 [jackrabbit-oak]
rootpea commented on PR #1485: URL: https://github.com/apache/jackrabbit-oak/pull/1485#issuecomment-2129047177 @t-rana the jenkins test failed, could you please re-run -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@jackrabbit.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Issue/oak 8848 [jackrabbit-oak]
reschke commented on code in PR #1474: URL: https://github.com/apache/jackrabbit-oak/pull/1474#discussion_r1613082105 ## oak-jcr/src/test/java/org/apache/jackrabbit/oak/jcr/MoveVersionableNodeWithMemoryRepositoryTest.java: ## @@ -0,0 +1,26 @@ +package org.apache.jackrabbit.oak.jcr; Review Comment: So, IIUC, you have created mutliple test classes so that the different mixtures are covered automatically. That was certainly helpful for testing, but it wouldn't scale if we did this everywhere. So I would propose to simplify that again; and we also should wortk on the orthogonal issue of documenting and potentially fixing the CI pipelines. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@jackrabbit.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] OAK-10826 - Misleading warning about missing checkpoint [jackrabbit-oak]
rishabhdaim merged PR #1479: URL: https://github.com/apache/jackrabbit-oak/pull/1479 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@jackrabbit.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] OAK-10830: Upgrade jackson-databind dependency to 2.17.1 [jackrabbit-oak]
reschke merged PR #1486: URL: https://github.com/apache/jackrabbit-oak/pull/1486 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@jackrabbit.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] OAK-10830: Upgrade jackson-databind dependency to 2.17.1 [jackrabbit-oak]
reschke opened a new pull request, #1486: URL: https://github.com/apache/jackrabbit-oak/pull/1486 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@jackrabbit.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] OAK-10780: add azure access token refresh logic [jackrabbit-oak]
smiroslav merged PR #1441: URL: https://github.com/apache/jackrabbit-oak/pull/1441 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@jackrabbit.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (JCR-5062) Attempt to return connection twice
[
https://issues.apache.org/jira/browse/JCR-5062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849201#comment-17849201
]
Julian Reschke commented on JCR-5062:
-
We could add debug information to find out why the connection is closed twice.
But it's a bit of work. (We have similar code in Oak's RDB code).
> Attempt to return connection twice
> --
>
> Key: JCR-5062
> URL: https://issues.apache.org/jira/browse/JCR-5062
> Project: Jackrabbit Content Repository
> Issue Type: Bug
>Affects Versions: 2.20.16
>Reporter: Henrik Hald Nørgaard
>Priority: Minor
> Attachments: stacktrace.txt
>
>
> We are using a JackRabbit content repository and are going to update some
> nodes with a new mixin. While iterating through the nodes that should be
> updated, the following warning sometimes appears in the log:
> {{WARN [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool]
> (ServerService Thread Pool -- 88) IJ000609: Attempt to return connection
> twice:
> org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener@f6ff104[state=NORMAL
> managed
> connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@7f3ceaf7
> connection handles=0 lastReturned=1715156332529 lastValidated=1715156332529
> lastCheckedOut=1715156332528 trackByTx=false
> pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@16cc91e8
> mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@53f6d09[pool=DezideAuthorJackrabbit]]:
> java.lang.Throwable: STACKTRACE}}
> I have attached a stacktrace.
> It seems that the database connection handled by the
> BundleDbPersistenceManager can in some situations be closed twice?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCR-5062) Attempt to return connection twice
[
https://issues.apache.org/jira/browse/JCR-5062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849200#comment-17849200
]
Henrik Hald Nørgaard commented on JCR-5062:
---
Thank you for your reply.
It will be difficult to write a test case that consistently reproduces this
warning. It seems to appear when we are doing "a lot" of work in the repository
and it does not appear consistently.
I think that we will close our eyes and ignore this warning for now.
> Attempt to return connection twice
> --
>
> Key: JCR-5062
> URL: https://issues.apache.org/jira/browse/JCR-5062
> Project: Jackrabbit Content Repository
> Issue Type: Bug
>Affects Versions: 2.20.16
>Reporter: Henrik Hald Nørgaard
>Priority: Minor
> Attachments: stacktrace.txt
>
>
> We are using a JackRabbit content repository and are going to update some
> nodes with a new mixin. While iterating through the nodes that should be
> updated, the following warning sometimes appears in the log:
> {{WARN [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool]
> (ServerService Thread Pool -- 88) IJ000609: Attempt to return connection
> twice:
> org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener@f6ff104[state=NORMAL
> managed
> connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@7f3ceaf7
> connection handles=0 lastReturned=1715156332529 lastValidated=1715156332529
> lastCheckedOut=1715156332528 trackByTx=false
> pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@16cc91e8
> mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@53f6d09[pool=DezideAuthorJackrabbit]]:
> java.lang.Throwable: STACKTRACE}}
> I have attached a stacktrace.
> It seems that the database connection handled by the
> BundleDbPersistenceManager can in some situations be closed twice?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] Issues/oak 10781 [jackrabbit-oak]
t-rana opened a new pull request, #1485: URL: https://github.com/apache/jackrabbit-oak/pull/1485 Azure Access token is valid for 1 hour only. Added code to check for token expiry every 1 minutes, and refresh the token when the same is close to expiry. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@jackrabbit.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
