[jira] [Commented] (OAK-10413) DocumentStoreIndexerIT hanging
[
https://issues.apache.org/jira/browse/OAK-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770138#comment-17770138
]
Julian Reschke commented on OAK-10413:
--
trunk:
[0be1b75008|https://github.com/apache/jackrabbit-oak/commit/0be1b75008ec5a1a14634ae480f8c8be4546750b]
[634cb27d07|https://github.com/apache/jackrabbit-oak/commit/634cb27d070fef52c729401375e3203ada7a3185]
> DocumentStoreIndexerIT hanging
> --
>
> Key: OAK-10413
> URL: https://issues.apache.org/jira/browse/OAK-10413
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: documentmk, run
>Reporter: Julian Reschke
>Priority: Major
> Fix For: 1.58.0
>
>
> {noformat}
> [INFO] Running org.apache.jackrabbit.oak.index.DocumentStoreIndexerIT
> [WARNING] Corrupted STDOUT by directly writing to native stream in forked JVM
> 1. See FAQ web page and the dump file
> /home/jenkins/jenkins-agent/workspace/Jackrabbit/jackrabbit-oak-trunk/oak-run/target/failsafe-reports/2023-08-22T13-46-32_129-jvmRun1.dumpstream
> {noformat}
> ...and then eventually times out.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-10413) DocumentStoreIndexerIT hanging
[
https://issues.apache.org/jira/browse/OAK-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762585#comment-17762585
]
Marcel Reutegger commented on OAK-10413:
Created OAK-10432.
> DocumentStoreIndexerIT hanging
> --
>
> Key: OAK-10413
> URL: https://issues.apache.org/jira/browse/OAK-10413
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: documentmk, run
>Reporter: Julian Reschke
>Priority: Major
>
> {noformat}
> [INFO] Running org.apache.jackrabbit.oak.index.DocumentStoreIndexerIT
> [WARNING] Corrupted STDOUT by directly writing to native stream in forked JVM
> 1. See FAQ web page and the dump file
> /home/jenkins/jenkins-agent/workspace/Jackrabbit/jackrabbit-oak-trunk/oak-run/target/failsafe-reports/2023-08-22T13-46-32_129-jvmRun1.dumpstream
> {noformat}
> ...and then eventually times out.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-10413) DocumentStoreIndexerIT hanging
[
https://issues.apache.org/jira/browse/OAK-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762583#comment-17762583
]
Marcel Reutegger commented on OAK-10413:
It is failing, but at least not hanging anymore as reported in this issue. The
test failures were also reported in OAK-10381, but it doesn't look like the
test issue is fixed. I'm going to create a new issue, because OAK-10381 was
resolved and marked fixed in 1.56.0, which is already released.
> DocumentStoreIndexerIT hanging
> --
>
> Key: OAK-10413
> URL: https://issues.apache.org/jira/browse/OAK-10413
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: documentmk, run
>Reporter: Julian Reschke
>Priority: Major
>
> {noformat}
> [INFO] Running org.apache.jackrabbit.oak.index.DocumentStoreIndexerIT
> [WARNING] Corrupted STDOUT by directly writing to native stream in forked JVM
> 1. See FAQ web page and the dump file
> /home/jenkins/jenkins-agent/workspace/Jackrabbit/jackrabbit-oak-trunk/oak-run/target/failsafe-reports/2023-08-22T13-46-32_129-jvmRun1.dumpstream
> {noformat}
> ...and then eventually times out.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-10413) DocumentStoreIndexerIT hanging
[
https://issues.apache.org/jira/browse/OAK-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762272#comment-17762272
]
Mohit Kataria commented on OAK-10413:
-
Looks like this is still failing
[https://ci-builds.apache.org/blue/organizations/jenkins/Jackrabbit%2Foak-trunk-pr/detail/PR-1094/25/pipeline]
> DocumentStoreIndexerIT hanging
> --
>
> Key: OAK-10413
> URL: https://issues.apache.org/jira/browse/OAK-10413
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: documentmk, run
>Reporter: Julian Reschke
>Priority: Major
>
> {noformat}
> [INFO] Running org.apache.jackrabbit.oak.index.DocumentStoreIndexerIT
> [WARNING] Corrupted STDOUT by directly writing to native stream in forked JVM
> 1. See FAQ web page and the dump file
> /home/jenkins/jenkins-agent/workspace/Jackrabbit/jackrabbit-oak-trunk/oak-run/target/failsafe-reports/2023-08-22T13-46-32_129-jvmRun1.dumpstream
> {noformat}
> ...and then eventually times out.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-10413) DocumentStoreIndexerIT hanging
[
https://issues.apache.org/jira/browse/OAK-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762065#comment-17762065
]
Marcel Reutegger commented on OAK-10413:
PR #1099 (including changes from OAK-10427) ran all tests successfully. I
merged the changes into trunk. Let's wait for some more runs until we resolve
this issue.
> DocumentStoreIndexerIT hanging
> --
>
> Key: OAK-10413
> URL: https://issues.apache.org/jira/browse/OAK-10413
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: documentmk, run
>Reporter: Julian Reschke
>Priority: Major
>
> {noformat}
> [INFO] Running org.apache.jackrabbit.oak.index.DocumentStoreIndexerIT
> [WARNING] Corrupted STDOUT by directly writing to native stream in forked JVM
> 1. See FAQ web page and the dump file
> /home/jenkins/jenkins-agent/workspace/Jackrabbit/jackrabbit-oak-trunk/oak-run/target/failsafe-reports/2023-08-22T13-46-32_129-jvmRun1.dumpstream
> {noformat}
> ...and then eventually times out.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-10413) DocumentStoreIndexerIT hanging
[
https://issues.apache.org/jira/browse/OAK-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761867#comment-17761867
]
Marcel Reutegger commented on OAK-10413:
Proposed changes to PurgeOldIndexVersionIT:
https://github.com/apache/jackrabbit-oak/pull/1099
> DocumentStoreIndexerIT hanging
> --
>
> Key: OAK-10413
> URL: https://issues.apache.org/jira/browse/OAK-10413
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: documentmk, run
>Reporter: Julian Reschke
>Priority: Major
>
> {noformat}
> [INFO] Running org.apache.jackrabbit.oak.index.DocumentStoreIndexerIT
> [WARNING] Corrupted STDOUT by directly writing to native stream in forked JVM
> 1. See FAQ web page and the dump file
> /home/jenkins/jenkins-agent/workspace/Jackrabbit/jackrabbit-oak-trunk/oak-run/target/failsafe-reports/2023-08-22T13-46-32_129-jvmRun1.dumpstream
> {noformat}
> ...and then eventually times out.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-10413) DocumentStoreIndexerIT hanging
[
https://issues.apache.org/jira/browse/OAK-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761865#comment-17761865
]
Marcel Reutegger commented on OAK-10413:
See OAK-10427 for the changes related to MongoDocumentStore. The actual change
is in NodeDocumentCache.
> DocumentStoreIndexerIT hanging
> --
>
> Key: OAK-10413
> URL: https://issues.apache.org/jira/browse/OAK-10413
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: documentmk, run
>Reporter: Julian Reschke
>Priority: Major
>
> {noformat}
> [INFO] Running org.apache.jackrabbit.oak.index.DocumentStoreIndexerIT
> [WARNING] Corrupted STDOUT by directly writing to native stream in forked JVM
> 1. See FAQ web page and the dump file
> /home/jenkins/jenkins-agent/workspace/Jackrabbit/jackrabbit-oak-trunk/oak-run/target/failsafe-reports/2023-08-22T13-46-32_129-jvmRun1.dumpstream
> {noformat}
> ...and then eventually times out.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-10413) DocumentStoreIndexerIT hanging
[
https://issues.apache.org/jira/browse/OAK-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761864#comment-17761864
]
Marcel Reutegger commented on OAK-10413:
Looks like there is not much that can be done about FlatFileStore instances.
There is a list of FlatFileStore created by one of the tests with 24 instances.
The CachingSegmentReader are opened by Persistence used in
PurgeOldIndexVersionIT. The tests in there do not close Persistence in some
cases.
The MongoDocumentStore instances are actually closed, but referenced by
TraverseWithSortStrategy.MemoryListener. This (non-static) inner class
indirectly references a MongoDocumentStore. These listeners are added but never
removed from the MemoryMXBean. A quick fix would be to reduce memory of closed
MongoDocumentStore instances. Currently they are quite heavy weight because
caches they reference are not cleared on close/dispose.
With the two changes mentioned above, I can reliable run oak-run integration
tests in the sequence seen on Jenkins without a memory issue.
> DocumentStoreIndexerIT hanging
> --
>
> Key: OAK-10413
> URL: https://issues.apache.org/jira/browse/OAK-10413
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: documentmk, run
>Reporter: Julian Reschke
>Priority: Major
>
> {noformat}
> [INFO] Running org.apache.jackrabbit.oak.index.DocumentStoreIndexerIT
> [WARNING] Corrupted STDOUT by directly writing to native stream in forked JVM
> 1. See FAQ web page and the dump file
> /home/jenkins/jenkins-agent/workspace/Jackrabbit/jackrabbit-oak-trunk/oak-run/target/failsafe-reports/2023-08-22T13-46-32_129-jvmRun1.dumpstream
> {noformat}
> ...and then eventually times out.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-10413) DocumentStoreIndexerIT hanging
[
https://issues.apache.org/jira/browse/OAK-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761853#comment-17761853
]
Julian Reschke commented on OAK-10413:
--
Of these tests, {ReindexIT} is the one with the latest changes...
> DocumentStoreIndexerIT hanging
> --
>
> Key: OAK-10413
> URL: https://issues.apache.org/jira/browse/OAK-10413
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: documentmk, run
>Reporter: Julian Reschke
>Priority: Major
>
> {noformat}
> [INFO] Running org.apache.jackrabbit.oak.index.DocumentStoreIndexerIT
> [WARNING] Corrupted STDOUT by directly writing to native stream in forked JVM
> 1. See FAQ web page and the dump file
> /home/jenkins/jenkins-agent/workspace/Jackrabbit/jackrabbit-oak-trunk/oak-run/target/failsafe-reports/2023-08-22T13-46-32_129-jvmRun1.dumpstream
> {noformat}
> ...and then eventually times out.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-10413) DocumentStoreIndexerIT hanging
[
https://issues.apache.org/jira/browse/OAK-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761793#comment-17761793
]
Marcel Reutegger commented on OAK-10413:
The majority of memory is occupied by:
- 26 instances of FlatFileStore (144 MB)
- 6 instances of CachingSegmentReader (61 MB)
- 7 instances of MongoDocumentStore (44 MB)
I suspect some tests are not releasing resources.
> DocumentStoreIndexerIT hanging
> --
>
> Key: OAK-10413
> URL: https://issues.apache.org/jira/browse/OAK-10413
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: documentmk, run
>Reporter: Julian Reschke
>Priority: Major
>
> {noformat}
> [INFO] Running org.apache.jackrabbit.oak.index.DocumentStoreIndexerIT
> [WARNING] Corrupted STDOUT by directly writing to native stream in forked JVM
> 1. See FAQ web page and the dump file
> /home/jenkins/jenkins-agent/workspace/Jackrabbit/jackrabbit-oak-trunk/oak-run/target/failsafe-reports/2023-08-22T13-46-32_129-jvmRun1.dumpstream
> {noformat}
> ...and then eventually times out.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-10413) DocumentStoreIndexerIT hanging
[
https://issues.apache.org/jira/browse/OAK-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761775#comment-17761775
]
Marcel Reutegger commented on OAK-10413:
I'm able to reproduce when running tests in the same sequence like Jenkins
does. Added the following test and then specified it on the command line with
-Dtest=MyTest.
{noformat}
@RunWith(Suite.class)
@Suite.SuiteClasses({
PurgeOldIndexVersionIT.class,
LuceneIndexCommandIT.class,
ReindexIT.class,
DocumentStoreIndexerIT.class
})
public class MyTest {
}
{noformat}
Analyzing the heap dump...
> DocumentStoreIndexerIT hanging
> --
>
> Key: OAK-10413
> URL: https://issues.apache.org/jira/browse/OAK-10413
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: documentmk, run
>Reporter: Julian Reschke
>Priority: Major
>
> {noformat}
> [INFO] Running org.apache.jackrabbit.oak.index.DocumentStoreIndexerIT
> [WARNING] Corrupted STDOUT by directly writing to native stream in forked JVM
> 1. See FAQ web page and the dump file
> /home/jenkins/jenkins-agent/workspace/Jackrabbit/jackrabbit-oak-trunk/oak-run/target/failsafe-reports/2023-08-22T13-46-32_129-jvmRun1.dumpstream
> {noformat}
> ...and then eventually times out.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-10413) DocumentStoreIndexerIT hanging
[
https://issues.apache.org/jira/browse/OAK-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761171#comment-17761171
]
Mohit Kataria commented on OAK-10413:
-
Tested with following changes:
# Added Eventual assertions: Jenkins job failed
# Reverted changes for OAK-10273 and OAK-10381 : jenkins job failed
So looks like this test was flaky from start. As there are also OOME in some
runs, job's resource constraint may be leading to test failure.
On local with sufficient resource tests are always passing.
> DocumentStoreIndexerIT hanging
> --
>
> Key: OAK-10413
> URL: https://issues.apache.org/jira/browse/OAK-10413
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: documentmk, run
>Reporter: Julian Reschke
>Priority: Major
>
> {noformat}
> [INFO] Running org.apache.jackrabbit.oak.index.DocumentStoreIndexerIT
> [WARNING] Corrupted STDOUT by directly writing to native stream in forked JVM
> 1. See FAQ web page and the dump file
> /home/jenkins/jenkins-agent/workspace/Jackrabbit/jackrabbit-oak-trunk/oak-run/target/failsafe-reports/2023-08-22T13-46-32_129-jvmRun1.dumpstream
> {noformat}
> ...and then eventually times out.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-10413) DocumentStoreIndexerIT hanging
[
https://issues.apache.org/jira/browse/OAK-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760922#comment-17760922
]
Mohit Kataria commented on OAK-10413:
-
Looks like a concurrency issue. With eventual Assertion the tests are passing.
Old implementation of reindexing via oak-run used to take more time as
unnecessary data was serialised giving system more time and passing assertion.
> DocumentStoreIndexerIT hanging
> --
>
> Key: OAK-10413
> URL: https://issues.apache.org/jira/browse/OAK-10413
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: documentmk, run
>Reporter: Julian Reschke
>Priority: Major
>
> {noformat}
> [INFO] Running org.apache.jackrabbit.oak.index.DocumentStoreIndexerIT
> [WARNING] Corrupted STDOUT by directly writing to native stream in forked JVM
> 1. See FAQ web page and the dump file
> /home/jenkins/jenkins-agent/workspace/Jackrabbit/jackrabbit-oak-trunk/oak-run/target/failsafe-reports/2023-08-22T13-46-32_129-jvmRun1.dumpstream
> {noformat}
> ...and then eventually times out.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-10413) DocumentStoreIndexerIT hanging
[
https://issues.apache.org/jira/browse/OAK-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758433#comment-17758433
]
Julian Reschke commented on OAK-10413:
--
(failing again)
> DocumentStoreIndexerIT hanging
> --
>
> Key: OAK-10413
> URL: https://issues.apache.org/jira/browse/OAK-10413
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: documentmk
>Reporter: Julian Reschke
>Priority: Major
>
> {noformat}
> [INFO] Running org.apache.jackrabbit.oak.index.DocumentStoreIndexerIT
> [WARNING] Corrupted STDOUT by directly writing to native stream in forked JVM
> 1. See FAQ web page and the dump file
> /home/jenkins/jenkins-agent/workspace/Jackrabbit/jackrabbit-oak-trunk/oak-run/target/failsafe-reports/2023-08-22T13-46-32_129-jvmRun1.dumpstream
> {noformat}
> ...and then eventually times out.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-10413) DocumentStoreIndexerIT hanging
[
https://issues.apache.org/jira/browse/OAK-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17757557#comment-17757557
]
Julian Reschke commented on OAK-10413:
--
[~tihom88] - could you please have a look?
> DocumentStoreIndexerIT hanging
> --
>
> Key: OAK-10413
> URL: https://issues.apache.org/jira/browse/OAK-10413
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: documentmk
>Reporter: Julian Reschke
>Priority: Major
>
> {noformat}
> [INFO] Running org.apache.jackrabbit.oak.index.DocumentStoreIndexerIT
> [WARNING] Corrupted STDOUT by directly writing to native stream in forked JVM
> 1. See FAQ web page and the dump file
> /home/jenkins/jenkins-agent/workspace/Jackrabbit/jackrabbit-oak-trunk/oak-run/target/failsafe-reports/2023-08-22T13-46-32_129-jvmRun1.dumpstream
> {noformat}
> ...and then eventually times out.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
