Hi all,
We are indexing documents using apache lucene using several parallel
indexing pipelines(java process) to NFS mounted directory.
All of them follows same code and workflow most of the pipelines succeeds
without any issue, but only only few indexing pipelines remains in idle and
in RUN state forever , we observed the thread dump as well , it's not moving
at all.
if anyone is facing this issue and found solution do share with me
Note: We are using LSF cluster for our parallel indexings(java process) and
we launch these jobs using dynamic resource like cpu and memory , but
indexing process of individual lucene index serve by single host
Thread dump :
2021-05-10 09:26:22
Full thread dump OpenJDK 64-Bit Server VM (11.0.4+11 mixed mode):
Threads class SMR info:
_java_thread_list=0x00002b9174000df0, length=14, elements={
0x00002b90b8012000, 0x00002b90ba0b5000, 0x00002b90ba0b9000,
0x00002b90ba0cc000,
0x00002b90ba0ce000, 0x00002b90ba0d0000, 0x00002b90ba0d2000,
0x00002b90ba130000,
0x00002b90ba144000, 0x00002b90ba807800, 0x00002b90ba817000,
0x00002b9140001000,
0x00002b9168019800, 0x00002b916801e800
}
"main" #1 prio=5 os_prio=0 cpu=17492.51ms elapsed=24411.44s
tid=0x00002b90b8012000 nid=0x600f1 runnable [0x00002b90b423a000]
java.lang.Thread.State: RUNNABLE
at sun.nio.fs.UnixNativeDispatcher.stat0(java.base@11.0.4/Native
Method)
at
sun.nio.fs.UnixNativeDispatcher.stat(java.base@11.0.4/UnixNativeDispatcher.java:291)
at
sun.nio.fs.UnixFileAttributes.get(java.base@11.0.4/UnixFileAttributes.java:70)
at
sun.nio.fs.UnixFileStore.devFor(java.base@11.0.4/UnixFileStore.java:57)
at
sun.nio.fs.UnixFileStore.<init>(java.base@11.0.4/UnixFileStore.java:72)
at
sun.nio.fs.LinuxFileStore.<init>(java.base@11.0.4/LinuxFileStore.java:53)
at
sun.nio.fs.LinuxFileSystem.getFileStore(java.base@11.0.4/LinuxFileSystem.java:112)
at
sun.nio.fs.UnixFileSystem$FileStoreIterator.readNext(java.base@11.0.4/UnixFileSystem.java:212)
at
sun.nio.fs.UnixFileSystem$FileStoreIterator.hasNext(java.base@11.0.4/UnixFileSystem.java:223)
- locked <0x00000007e75ab7e0> (a
sun.nio.fs.UnixFileSystem$FileStoreIterator)
at org.apache.lucene.util.IOUtils.getFileStore(IOUtils.java:595)
at org.apache.lucene.util.IOUtils.spinsLinux(IOUtils.java:539)
at org.apache.lucene.util.IOUtils.spins(IOUtils.java:528)
at org.apache.lucene.util.IOUtils.spins(IOUtils.java:503)
at
org.apache.lucene.index.ConcurrentMergeScheduler.initDynamicDefaults(ConcurrentMergeScheduler.java:412)
- locked <0x00000007e7146348> (a
org.apache.lucene.index.ConcurrentMergeScheduler)
at
org.apache.lucene.index.ConcurrentMergeScheduler.merge(ConcurrentMergeScheduler.java:501)
- locked <0x00000007e7146348> (a
org.apache.lucene.index.ConcurrentMergeScheduler)
at
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2158)
at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:548)
at
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:116)
at
org.apache.lucene.index.ReaderManager.<init>(ReaderManager.java:72)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.initReaderManager(DirectoryTaxonomyWriter.java:279)
- locked <0x00000007e70bbff8> (a
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.getTaxoArrays(DirectoryTaxonomyWriter.java:749)
- locked <0x00000007e70bbff8> (a
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCategoryDocument(DirectoryTaxonomyWriter.java:508)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.internalAddCategory(DirectoryTaxonomyWriter.java:462)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCategory(DirectoryTaxonomyWriter.java:429)
- locked <0x00000007e70bbff8> (a
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.<init>(DirectoryTaxonomyWriter.java:209)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.<init>(DirectoryTaxonomyWriter.java:293)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.<init>(DirectoryTaxonomyWriter.java:309)
at
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing.createTaxonomyWriter(IndexStepIndexing.java:304)
at
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing.prepareIndexWriter(IndexStepIndexing.java:217)
- locked <0x00000007ebf96b70> (a
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing)
at
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing.prepareIndexWriter(IndexStepIndexing.java:206)
- locked <0x00000007ebf96b70> (a
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing)
at
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing.index(IndexStepIndexing.java:133)
at
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing.process(IndexStepIndexing.java:80)
at
uk.ac.ebi.ebinocle.indexer.steps.substeps.IndexStepIndexing.process(IndexStepIndexing.java:61)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline.lambda$pipe$0(Pipeline.java:28)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline$$Lambda$202/0x000000080041f840.process(Unknown
Source)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline.lambda$pipe$0(Pipeline.java:28)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline$$Lambda$202/0x000000080041f840.process(Unknown
Source)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline.lambda$pipe$0(Pipeline.java:28)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline$$Lambda$202/0x000000080041f840.process(Unknown
Source)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline.lambda$pipe$0(Pipeline.java:28)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline$$Lambda$202/0x000000080041f840.process(Unknown
Source)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline.execute(Pipeline.java:32)
at
uk.ac.ebi.ebinocle.indexer.steps.IndexStep.process(IndexStep.java:54)
at
uk.ac.ebi.ebinocle.indexer.steps.IndexStep.process(IndexStep.java:39)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline.lambda$pipe$0(Pipeline.java:28)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline$$Lambda$202/0x000000080041f840.process(Unknown
Source)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline.lambda$pipe$0(Pipeline.java:28)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline$$Lambda$202/0x000000080041f840.process(Unknown
Source)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline.lambda$pipe$0(Pipeline.java:28)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline$$Lambda$202/0x000000080041f840.process(Unknown
Source)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline.lambda$pipe$0(Pipeline.java:28)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline$$Lambda$202/0x000000080041f840.process(Unknown
Source)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline.lambda$pipe$0(Pipeline.java:28)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline$$Lambda$202/0x000000080041f840.process(Unknown
Source)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.Pipeline.execute(Pipeline.java:32)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.LsfIndexerJob.run(LsfIndexerJob.java:110)
at
org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:800)
at
org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:784)
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:338)
at
uk.ac.ebi.ebinocle.indexer.indexingjob.LsfIndexerJob.main(LsfIndexerJob.java:67)
Locked ownable synchronizers:
- None
Regards,
Prasad
--
Sent from: https://lucene.472066.n3.nabble.com/Lucene-Java-Users-f532864.html
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
