Note that the -Xint flag will make your code run tremendously more slowly! Likely to the point of not really being usable. But it'd be interesting to see if that side-steps the bug.
Is it possible to test with OpenJDK as well? The BKDWriter code is quite complex, so it is also possible there is a Lucene bug at work. Can you open an issue in Lucene's jira and we can iterate there? Thanks, Mike McCandless http://blog.mikemccandless.com On Wed, May 1, 2019 at 9:34 AM Torben Riis <t...@multi-support.com> wrote: > Hi, > > > > I’m a bit stuck here and needs a clue or two in order to continue our > investigations. Hope that someone can help. :) > > > > Periodically, around once a month, we get the below > ArrayIndexOutOfBoundsException on our system. We use multiple indexes and > the error can originate from any of them, but the error always occurs in > line 1217 in BKDWriter (during a System.arraycopy). > > > > We found a couple of issues on the net regarding JIT optimization problem > related to J9, but they all looks like they have been resolved and cannot > be reproduced anymore. But nevertheless, we have just added the -Xint flag > (disable JIT compiler) in order to see whether this has any impact. > Unfortunately we do not have the result of this yet, but I’ll of course > post it when it is known. > > > > Are there any of you clever guys out there, that has some good ideas > further investigations? Or have seen such issue before? > > > > We are using Lucene 6.6.0 and runs on IBM J9 on the IBM I platform. > > > > > > *Java version* > > java version "1.8.0_191" > > Java(TM) SE Runtime Environment (build 8.0.5.25 - > pap6480sr5fp25-20181030_01(SR5 FP25)) > > IBM J9 VM (build 2.9, JRE 1.8.0 OS/400 ppc64-64-Bit Compressed References > 20181029_400846 (JIT enabled, AOT enabled) > > OpenJ9 - c5c78da > > OMR - 3d5ac33 > > IBM - 8c1bdc2) > > JCL - 20181022_01 based on Oracle jdk8u191-b26 > > NOTICE: If no version information is found above, this could indicate a > corrupted Java installation! > > Java detected was: /QOpenSys/QIBM/ProdData/JavaVM/jdk80/64bit/bin/java > > -Dmultiarchive.basepath=/home/NEXTOWN/Multi-Support/Next -Xms128m -Xmx2048m > > > > > > *Stacktrace* > > Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:07.970 CEST > [Lucene Merge Thread #0] > org.apache.lucene.index.MergePolicy$MergeException: > java.lang.ArrayIndexOutOfBoundsException > > at > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703) > > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683) > > Caused by: 2019-05-01T06:10:07.971 CEST [Lucene Merge Thread #0] > java.lang.ArrayIndexOutOfBoundsException > > at > org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217) > > at > org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197) > > at > org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078) > > at > org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245) > > at > org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82) > > at > org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWriter.java:648) > > at > org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560) > > at > org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:212) > > at > org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:173) > > at > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:122) > > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356) > > at > org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931) > > at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) > > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661) > > Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:08.075 CEST > [Lucene Merge Thread #0] > org.apache.lucene.index.MergePolicy$MergeException: > java.lang.ArrayIndexOutOfBoundsException > > at > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703) > > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683) > > Caused by: 2019-05-01T06:10:08.076 CEST [Lucene Merge Thread #0] > java.lang.ArrayIndexOutOfBoundsException > > at > org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217) > > at > org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197) > > at > org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078) > > at > org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245) > > at > org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82) > > at > org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWriter.java:648) > > at > org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560) > > at > org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:212) > > at > org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:173) > > at > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:122) > > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356) > > at > org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931) > > at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) > > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler > > > > > > *BKDWriter.java* > > > > > > > Kind regards > > Torben Riis > > ————————————————————————— > Torben Riis > Lead Architect > > > Multi Support · Making good business run better > +45 96 600 600 · www.multi-support.com > > > > > > >