I do have a test that will reproduce. I'm not adding categories with very deep hierarchies, I'm adding 129 category paths per document (all docs have paths with same label) with each path having one value. All of the values are completely random and likely unique. It's basically a worst case test for our app but the condition has been seen in the field (the error has been encountered at less than worst case scenario). The test i have reproduces it very quickly, Only have to index ~330K docs.
On Fri, Jan 17, 2014 at 3:27 PM, Shai Erera <ser...@gmail.com> wrote: > Do you have a test which reproduces the error? Are you adding categories > with very deep hierarchies? > > Shai > > > On Fri, Jan 17, 2014 at 11:59 PM, Matthew Petersen <mdpe...@gmail.com > >wrote: > > > I've confirmed that using the LruTaxonomyWriterCache solves the issue for > > me. It would appear there is in fact a bug in the > Cl20TaxonomyWriterCache > > or I am using it incorrectly (I use it as default, no customization). > > > > > > On Fri, Jan 17, 2014 at 9:29 AM, Matthew Petersen <mdpe...@gmail.com> > > wrote: > > > > > I'm sure. I had seen that issue and it looked similar but the stack > > trace > > > is slightly different. I've found that if I replace the > > > Cl2oTaxonomyWriterCache with the LruTaxonomyWriterCache the problem > seems > > > to go away. I'm working right now on running a test that will prove > this > > > but it takes a while as the cache needs to get very large. If this > > proves > > > to solve the problem then I'd say there is still a bug in the > > > Cl2oTaxonomyWriterCache implementation. > > > > > > Thanks for the response. > > > Matt > > > > > > > > > On Fri, Jan 17, 2014 at 6:36 AM, Michael McCandless < > > > luc...@mikemccandless.com> wrote: > > > > > >> Are you sure you're using 4.4? > > >> > > >> Because ... this looks like > > >> https://issues.apache.org/jira/browse/LUCENE-5048 but that was > > >> supposedly fixed in 4.4. > > >> > > >> Mike McCandless > > >> > > >> http://blog.mikemccandless.com > > >> > > >> > > >> On Thu, Jan 16, 2014 at 5:33 PM, Matthew Petersen <mdpe...@gmail.com> > > >> wrote: > > >> > I’m having an issue with an index when adding category paths to a > > >> document. > > >> > They seem to be added without issue for a long period of time, then > > for > > >> > some unknown reason the addition fails with an ArrayIndexOutOfBounds > > >> > exception. Subsequent attempts to add category paths fail with the > > same > > >> > exception. I’ve run CheckIndex on both the index and the taxonomy > > >> > directory and both come back as clean with no issues. I cannot fix > > the > > >> > index because according to lucene it is not broken. Could this be a > > >> bug in > > >> > lucene? Below is the stack trace when the exception occurs: > > >> > > > >> > > > >> > Lucene v4.4.0 > > >> > > > >> > > > >> > java.lang.ArrayIndexOutOfBoundsException: -65535 > > >> > > > >> > at java.util.ArrayList.elementData(ArrayList.java:371) > > >> > > > >> > at java.util.ArrayList.get(ArrayList.java:384) > > >> > > > >> > at > > >> > > > >> > > > org.apache.lucene.facet.taxonomy.writercache.cl2o.CharBlockArray.charAt(CharBlockArray.java:152) > > >> > > > >> > at > > >> > > > >> > > > org.apache.lucene.facet.taxonomy.writercache.cl2o.CategoryPathUtils.equalsToSerialized(CategoryPathUtils.java:61) > > >> > > > >> > at > > >> > > > >> > > > org.apache.lucene.facet.taxonomy.writercache.cl2o.CompactLabelToOrdinal.getOrdinal(CompactLabelToOrdinal.java:257) > > >> > > > >> > at > > >> > > > >> > > > org.apache.lucene.facet.taxonomy.writercache.cl2o.CompactLabelToOrdinal.getOrdinal(CompactLabelToOrdinal.java:140) > > >> > > > >> > at > > >> > > > >> > > > org.apache.lucene.facet.taxonomy.writercache.cl2o.Cl2oTaxonomyWriterCache.get(Cl2oTaxonomyWriterCache.java:74) > > >> > > > >> > at > > >> > > > >> > > > org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.addCategory(DirectoryTaxonomyWriter.java:455) > > >> > > > >> > at > > >> > > org.apache.lucene.facet.index.FacetFields.addFields(FacetFields.java:175) > > >> > > > >> > at > > >> > > > >> > > > com.logrhythm.messaging.indexing.LogIndexerImpl.getDocument(LogIndexerImpl.java:478) > > >> > > > >> > at > > >> > > > >> > > > com.logrhythm.messaging.indexing.LogIndexerImpl.indexLog(LogIndexerImpl.java:392) > > >> > > > >> > at > > >> > > > >> > > > com.logrhythm.messaging.indexing.LogIndexerImpl.indexLogs(LogIndexerImpl.java:357) > > >> > > > >> > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > > >> > > > >> > at > > >> > > > >> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > >> > > > >> > at java.lang.reflect.Method.invoke(Method.java:601) > > >> > > > >> > at > > >> > > > >> > > > com.logrhythm.tests.unit.messaging.indexing.LogIndexerTests.logIndexerLoadTest(LogIndexerTests.java:752) > > >> > > > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > >> > > > >> > at > > >> > > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > >> > > > >> > at > > >> > > > >> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > >> > > > >> > at java.lang.reflect.Method.invoke(Method.java:601) > > >> > > > >> > at > > >> > > > >> > > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) > > >> > > > >> > at > > >> > > > >> > > > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > > >> > > > >> > at > > >> > > > >> > > > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) > > >> > > > >> > at > > >> > > > >> > > > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) > > >> > > > >> > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) > > >> > > > >> > at > > >> > > > >> > > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) > > >> > > > >> > at > > >> > > > >> > > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) > > >> > > > >> > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) > > >> > > > >> > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) > > >> > > > >> > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) > > >> > > > >> > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) > > >> > > > >> > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) > > >> > > > >> > at org.junit.runners.ParentRunner.run(ParentRunner.java:300) > > >> > > > >> > at org.junit.runner.JUnitCore.run(JUnitCore.java:157) > > >> > > > >> > at > > >> > > > >> > > > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74) > > >> > > > >> > at > > >> > > > >> > > > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:202) > > >> > > > >> > at > > >> > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65) > > >> > > > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > >> > > > >> > at > > >> > > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > >> > > > >> > at > > >> > > > >> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > >> > > > >> > at java.lang.reflect.Method.invoke(Method.java:601) > > >> > > > >> > at > > com.intellij.rt.execution.application.AppMain.main(AppMain.java:120) > > >> > > >> --------------------------------------------------------------------- > > >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > >> For additional commands, e-mail: java-user-h...@lucene.apache.org > > >> > > >> > > > > > >
