[GitHub] [lucene-solr] mikemccand commented on pull request #1623: LUCENE-8962: Merge segments on getReader
mikemccand commented on pull request #1623: URL: https://github.com/apache/lucene-solr/pull/1623#issuecomment-679088291 > @mikemccand this test reproduces on master too I opened https://issues.apache.org/jira/browse/LUCENE-9477 Great, thanks @s1monw! Given that beasting is now uncovering pre-existing issues I think we should push this PR! Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on pull request #1623: LUCENE-8962: Merge segments on getReader
mikemccand commented on pull request #1623: URL: https://github.com/apache/lucene-solr/pull/1623#issuecomment-678796112 Hmm, not that I am using the larger (`2000mb`) line docs file, and that might be needed to provoke failure. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on pull request #1623: LUCENE-8962: Merge segments on getReader
mikemccand commented on pull request #1623: URL: https://github.com/apache/lucene-solr/pull/1623#issuecomment-678796020 > @mikemccand I think we are ready here WDYT? +1! Thanks @s1monw! I will continue beasting. The failures are very rare now ... I ran 196 iters of slow + nightly Lucene core + modules tests, and hit only ~4 interesting failures. E.g. this failing seed repros on the PR but not on mainline: ``` org.apache.lucene.index.TestIndexWriterOnVMError > testUnknownError FAILED org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(MockIndexInputWrapper((clone of) ByteBuffersIndexInput (file=pending_segments_2, buffers\ =258 bytes, block size: 1, blocks: 1, position: 0 at __randomizedtesting.SeedInfo.seed([587A104EFE0C57E1:B32CCFCEFC8BC1D1]:0) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:300) at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:521) at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:301) at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:836) at org.apache.lucene.index.TestIndexWriterOnVMError.doTest(TestIndexWriterOnVMError.java:89) at org.apache.lucene.index.TestIndexWriterOnVMError.testUnknownError(TestIndexWriterOnVMError.java:251) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at
[GitHub] [lucene-solr] mikemccand commented on pull request #1623: LUCENE-8962: Merge segments on getReader
mikemccand commented on pull request #1623: URL: https://github.com/apache/lucene-solr/pull/1623#issuecomment-677710206 I will restart beasting on the latest PR now. Thanks @s1monw! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on pull request #1623: LUCENE-8962: Merge segments on getReader
mikemccand commented on pull request #1623: URL: https://github.com/apache/lucene-solr/pull/1623#issuecomment-67545 Those were the only two failures found after 1033 test iterations! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on pull request #1623: LUCENE-8962: Merge segments on getReader
mikemccand commented on pull request #1623:
URL: https://github.com/apache/lucene-solr/pull/1623#issuecomment-675459732
Another failure that reproduces only on the PR:
```
[junit4:pickseed] Seed property 'tests.seed' already defined:
CADC6D7945159855
[junit4] says jolly good day! Master seed: CADC6D7945159855
[junit4] Executing 1 suite with 1 JVM.
[junit4]
[junit4] Started J0 PID(1047056@localhost).
[junit4] Suite: org.apache.lucene.spatial.prefix.NumberRangeFacetsTest
[junit4] OK 0.37s | NumberRangeFacetsTest.test
{seed=[CADC6D7945159855:428852A3EBE9F5AD]}
[junit4] OK 0.07s | NumberRangeFacetsTest.test
{seed=[CADC6D7945159855:F6DEEE5FDF2B3E81]}
[junit4] OK 0.02s | NumberRangeFacetsTest.test
{seed=[CADC6D7945159855:783778838EEF764A]}
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=NumberRangeFacetsTest -Dtests.method=test
-Dtests.seed=CADC6D7945159855 -Dtests.slow=true -Dtests.badapples=true
-Dtests.locale=lu-CD -Dtests.\
timezone=America/Iqaluit -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.10s | NumberRangeFacetsTest.test
{seed=[CADC6D7945159855:49D9D366E2112D63]} <<<
[junit4]> Throwable #1: java.nio.file.NoSuchFileException: _3.fdx
[junit4]>at
org.apache.lucene.store.ByteBuffersDirectory.deleteFile(ByteBuffersDirectory.java:148)
[junit4]>at
org.apache.lucene.store.MockDirectoryWrapper.deleteFile(MockDirectoryWrapper.java:607)
[junit4]>at
org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:38)
[junit4]>at
org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:696)
[junit4]>at
org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:690)
[junit4]>at
org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:589)
[junit4]>at
org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:620)
[junit4]>at
org.apache.lucene.index.IndexWriter.decRefDeleter(IndexWriter.java:5354)
[junit4]>at
org.apache.lucene.index.StandardDirectoryReader.lambda$doClose$1(StandardDirectoryReader.java:370)
[junit4]>at
org.apache.lucene.index.StandardDirectoryReader.doClose(StandardDirectoryReader.java:384)
[junit4]>at
org.apache.lucene.index.IndexReader.decRef(IndexReader.java:244)
[junit4]>at
org.apache.lucene.index.IndexReader.close(IndexReader.java:385)
[junit4]>at
org.apache.lucene.util.IOUtils.close(IOUtils.java:89)
[junit4]>at
org.apache.lucene.util.IOUtils.close(IOUtils.java:77)
[junit4]>at
org.apache.lucene.spatial.SpatialTestCase.commit(SpatialTestCase.java:97)
[junit4]>at
org.apache.lucene.spatial.prefix.NumberRangeFacetsTest.test(NumberRangeFacetsTest.java:92)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4]>at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4]>at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
[junit4]>at
java.base/java.lang.Thread.run(Thread.java:834)Throwable #2:
java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still
2 open files: {_5.cfs=1, _4.cfs=1}
[junit4]>at
org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:812)
[junit4]>at
org.apache.lucene.util.IOUtils.close(IOUtils.java:89)
[junit4]>at
org.apache.lucene.util.IOUtils.close(IOUtils.java:77)
[junit4]>at
org.apache.lucene.spatial.SpatialTestCase.tearDown(SpatialTestCase.java:72)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4]>at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4]>at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
[junit4]>at java.base/java.lang.Thread.run(Thread.java:834)
[junit4]> Caused by: java.lang.RuntimeException: unclosed IndexInput:
_4.cfs
[junit4]>at
org.apache.lucene.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:730)
[junit4]>at
org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:773)
[junit4]>
[GitHub] [lucene-solr] mikemccand commented on pull request #1623: LUCENE-8962: Merge segments on getReader
mikemccand commented on pull request #1623:
URL: https://github.com/apache/lucene-solr/pull/1623#issuecomment-675458466
Beasting ran all night (1031 iterations) and uncovered this failure, which
reproduces with this PR but not on mainline:
```
[mkdir] Created dir: /l/simon/lucene/build/core/test
[junit4:pickseed] Seed property 'tests.seed' already defined:
313A1CF00C235D4F
[mkdir] Created dir: /l/simon/lucene/build/core/test/temp
[mkdir] Created dir: /l/simon/.caches/test-stats/core
[junit4] says hi! Master seed: 313A1CF00C235D4F
[junit4] Executing 1 suite with 1 JVM.
[junit4]
[junit4] Started J0 PID(972126@localhost).
[junit4] Suite:
org.apache.lucene.codecs.perfield.TestPerFieldDocValuesFormat
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestPerFieldDocValuesFormat
-Dtests.method=testSparseBinaryVariableLengthVsStoredFields
-Dtests.seed=313A1CF00C235D4F -Dtests.slow=true -Dtest\
s.badapples=true -Dtests.locale=ur -Dtests.timezone=Mexico/BajaNorte
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.43s |
TestPerFieldDocValuesFormat.testSparseBinaryVariableLengthVsStoredFields <<<
[junit4]> Throwable #1: java.nio.file.NoSuchFileException: _j.si
[junit4]>at
__randomizedtesting.SeedInfo.seed([313A1CF00C235D4F:5C414B294A4A9C4D]:0)
[junit4]>at
org.apache.lucene.store.ByteBuffersDirectory.deleteFile(ByteBuffersDirectory.java:148)
[junit4]>at
org.apache.lucene.store.MockDirectoryWrapper.deleteFile(MockDirectoryWrapper.java:607)
[junit4]>at
org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:38)
[junit4]>at
org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:696)
[junit4]>at
org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:690)
[junit4]>at
org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:589)
[junit4]>at
org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:620)
[junit4]>at
org.apache.lucene.index.IndexWriter.decRefDeleter(IndexWriter.java:5354)
[junit4]>at
org.apache.lucene.index.StandardDirectoryReader.lambda$doClose$1(StandardDirectoryReader.java:370)
[junit4]>at
org.apache.lucene.index.StandardDirectoryReader.doClose(StandardDirectoryReader.java:384)
[junit4]>at
org.apache.lucene.index.IndexReader.decRef(IndexReader.java:244)
[junit4]>at
org.apache.lucene.index.IndexReader.close(IndexReader.java:385)
[junit4]>at
org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestBinaryVsStoredFields(BaseDocValuesFormatTestCase.java:1527)
[junit4]>at
org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestBinaryVariableLengthVsStoredFields(BaseDocValuesFormatTestCase.java:1585)
[junit4]>at
org.apache.lucene.index.BaseDocValuesFormatTestCase.testSparseBinaryVariableLengthVsStoredFields(BaseDocValuesFormatTestCase.java:1579)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4]>at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4]>at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
[junit4]>at java.base/java.lang.Thread.run(Thread.java:834)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene86): {},
docValues:{}, maxPointsInLeafNode=1657, maxMBSortInHeap=6.829528977246159,
sim=Asserting(RandomSimilarity(queryNorm=true): {}), loc\
ale=ur, timezone=Mexico/BajaNorte
[junit4] 2> NOTE: Linux 5.5.6-arch1-1 amd64/Oracle Corporation 11.0.6
(64-bit)/cpus=128,threads=1,free=516818968,total=536870912
[junit4] 2> NOTE: All tests run in this JVM:
[TestPerFieldDocValuesFormat]
[junit4] Completed [1/1 (1!)] in 0.62s, 1 test, 1 error <<< FAILURES!
[junit4]
[junit4]
[junit4] Tests with failures [seed: 313A1CF00C235D4F]:
[junit4] -
org.apache.lucene.codecs.perfield.TestPerFieldDocValuesFormat.testSparseBinaryVariableLengthVsStoredFields
[junit4]
[junit4]
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail:
[GitHub] [lucene-solr] mikemccand commented on pull request #1623: LUCENE-8962: Merge segments on getReader
mikemccand commented on pull request #1623: URL: https://github.com/apache/lucene-solr/pull/1623#issuecomment-674391203 919 iterations and only this failure! ``` 1> TEST: now get reader > java.io.IOException: MockDirectoryWrapper: file "_3.cfs" is still open: cannot delete > at __randomizedtesting.SeedInfo.seed([9EEEDD0028F78B57:7E45B4FC8975A53E]:0) > at org.apache.lucene.store.MockDirectoryWrapper.deleteFile(MockDirectoryWrapper.java:600) > at org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:38) > at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:696) > at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:690) > at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:589) > at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:531) > at org.apache.lucene.index.IndexWriter.checkpoint(IndexWriter.java:2625) > at org.apache.lucene.index.IndexWriter.commitMerge(IndexWriter.java:4201) > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4818) > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4265) > at org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:5816) > at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40) > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:639) > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:471) > at org.apache.lucene.index.TestIndexWriterReader.doTestIndexWriterReopenSegment(TestIndexWriterReader.java:573) > at org.apache.lucene.index.TestIndexWriterReader.testIndexWriterReopenSegmentFullMerge(TestIndexWriterReader.java:523) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754) > at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942) > at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978) > at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992) > at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370) > at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819) > at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470) > at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951) > at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836) > at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887) > at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898) > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > at
[GitHub] [lucene-solr] mikemccand commented on pull request #1623: LUCENE-8962: Merge segments on getReader
mikemccand commented on pull request #1623: URL: https://github.com/apache/lucene-solr/pull/1623#issuecomment-674270585 256 iterations and no failures yet! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on pull request #1623: LUCENE-8962: Merge segments on getReader
mikemccand commented on pull request #1623: URL: https://github.com/apache/lucene-solr/pull/1623#issuecomment-674118327 > @mikemccand nevermind I think I found the issue. I think it's ready now. Yay! I will kick off new beasting (of all Lucene core + module tests) on my 128 core box! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on pull request #1623: LUCENE-8962: Merge segments on getReader
mikemccand commented on pull request #1623: URL: https://github.com/apache/lucene-solr/pull/1623#issuecomment-673727019 Ugh, also these test failures that I put onto the wrong PR: https://github.com/apache/lucene-solr/pull/1743#issuecomment-673726633 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on pull request #1623: LUCENE-8962: Merge segments on getReader
mikemccand commented on pull request #1623: URL: https://github.com/apache/lucene-solr/pull/1623#issuecomment-673725282 > I kicked off beasting of all Lucene (core + modules) tests with this change ... no failures yet after 31 iterations. OK, it has run 1061 iterations now of all Lucene (core + modules) tests, and some interesting (~15) failures: ``` ant test -Dtestcase=TestIndexWriter -Dtests.method=testRandomOperations -Dtests.seed=432EB011B0898067 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=pt-ST -Dtests.timezone=US/Mountain -Dtests.asserts=true -Dtest\ s.file.encoding=UTF-8 [junit4] 2> ago 13, 2020 3:37:55 DA TARDE com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException [junit4] 2> WARNING: Uncaught exception in thread: Thread[Thread-0,5,TGRP-TestIndexWriter] [junit4] 2> java.lang.AssertionError: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot commit [junit4] 2>at __randomizedtesting.SeedInfo.seed([432EB011B0898067]:0) [junit4] 2>at org.apache.lucene.index.TestIndexWriter.lambda$testRandomOperations$48(TestIndexWriter.java:3886) [junit4] 2>at java.base/java.lang.Thread.run(Thread.java:834) [junit4] 2> Caused by: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot commit [junit4] 2>at org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4930) [junit4] 2>at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3365) [junit4] 2>at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3664) [junit4] 2>at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3622) [junit4] 2>at org.apache.lucene.index.TestIndexWriter.lambda$testRandomOperations$48(TestIndexWriter.java:3879) [junit4] 2>... 1 more [junit4] 2>Suppressed: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception [junit4] 2>at org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:349) [junit4] 2>at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:669) [junit4] 2>at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:589) [junit4] 2>at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3375) [junit4] 2>... 4 more [junit4] 2>Caused by: org.apache.lucene.index.CorruptIndexException: Problem reading index from MockDirectoryWrapper(ByteBuffersDirectory@ebddcb6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2bd239b2) (resource=MockDir\ ectoryWrapper(ByteBuffersDirectory@ebddcb6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2bd239b2)) [junit4] 2>at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:142) [junit4] 2>at org.apache.lucene.index.SegmentReader.(SegmentReader.java:83) [junit4] 2>at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:171) [junit4] 2>at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:213) [junit4] 2>at org.apache.lucene.index.IndexWriter.lambda$getReader$0(IndexWriter.java:568) [junit4] 2>at org.apache.lucene.index.IndexWriter.lambda$getReader$1(IndexWriter.java:614) [junit4] 2>at org.apache.lucene.index.IndexWriter$2.onMergeComplete(IndexWriter.java:3461) [junit4] 2>at org.apache.lucene.index.IndexWriter.commitMerge(IndexWriter.java:4078) [junit4] 2>at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4697) [junit4] 2>at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4257) [junit4] 2>at org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:5808) [junit4] 2>at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) [junit4] 2>at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:682) [junit4] 2>Caused by: java.io.FileNotFoundException: _1m.fnm in dir=ByteBuffersDirectory@ebddcb6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2bd239b2 [junit4] 2>at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:748) [junit4] 2>at
[GitHub] [lucene-solr] mikemccand commented on pull request #1623: LUCENE-8962: Merge segments on getReader
mikemccand commented on pull request #1623: URL: https://github.com/apache/lucene-solr/pull/1623#issuecomment-673112665 Awesome, thanks @s1monw! I will try to have a look soon. I kicked off beasting of all Lucene (core + modules) tests with this change ... no failures yet after 31 iterations. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on pull request #1623: LUCENE-8962: Merge segments on getReader
mikemccand commented on pull request #1623: URL: https://github.com/apache/lucene-solr/pull/1623#issuecomment-651167468 A different, reproducing, test failure, likely from the same cause: ``` > java.lang.AssertionError > at __randomizedtesting.SeedInfo.seed([C64EB0BA0CE0061F:FB961E96340E586F]:0) > at org.apache.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:5026) > at org.apache.lucene.index.IndexWriter.tragicEvent(IndexWriter.java:5019) > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4214) > at org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:5735) > at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40) > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:581) > at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103) > at org.apache.lucene.index.TestIndexWriterExceptions2.testBasics(TestIndexWriterExceptions2.java:205) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754) > at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942) > at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978) > at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992) > at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370) > at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819) > at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470) > at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951) > at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836) > at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887) > at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898) > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at
[GitHub] [lucene-solr] mikemccand commented on pull request #1623: LUCENE-8962: Merge segments on getReader
mikemccand commented on pull request #1623:
URL: https://github.com/apache/lucene-solr/pull/1623#issuecomment-651166475
Here's a fun tragic failure test that repros:
```
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestIndexWriterOnVMError -Dtests.method=testUnknownError
-Dtests.seed=BBEAF0EBC40AB8F7 -Dtests.slow=true -Dtests.badapples=true \
-Dtests.locale=brx -Dtests.timezone=Indian/Chagos -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
[junit4] FAILURE 0.30s | TestIndexWriterOnVMError.testUnknownError <<<
[junit4]> Throwable #1: java.lang.AssertionError
[junit4]>at
__randomizedtesting.SeedInfo.seed([BBEAF0EBC40AB8F7:50BC2F6BC68D2EC7]:0)
[junit4]>at
org.apache.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:5026)
[junit4]>at
org.apache.lucene.index.IndexWriter.tragicEvent(IndexWriter.java:5019)
[junit4]>at
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4214)
[junit4]>at
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:5735)
[junit4]>at
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
[junit4]>at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:581)
[junit4]>at
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103)
[junit4]>at
org.apache.lucene.index.TestIndexWriterOnVMError.doTest(TestIndexWriterOnVMError.java:175)
[junit4]>at
org.apache.lucene.index.TestIndexWriterOnVMError.testUnknownError(TestIndexWriterOnVMError.java:251)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4]>at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4]>at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
[junit4]>at java.base/java.lang.Thread.run(Thread.java:834)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene86):
{text_payloads=PostingsFormat(name=Direct),
text_vectors=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Luc\
ene84)),
text1=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene84)),
id=PostingsFormat(name=Direct)},
docValues:{dv3=DocValuesFormat(name=Asserting), dv2=DocValuesFormat\
(name=Lucene80), dv5=DocValuesFormat(name=Asserting),
dv=DocValuesFormat(name=Lucene80), dv4=DocValuesFormat(name=Lucene80)},
maxPointsInLeafNode=1228, maxMBSortInHeap=6.874571632539512, s\
im=Asserting(RandomSimilarity(queryNorm=true): {text_payloads=IB SPL-L1,
text_vectors=BM25(k1=1.2,b=0.75), text1=DFI(Saturated)}), locale=brx,
timezone=Indian/Chagos
[junit4] 2> NOTE: Linux 5.5.6-arch1-1 amd64/Oracle Corporation 11.0.6
(64-bit)/cpus=128,threads=1,free=522930136,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [TestIndexWriterOnVMError]
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on pull request #1623: LUCENE-8962: Merge segments on getReader
mikemccand commented on pull request #1623: URL: https://github.com/apache/lucene-solr/pull/1623#issuecomment-651156942 > So this would merge small commits on refresh? Small segments, yes. > I wonder if it would be more obvious to users if we call the MergeTrigger REFRESH? +1, refresh is more recognized in the outside world :) I have been beasting this and uncovering small test failures, in tests that are confused that they do not have the expected number of segments. I'll push some fixes for those ... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on pull request #1623: LUCENE-8962: Merge segments on getReader
mikemccand commented on pull request #1623: URL: https://github.com/apache/lucene-solr/pull/1623#issuecomment-650640435 Wow! This is incredibly simple! Thanks to the clean approach @s1monw worked out for commit-on-merge, awesome! I think tests would exercise this due to `MockRandomMergePolicy`? I'll try beasting :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
