[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17053231#comment-17053231
]
Simon Willnauer commented on LUCENE-8962:
-
I read through this issue and I want to share some of my thoughts. First, I
understand the need for this and the motivation, yet every time we add
something like this to the IndexWriter to do something _as part of_ another
method it triggers an alarm on my end. I have spent hours and days thinking
about how IW can be simpler and the biggest issues that I see is that the
primitives on IW like commit or openReader are doing too much. Just look at
openReader it's pretty involved and changing the bus factor or making it easier
to understand is hard. Adding stuff like _wait for merge_ with something like a
timeout is not what I think we should do neither to _openReader_ nor to
_commit_.
That said, I think we can make the same things happen but we should think in
primitives rather than changing method behavior with configuration. Let me
explain what I mean:
Lets say we keep _commit_ and _openReader_ the way it is and would instead
allow to use an existing reader NRT or not and allow itself to _optimize_
itself (yeah I said that - it might be a good name after all). With a slightly
refactored IW we can share the merge logic and let the reader re-write itself
since we are talking about very small segments the overhead is very small. This
would in turn mean that we are doing the work twice ie. the IW would do its
normal work and might merge later etc. We might even merge this stuff into
heap-space or so if we have enough I haven't thought too much about that. This
way we can clean up IW potentially and add a very nice optimization that works
for commit as well as NRT. We should strive for making IW simpler not do more.
I hope I wasn't too discouraging.
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: 8.5
>
> Attachments: LUCENE-8962_demo.png
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9276) Consolidate DW(PT)#updateDocument and #updateDocuments
Simon Willnauer created LUCENE-9276: --- Summary: Consolidate DW(PT)#updateDocument and #updateDocuments Key: LUCENE-9276 URL: https://issues.apache.org/jira/browse/LUCENE-9276 Project: Lucene - Core Issue Type: Improvement Affects Versions: master (9.0), 8.5 Reporter: Simon Willnauer While I was working on another IW related issue I made some changes to DW#updateDocument but forgot DW#updateDocuments which is annoying since the code is 99% identical. The same applies to DWPT#updateDocument[s]. IMO this is the wrong place to optimize in order to safe one or two object creations. Maybe we can remove this code duplication. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054506#comment-17054506
]
Simon Willnauer commented on LUCENE-8962:
-
{quote}
I don't think we should do this – IndexWriter's purpose is making changes to
the index, and IndexReader simply reads what IndexWriter created. There are
wildly diverse users of Lucene and if we now set down the path of
expecting/allowing IndexReader to do it's own "little" optimizations on
startup, that can add a lot of unexpected cost, and surprising bugs, to many
use cases. IndexWriter is indeed complex, but we should find ways to reduce
that complexity so that we can implement features in the right classes, rather
than shifting index-changing features out to IndexReader.
{quote}
This was really just an idea of how this could be done without adding much
complexity to existing methods. I can see this being a method like this
_IndexReader optimize(IndexReader reader, MergePolicy policy)_ than can even
commit the changes. It can run in parallel to the normal indexing but is
idempotent and much simpler to test. Again, just an idea to another approach.
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: 8.5
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch
>
> Time Spent: 9.5h
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054111#comment-17054111
]
Simon Willnauer commented on LUCENE-8962:
-
Wouldn’t it be easier to sort it all out in a branch rather on master?
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: 8.5
>
> Attachments: LUCENE-8962_demo.png
>
> Time Spent: 9.5h
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9164) Should not consider ACE a tragedy if IW is closed
[ https://issues.apache.org/jira/browse/LUCENE-9164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer updated LUCENE-9164: Fix Version/s: (was: 8.5) 8.6 > Should not consider ACE a tragedy if IW is closed > - > > Key: LUCENE-9164 > URL: https://issues.apache.org/jira/browse/LUCENE-9164 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: master (9.0), 8.5, 8.4.2 >Reporter: Nhat Nguyen >Assignee: Nhat Nguyen >Priority: Major > Fix For: master (9.0), 8.6 > > Attachments: LUCENE-9164.patch, LUCENE-9164.patch > > Time Spent: 7h 20m > Remaining Estimate: 0h > > If IndexWriter is closed or being closed, AlreadyClosedException is expected. > We should not consider it a tragic event in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9271) Make BufferedIndexInput work on a ByteBuffer
[
https://issues.apache.org/jira/browse/LUCENE-9271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078690#comment-17078690
]
Simon Willnauer commented on LUCENE-9271:
-
thanks [~jpountz]
> Make BufferedIndexInput work on a ByteBuffer
>
>
> Key: LUCENE-9271
> URL: https://issues.apache.org/jira/browse/LUCENE-9271
> Project: Lucene - Core
> Issue Type: Task
>Reporter: Adrien Grand
>Priority: Minor
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Currently {{BufferedIndexInput}} works on a {{byte[]}} but its main
> implementation, in NIOFSDirectory, has to implement a hack to maintain a
> ByteBuffer view of it that it can use in calls to the FileChannel API. Maybe
> we should instead make {{BufferedIndexInput}} work directly on a
> {{ByteBuffer}}? This would also help reuse the existing
> {{ByteBuffer#get(|Short|Int|long)}} methods instead of duplicating them from
> {{DataInput}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9309) IW#addIndices(CodecReader) might delete files concurrently to IW#rollback
[
https://issues.apache.org/jira/browse/LUCENE-9309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078675#comment-17078675
]
Simon Willnauer commented on LUCENE-9309:
-
[~mikemccand] can you take a look at the PR for this issue
> IW#addIndices(CodecReader) might delete files concurrently to IW#rollback
> -
>
> Key: LUCENE-9309
> URL: https://issues.apache.org/jira/browse/LUCENE-9309
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Simon Willnauer
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> During work on LUCENE-9304 [~mikemccand] ran into a failure:
> {noformat}
> org.apache.lucene.index.TestAddIndexes > test suite's output saved to
> /home/mike/src/simon/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.index.TestAddIndexes.txt,
> copied below:
>> java.nio.file.NoSuchFileException:
> _gt_Lucene85FieldsIndex-doc_ids_6u.tmp
>> at
> __randomizedtesting.SeedInfo.seed([4760FA81FBD4B2CE:A147156E5F7BD9B0]:0)
>> at
> org.apache.lucene.store.ByteBuffersDirectory.deleteFile(ByteBuffersDirectory.java:148)
>> at
> org.apache.lucene.store.MockDirectoryWrapper.deleteFile(MockDirectoryWrapper.java:607)
>> at
> org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:38)
>> at
> org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:696)
>> at
> org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:690)
>> at
> org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:449)
>> at
> org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2334)
>> at
> org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2275)
>> at
> org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2268)
>> at
> org.apache.lucene.index.TestAddIndexes.testAddIndexesWithRollback(TestAddIndexes.java:974)
> 2> NOTE: reproduce with: ant test -Dtestcase=TestAddIndexes
> -Dtests.method=testAddIndexesWithRollback -Dtests.seed=4760FA81FBD4B2CE
> -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=fr-GP -Dtests.t\
> imezone=Asia/Tbilisi -Dtests.asserts=true -Dtests.file.encoding=UTF-8
> 2> NOTE: test params are: codec=Asserting(Lucene84):
> {c=PostingsFormat(name=LuceneFixedGap),
> id=PostingsFormat(name=LuceneFixedGap),
> f1=PostingsFormat(name=LuceneFixedGap), f2=BlockTreeOrds(blocksize=128)\
> , version=BlockTreeOrds(blocksize=128), content=FST50},
> docValues:{dv=DocValuesFormat(name=Lucene80),
> soft_delete=DocValuesFormat(name=Lucene80),
> doc=DocValuesFormat(name=Lucene80), id=DocValuesFormat(name=\
> Asserting), content=DocValuesFormat(name=Asserting),
> doc2d=DocValuesFormat(name=Lucene80)}, maxPointsInLeafNode=982,
> maxMBSortInHeap=5.837219998050092,
> sim=Asserting(org.apache.lucene.search.similarities.As\
> sertingSimilarity@6ce38471), locale=fr-GP, timezone=Asia/Tbilisi
> {noformat}
> While this unfortunately doesn't reproduce it's likely a bug that exists for
> quite some time but never showed up until LUCENE-9147 which uses a temporary
> output. That's fine but with IW#addIndices(CodecReader...) not registering
> the merge it does in the IW we never wait for the merge to finish while
> rollback and if that merge finishes concurrently it will also remove these
> .tmp files.
> There are many ways to fix this and I can work on a patch, but hey do we
> really need to be able to add indices while we index and do that on an open
> and live IW or can it be a tool on top of it?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9271) Make BufferedIndexInput work on a ByteBuffer
[
https://issues.apache.org/jira/browse/LUCENE-9271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078627#comment-17078627
]
Simon Willnauer commented on LUCENE-9271:
-
We ran into a test failure
[here|https://github.com/apache/lucene-solr/pull/1397#issuecomment-610930218]:
{noformat}
org.apache.lucene.index.TestIndexManyDocuments > test suite's output saved to
/home/mike/src/simon/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.index.TestIndexManyDocuments.txt,
copi\
ed below:
1> CheckIndex failed
1> 0.00% total deletions; 10976 documents; 0 deleteions
1> Segments file=segments_1 numSegments=4 version=9.0.0
id=cekay2d5izae12ssuqikoqgoc
1> 1 of 4: name=_d maxDoc=8700
1> version=9.0.0
1> id=cekay2d5izae12ssuqikoqgob
1> codec=Asserting(Lucene84)
1> compound=false
1> numFiles=11
1> size (MB)=0.003
1> diagnostics = {os.version=5.5.6-arch1-1, java.vendor=Oracle
Corporation, source=merge, os.arch=amd64, mergeFactor=10,
java.runtime.version=11.0.6+8-LTS, os=Linux, timestamp=1586347074798, lucene.ve\
rsion=9.0.0, java.vm.version=11.0.6+8-LTS, java.version=11.0.6,
mergeMaxNumSegments=-1}
1> no deletions
1> test: open reader.OK [took 0.001 sec]
1> test: check integrity.OK [took 0.000 sec]
1> test: check live docs.OK [took 0.000 sec]
1> test: field infos.OK [1 fields] [took 0.000 sec]
1> test: field norms.OK [1 fields] [took 0.001 sec]
1> test: terms, freq, prox...ERROR: java.lang.AssertionError:
buffer=java.nio.HeapByteBuffer[pos=0 lim=0 cap=0] bufferSize=1024
buffer.length=0
1> java.lang.AssertionError: buffer=java.nio.HeapByteBuffer[pos=0 lim=0
cap=0] bufferSize=1024 buffer.length=0
1>at
org.apache.lucene.store.BufferedIndexInput.setBufferSize(BufferedIndexInput.java:78)
1>at
org.apache.lucene.codecs.MultiLevelSkipListReader.loadSkipLevels(MultiLevelSkipListReader.java:241)
1>at
org.apache.lucene.codecs.MultiLevelSkipListReader.init(MultiLevelSkipListReader.java:208)
1>at
org.apache.lucene.codecs.lucene84.Lucene84SkipReader.init(Lucene84SkipReader.java:103)
1>at
org.apache.lucene.codecs.lucene84.Lucene84PostingsReader$EverythingEnum.advance(Lucene84PostingsReader.java:837)
1>at
org.apache.lucene.index.FilterLeafReader$FilterPostingsEnum.advance(FilterLeafReader.java:271)
1>at
org.apache.lucene.index.AssertingLeafReader$AssertingPostingsEnum.advance(AssertingLeafReader.java:377)
1>at org.apache.lucene.index.CheckIndex.checkFields(CheckIndex.java:1426)
1>at org.apache.lucene.index.CheckIndex.testPostings(CheckIndex.java:1867)
1>at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:720)
1>at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:301)
1>at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:286)
1>at
org.apache.lucene.store.BaseDirectoryWrapper.close(BaseDirectoryWrapper.java:45)
1>at org.apache.lucene.util.IOUtils.close(IOUtils.java:89)
1>at org.apache.lucene.util.IOUtils.close(IOUtils.java:77)
1>at
org.apache.lucene.index.TestIndexManyDocuments.test(TestIndexManyDocuments.java:69)
1>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
1>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
1>at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
1>at java.base/java.lang.reflect.Method.invoke(Method.java:566)
1>at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
1>at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
1>at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
1>at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
1>at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
1>at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
1>at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
1>at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
1>at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
1>at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
1>at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
1>at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
1>at
[jira] [Resolved] (LUCENE-9309) IW#addIndices(CodecReader) might delete files concurrently to IW#rollback
[
https://issues.apache.org/jira/browse/LUCENE-9309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer resolved LUCENE-9309.
-
Fix Version/s: 8.6
master (9.0)
Lucene Fields: New,Patch Available (was: New)
Assignee: Simon Willnauer
Resolution: Fixed
> IW#addIndices(CodecReader) might delete files concurrently to IW#rollback
> -
>
> Key: LUCENE-9309
> URL: https://issues.apache.org/jira/browse/LUCENE-9309
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Simon Willnauer
>Assignee: Simon Willnauer
>Priority: Major
> Fix For: master (9.0), 8.6
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> During work on LUCENE-9304 [~mikemccand] ran into a failure:
> {noformat}
> org.apache.lucene.index.TestAddIndexes > test suite's output saved to
> /home/mike/src/simon/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.index.TestAddIndexes.txt,
> copied below:
>> java.nio.file.NoSuchFileException:
> _gt_Lucene85FieldsIndex-doc_ids_6u.tmp
>> at
> __randomizedtesting.SeedInfo.seed([4760FA81FBD4B2CE:A147156E5F7BD9B0]:0)
>> at
> org.apache.lucene.store.ByteBuffersDirectory.deleteFile(ByteBuffersDirectory.java:148)
>> at
> org.apache.lucene.store.MockDirectoryWrapper.deleteFile(MockDirectoryWrapper.java:607)
>> at
> org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:38)
>> at
> org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:696)
>> at
> org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:690)
>> at
> org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:449)
>> at
> org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2334)
>> at
> org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2275)
>> at
> org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2268)
>> at
> org.apache.lucene.index.TestAddIndexes.testAddIndexesWithRollback(TestAddIndexes.java:974)
> 2> NOTE: reproduce with: ant test -Dtestcase=TestAddIndexes
> -Dtests.method=testAddIndexesWithRollback -Dtests.seed=4760FA81FBD4B2CE
> -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=fr-GP -Dtests.t\
> imezone=Asia/Tbilisi -Dtests.asserts=true -Dtests.file.encoding=UTF-8
> 2> NOTE: test params are: codec=Asserting(Lucene84):
> {c=PostingsFormat(name=LuceneFixedGap),
> id=PostingsFormat(name=LuceneFixedGap),
> f1=PostingsFormat(name=LuceneFixedGap), f2=BlockTreeOrds(blocksize=128)\
> , version=BlockTreeOrds(blocksize=128), content=FST50},
> docValues:{dv=DocValuesFormat(name=Lucene80),
> soft_delete=DocValuesFormat(name=Lucene80),
> doc=DocValuesFormat(name=Lucene80), id=DocValuesFormat(name=\
> Asserting), content=DocValuesFormat(name=Asserting),
> doc2d=DocValuesFormat(name=Lucene80)}, maxPointsInLeafNode=982,
> maxMBSortInHeap=5.837219998050092,
> sim=Asserting(org.apache.lucene.search.similarities.As\
> sertingSimilarity@6ce38471), locale=fr-GP, timezone=Asia/Tbilisi
> {noformat}
> While this unfortunately doesn't reproduce it's likely a bug that exists for
> quite some time but never showed up until LUCENE-9147 which uses a temporary
> output. That's fine but with IW#addIndices(CodecReader...) not registering
> the merge it does in the IW we never wait for the merge to finish while
> rollback and if that merge finishes concurrently it will also remove these
> .tmp files.
> There are many ways to fix this and I can work on a patch, but hey do we
> really need to be able to add indices while we index and do that on an open
> and live IW or can it be a tool on top of it?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9298) Improve RAM accounting in BufferedUpdates when deleted doc IDs and terms are cleared
[
https://issues.apache.org/jira/browse/LUCENE-9298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer resolved LUCENE-9298.
-
Fix Version/s: 8.6
master (9.0)
Lucene Fields: New,Patch Available (was: New)
Assignee: Simon Willnauer
Resolution: Fixed
> Improve RAM accounting in BufferedUpdates when deleted doc IDs and terms are
> cleared
>
>
> Key: LUCENE-9298
> URL: https://issues.apache.org/jira/browse/LUCENE-9298
> Project: Lucene - Core
> Issue Type: Bug
>Affects Versions: 8.5
>Reporter: yubinglei
>Assignee: Simon Willnauer
>Priority: Minor
> Labels: easyfix, newbie, pull-request-available
> Fix For: master (9.0), 8.6
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> As mentioned in this
> [https://github.com/apache/lucene-solr/pull/513#discussion_r399131681]
> the method clearDeletedDocIds in BufferedUpdates.java has a bug, it can't
> reset bytesUsed correctly
> {code:java}
> void clearDeletedDocIds() {
> deleteDocIDs.clear();
> bytesUsed.addAndGet(-deleteDocIDs.size() *
> BufferedUpdates.BYTES_PER_DEL_DOCID);
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9304) Clean up DWPTPool
[ https://issues.apache.org/jira/browse/LUCENE-9304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer resolved LUCENE-9304. - Fix Version/s: 8.6 master (9.0) Lucene Fields: New,Patch Available (was: New) Assignee: Simon Willnauer Resolution: Fixed > Clean up DWPTPool > -- > > Key: LUCENE-9304 > URL: https://issues.apache.org/jira/browse/LUCENE-9304 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: master (9.0), 8.6 >Reporter: Simon Willnauer >Assignee: Simon Willnauer >Priority: Major > Fix For: master (9.0), 8.6 > > Time Spent: 9h 10m > Remaining Estimate: 0h > > DWPTPool currently uses an indirection called ThreadState to hold DWPT > instances. This class holds several information that belongs in other places, > inherits from ReentrantLock and has a mutable nature. Instead we could pool > the DWPT directly and remove other indirections inside DWPTFlushControl if > we move some of the ThreadState properties to DWPT directly. The threadpool > also has a problem that is grows it's ThreadStates to the number of > concurrently indexing threads but never shrinks it if they are reduced. With > pooling DWPT directly this limitation could be removed. > In summary, this component has seen quite some refactoring and requires some > cleanups and docs changes in order to stay the test of time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (LUCENE-9324) Give IDs to SegmentCommitInfo
[ https://issues.apache.org/jira/browse/LUCENE-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer reassigned LUCENE-9324: --- Assignee: Simon Willnauer > Give IDs to SegmentCommitInfo > - > > Key: LUCENE-9324 > URL: https://issues.apache.org/jira/browse/LUCENE-9324 > Project: Lucene - Core > Issue Type: New Feature >Reporter: Adrien Grand >Assignee: Simon Willnauer >Priority: Minor > Time Spent: 1h > Remaining Estimate: 0h > > We already have IDs in SegmentInfo, which are useful to uniquely identify > segments. Having IDs on SegmentCommitInfo would be useful too in order to > compare commits for equality and make snapshots incremental on generational > files too. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9324) Give IDs to SegmentCommitInfo
[ https://issues.apache.org/jira/browse/LUCENE-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer updated LUCENE-9324: Fix Version/s: 8.6 master (9.0) > Give IDs to SegmentCommitInfo > - > > Key: LUCENE-9324 > URL: https://issues.apache.org/jira/browse/LUCENE-9324 > Project: Lucene - Core > Issue Type: New Feature >Reporter: Adrien Grand >Assignee: Simon Willnauer >Priority: Minor > Fix For: master (9.0), 8.6 > > Time Spent: 1h > Remaining Estimate: 0h > > We already have IDs in SegmentInfo, which are useful to uniquely identify > segments. Having IDs on SegmentCommitInfo would be useful too in order to > compare commits for equality and make snapshots incremental on generational > files too. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9324) Give IDs to SegmentCommitInfo
[ https://issues.apache.org/jira/browse/LUCENE-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer resolved LUCENE-9324. - Lucene Fields: New,Patch Available (was: New) Resolution: Fixed > Give IDs to SegmentCommitInfo > - > > Key: LUCENE-9324 > URL: https://issues.apache.org/jira/browse/LUCENE-9324 > Project: Lucene - Core > Issue Type: New Feature >Reporter: Adrien Grand >Assignee: Simon Willnauer >Priority: Minor > Fix For: master (9.0), 8.6 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > We already have IDs in SegmentInfo, which are useful to uniquely identify > segments. Having IDs on SegmentCommitInfo would be useful too in order to > compare commits for equality and make snapshots incremental on generational > files too. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9309) IW#addIndices(CodecReader) might delete files concurrently to IW#rollback
[
https://issues.apache.org/jira/browse/LUCENE-9309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081395#comment-17081395
]
Simon Willnauer commented on LUCENE-9309:
-
the push above was to fix a test failure we have on elastic CI:
{noformat}
15:10:24[junit4] Suite: org.apache.lucene.index.TestIndexWriterMerging
15:10:24[junit4] 1>
org.apache.lucene.index.MergePolicy$MergeAbortedException: merge is aborted:
_2u(9.0.0):c199/18:[diagnostics={source=flush, os.arch=amd64,
java.runtime.version=11.0.2+7, os.version=4.15.0-1060-gcp, os=Linux,
java.vendor=Oracle Corporation, timestamp=1586617824513, java.version=11.0.2,
java.vm.version=11.0.2+7,
lucene.version=9.0.0}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=1
_2x(9.0.0):c425:[diagnostics={source=flush, os.arch=amd64,
java.runtime.version=11.0.2+7, os.version=4.15.0-1060-gcp, os=Linux,
java.vendor=Oracle Corporation, timestamp=1586617824590, java.version=11.0.2,
java.vm.version=11.0.2+7,
lucene.version=9.0.0}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]
15:10:24[junit4] 1> at
org.apache.lucene.index.IndexWriter.registerMerge(IndexWriter.java:4125)
15:10:24[junit4] 1> at
org.apache.lucene.index.IndexWriter.updatePendingMerges(IndexWriter.java:2203)
15:10:24[junit4] 1> at
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2160)
15:10:24[junit4] 1> at
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5142)
15:10:24[junit4] 1> at
org.apache.lucene.index.IndexWriter.maybeProcessEvents(IndexWriter.java:5132)
15:10:24[junit4] 1> at
org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1360)
15:10:24[junit4] 1> at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1647)
15:10:24[junit4] 1> at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1289)
15:10:24[junit4] 1> at
org.apache.lucene.index.TestIndexWriterMerging$1.run(TestIndexWriterMerging.java:416)
15:10:24[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestIndexWriterMerging -Dtests.method=testNoWaitClose
-Dtests.seed=1D4119EDE2022625 -Dtests.slow=true -Dtests.badapples=true
-Dtests.locale=sv-AX -Dtests.timezone=Europe/Busingen -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
15:10:24[junit4] ERROR 0.90s J2 | TestIndexWriterMerging.testNoWaitClose
<<<
15:10:24[junit4]> Throwable #1:
org.apache.lucene.index.MergePolicy$MergeAbortedException: merge is aborted:
_2u(9.0.0):c199/18:[diagnostics={source=flush, os.arch=amd64,
java.runtime.version=11.0.2+7, os.version=4.15.0-1060-gcp, os=Linux,
java.vendor=Oracle Corporation, timestamp=1586617824513, java.version=11.0.2,
java.vm.version=11.0.2+7,
lucene.version=9.0.0}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=1
_2x(9.0.0):c425:[diagnostics={source=flush, os.arch=amd64,
java.runtime.version=11.0.2+7, os.version=4.15.0-1060-gcp, os=Linux,
java.vendor=Oracle Corporation, timestamp=1586617824590, java.version=11.0.2,
java.vm.version=11.0.2+7,
lucene.version=9.0.0}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]
15:10:24[junit4]> at
__randomizedtesting.SeedInfo.seed([1D4119EDE2022625:3AB0B83C3114F6D1]:0)
15:10:24[junit4]> at
org.apache.lucene.index.IndexWriter.registerMerge(IndexWriter.java:4125)
15:10:24[junit4]> at
org.apache.lucene.index.IndexWriter.updatePendingMerges(IndexWriter.java:2203)
15:10:24[junit4]> at
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2160)
15:10:24[junit4]> at
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5142)
15:10:24[junit4]> at
org.apache.lucene.index.IndexWriter.maybeProcessEvents(IndexWriter.java:5132)
15:10:24[junit4]> at
org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1360)
15:10:24[junit4]> at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1647)
15:10:24[junit4]> at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1289)
15:10:24[junit4]> at
org.apache.lucene.index.TestIndexWriterMerging$1.run(TestIndexWriterMerging.java:416)
15:10:24[junit4] 2> NOTE: test params are: codec=Asserting(Lucene84),
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@285b2588),
locale=sv-AX, timezone=Europe/Busingen
15:10:24[junit4] 2> NOTE: Linux 4.15.0-1060-gcp amd64/Oracle Corporation
11.0.2 (64-bit)/cpus=32,threads=1,free=387385984,total=536870912
15:10:24[junit4] 2> NOTE: All tests run in this JVM:
[TestNRTReaderCleanup, TestMutableValues, TestDistributionSPL,
TestBinaryDocument, TestRollback, TestXYMultiPolygonShapeQueries,
TestTwoPhaseCommitTool, TestDocIDMerger,
[jira] [Commented] (LUCENE-9324) Give IDs to SegmentCommitInfo
[ https://issues.apache.org/jira/browse/LUCENE-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084682#comment-17084682 ] Simon Willnauer commented on LUCENE-9324: - I am trying to give a bit more context to this issue. Today we have _SegmentInfo_ which represents a segment once it's written to disk for instance at flush or merge time. We have a randomly generated ID in _SegmentInfo_ that can be used to verify if two segments are the same. Since we use incremental numbers for segment naming it's likely that two IndexWriters produce a segment with very similar contents and the same name. Yet, the _SegmentInfo_ id would be different. In addition to this ID we also have checksums on files which can be used to verify identity in addition to the ID but should not be treated identity by itself since they are very weak checksums. Now segments also get _updated_ for instance when a documents is marked as deleted or the segment receives a doc values update. The only thing that changes is the delete or update generation which also allow two IndexWriters that opened two copies of a segment (with the same segment ID) to produce a new delGen or dvGen that looks identical from the outside but are actually different. This is a problem that we see quite frequently in Elasticsearch and we'd like to prevent or have a better tool in our hands to distinguish _SegmentCommitInfo_ instances from another. If we'd have an ID on SegmentCommitInfo that changes each time one of these generations changes we could much easier tell if only the updated files (which are often very small) need to be replaced in order to recover an index. The plan is to implement this in a very similar fashion as we did on the _SegmentInfo_ but also invalidate the once any of the generations change in order to force a new _SegmentCommitInfo_ ID for the new generation. Yet, the IDs would not be the same if two IndexWriters start from the same segment making an identical change to the segment ie. it's not a replacement for a strong hash function. > Give IDs to SegmentCommitInfo > - > > Key: LUCENE-9324 > URL: https://issues.apache.org/jira/browse/LUCENE-9324 > Project: Lucene - Core > Issue Type: New Feature >Reporter: Adrien Grand >Priority: Minor > > We already have IDs in SegmentInfo, which are useful to uniquely identify > segments. Having IDs on SegmentCommitInfo would be useful too in order to > compare commits for equality and make snapshots incremental on generational > files too. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9309) IW#addIndices(CodecReader) might delete files concurrently to IW#rollback
Simon Willnauer created LUCENE-9309:
---
Summary: IW#addIndices(CodecReader) might delete files
concurrently to IW#rollback
Key: LUCENE-9309
URL: https://issues.apache.org/jira/browse/LUCENE-9309
Project: Lucene - Core
Issue Type: Bug
Reporter: Simon Willnauer
During work on LUCENE-9304 [~mikemccand] ran into a failure:
{noformat}
org.apache.lucene.index.TestAddIndexes > test suite's output saved to
/home/mike/src/simon/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.index.TestAddIndexes.txt,
copied below:
> java.nio.file.NoSuchFileException:
_gt_Lucene85FieldsIndex-doc_ids_6u.tmp
> at
__randomizedtesting.SeedInfo.seed([4760FA81FBD4B2CE:A147156E5F7BD9B0]:0)
> at
org.apache.lucene.store.ByteBuffersDirectory.deleteFile(ByteBuffersDirectory.java:148)
> at
org.apache.lucene.store.MockDirectoryWrapper.deleteFile(MockDirectoryWrapper.java:607)
> at
org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:38)
> at
org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:696)
> at
org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:690)
> at
org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:449)
> at
org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2334)
> at
org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2275)
> at
org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2268)
> at
org.apache.lucene.index.TestAddIndexes.testAddIndexesWithRollback(TestAddIndexes.java:974)
2> NOTE: reproduce with: ant test -Dtestcase=TestAddIndexes
-Dtests.method=testAddIndexesWithRollback -Dtests.seed=4760FA81FBD4B2CE
-Dtests.slow=true -Dtests.badapples=true -Dtests.locale=fr-GP -Dtests.t\
imezone=Asia/Tbilisi -Dtests.asserts=true -Dtests.file.encoding=UTF-8
2> NOTE: test params are: codec=Asserting(Lucene84):
{c=PostingsFormat(name=LuceneFixedGap), id=PostingsFormat(name=LuceneFixedGap),
f1=PostingsFormat(name=LuceneFixedGap), f2=BlockTreeOrds(blocksize=128)\
, version=BlockTreeOrds(blocksize=128), content=FST50},
docValues:{dv=DocValuesFormat(name=Lucene80),
soft_delete=DocValuesFormat(name=Lucene80), doc=DocValuesFormat(name=Lucene80),
id=DocValuesFormat(name=\
Asserting), content=DocValuesFormat(name=Asserting),
doc2d=DocValuesFormat(name=Lucene80)}, maxPointsInLeafNode=982,
maxMBSortInHeap=5.837219998050092,
sim=Asserting(org.apache.lucene.search.similarities.As\
sertingSimilarity@6ce38471), locale=fr-GP, timezone=Asia/Tbilisi
{noformat}
While this unfortunately doesn't reproduce it's likely a bug that exists for
quite some time but never showed up until LUCENE-9147 which uses a temporary
output. That's fine but with IW#addIndices(CodecReader...) not registering the
merge it does in the IW we never wait for the merge to finish while rollback
and if that merge finishes concurrently it will also remove these .tmp files.
There are many ways to fix this and I can work on a patch, but hey do we really
need to be able to add indices while we index and do that on an open and live
IW or can it be a tool on top of it?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9304) Clean up DWPTPool
Simon Willnauer created LUCENE-9304: --- Summary: Clean up DWPTPool Key: LUCENE-9304 URL: https://issues.apache.org/jira/browse/LUCENE-9304 Project: Lucene - Core Issue Type: Improvement Affects Versions: master (9.0), 8.6 Reporter: Simon Willnauer DWPTPool currently uses an indirection called ThreadState to hold DWPT instances. This class holds several information that belongs in other places, inherits from ReentrantLock and has a mutable nature. Instead we could pool the DWPT directly and remove other indirections inside DWPTFlushControl if we move some of the ThreadState properties to DWPT directly. The threadpool also has a problem that is grows it's ThreadStates to the number of concurrently indexing threads but never shrinks it if they are reduced. With pooling DWPT directly this limitation could be removed. In summary, this component has seen quite some refactoring and requires some cleanups and docs changes in order to stay the test of time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9345) Separate IndexWriter from MergeScheduler
[ https://issues.apache.org/jira/browse/LUCENE-9345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer resolved LUCENE-9345. - Fix Version/s: 8.6 master (9.0) Lucene Fields: New,Patch Available (was: New) Assignee: Simon Willnauer Resolution: Fixed > Separate IndexWriter from MergeScheduler > > > Key: LUCENE-9345 > URL: https://issues.apache.org/jira/browse/LUCENE-9345 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: master (9.0) >Reporter: Simon Willnauer >Assignee: Simon Willnauer >Priority: Major > Fix For: master (9.0), 8.6 > > Time Spent: 40m > Remaining Estimate: 0h > > MergeScheduler is tightly coupled with IndexWriter which causes IW to expose > unnecessary methods. For instance only the scheduler should call > IW#getNextMerge() but it's a public method. With some refactorings we can > nicely separate the two. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9337) CMS might miss to pickup pending merges when maxMergeCount changes while merges are running
[
https://issues.apache.org/jira/browse/LUCENE-9337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089085#comment-17089085
]
Simon Willnauer commented on LUCENE-9337:
-
here is a PR https://github.com/apache/lucene-solr/pull/1443/
> CMS might miss to pickup pending merges when maxMergeCount changes while
> merges are running
> ---
>
> Key: LUCENE-9337
> URL: https://issues.apache.org/jira/browse/LUCENE-9337
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Simon Willnauer
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We found a test hanging on an IW#forceMerge on elastics CI on an innocent
> looking test:
> {noformat}
> 14:52:06[junit4] 2> at
> java.base@11.0.2/java.lang.Object.wait(Native Method)
> 14:52:06[junit4] 2> at
> app//org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4722)
> 14:52:06[junit4] 2> at
> app//org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:2034)
> 14:52:06[junit4] 2> at
> app//org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1960)
> 14:52:06[junit4] 2> at
> app//org.apache.lucene.index.RandomIndexWriter.forceMerge(RandomIndexWriter.java:500)
> 14:52:06[junit4] 2> at
> app//org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestNumericsVsStoredFields(BaseDocValuesFormatTestCase.java:1301)
> 14:52:06[junit4] 2> at
> app//org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestNumericsVsStoredFields(BaseDocValuesFormatTestCase.java:1258)
> 14:52:06[junit4] 2> at
> app//org.apache.lucene.index.BaseDocValuesFormatTestCase.testZeroOrMin(BaseDocValuesFormatTestCase.java:2423)
> {noformat}
> after spending quite some time trying to reproduce without any luck I tried
> to review all involved code again to understand possible threading issues.
> What I found is that if maxMergeCount gets changed on CMS while there are
> merges running and the forceMerge gets kicked off at the same time the
> running merges return we might miss to pick up the final pending merges which
> causes the forceMerge to hang. I was able to build a test-case that is very
> likely to fail on every run without the fix. While I think this is not a
> critical bug from how likely it is to happen in practice, if it happens it's
> basically a deadlock unless the IW sees any other change that kicks off a
> merge.
> Lemme walk through the issue. Lets say we have 1 pending merge and 2 merge
> threads running on CMS. The forceMerge is already waiting for merges to
> finish. Once the first merge thread finishes we try to check if we need to
> stall it
> [here|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.1/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java#L580]
> but since it's a merge thread we return
> [here|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.1/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java#L596]
> and don't pick up another merge
> [here|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.1/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java#L526].
>
> Now the second running merge thread checks the condition
> [here|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.1/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java#L580]
> while the first one is finishing up. But before it can actually update the
> internal datastructures
> [here|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.1/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java#L688]
> it releases the CMS lock and the calculation in the stall method on how many
> threads are running is off causing the second thread also to step out of the
> maybeStall method not picking up the pending merge.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9337) CMS might miss to pickup pending merges when maxMergeCount changes while merges are running
Simon Willnauer created LUCENE-9337:
---
Summary: CMS might miss to pickup pending merges when
maxMergeCount changes while merges are running
Key: LUCENE-9337
URL: https://issues.apache.org/jira/browse/LUCENE-9337
Project: Lucene - Core
Issue Type: Bug
Reporter: Simon Willnauer
We found a test hanging on an IW#forceMerge on elastics CI on an innocent
looking test:
{noformat}
14:52:06[junit4] 2> at
java.base@11.0.2/java.lang.Object.wait(Native Method)
14:52:06[junit4] 2> at
app//org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4722)
14:52:06[junit4] 2> at
app//org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:2034)
14:52:06[junit4] 2> at
app//org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1960)
14:52:06[junit4] 2> at
app//org.apache.lucene.index.RandomIndexWriter.forceMerge(RandomIndexWriter.java:500)
14:52:06[junit4] 2> at
app//org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestNumericsVsStoredFields(BaseDocValuesFormatTestCase.java:1301)
14:52:06[junit4] 2> at
app//org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestNumericsVsStoredFields(BaseDocValuesFormatTestCase.java:1258)
14:52:06[junit4] 2> at
app//org.apache.lucene.index.BaseDocValuesFormatTestCase.testZeroOrMin(BaseDocValuesFormatTestCase.java:2423)
{noformat}
after spending quite some time trying to reproduce without any luck I tried to
review all involved code again to understand possible threading issues. What I
found is that if maxMergeCount gets changed on CMS while there are merges
running and the forceMerge gets kicked off at the same time the running merges
return we might miss to pick up the final pending merges which causes the
forceMerge to hang. I was able to build a test-case that is very likely to fail
on every run without the fix. While I think this is not a critical bug from how
likely it is to happen in practice, if it happens it's basically a deadlock
unless the IW sees any other change that kicks off a merge.
Lemme walk through the issue. Lets say we have 1 pending merge and 2 merge
threads running on CMS. The forceMerge is already waiting for merges to finish.
Once the first merge thread finishes we try to check if we need to stall it
[here|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.1/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java#L580]
but since it's a merge thread we return
[here|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.1/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java#L596]
and don't pick up another merge
[here|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.1/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java#L526].
Now the second running merge thread checks the condition
[here|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.1/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java#L580]
while the first one is finishing up. But before it can actually update the
internal datastructures
[here|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.1/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java#L688]
it releases the CMS lock and the calculation in the stall method on how many
threads are running is off causing the second thread also to step out of the
maybeStall method not picking up the pending merge.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9345) Separate IndexWriter from MergeScheduler
Simon Willnauer created LUCENE-9345: --- Summary: Separate IndexWriter from MergeScheduler Key: LUCENE-9345 URL: https://issues.apache.org/jira/browse/LUCENE-9345 Project: Lucene - Core Issue Type: Improvement Affects Versions: master (9.0) Reporter: Simon Willnauer MergeScheduler is tightly coupled with IndexWriter which causes IW to expose unnecessary methods. For instance only the scheduler should call IW#getNextMerge() but it's a public method. With some refactorings we can nicely separate the two. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9337) CMS might miss to pickup pending merges when maxMergeCount changes while merges are running
[
https://issues.apache.org/jira/browse/LUCENE-9337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer resolved LUCENE-9337.
-
Fix Version/s: 8.6
master (9.0)
Assignee: Simon Willnauer
Resolution: Fixed
> CMS might miss to pickup pending merges when maxMergeCount changes while
> merges are running
> ---
>
> Key: LUCENE-9337
> URL: https://issues.apache.org/jira/browse/LUCENE-9337
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Simon Willnauer
>Assignee: Simon Willnauer
>Priority: Major
> Fix For: master (9.0), 8.6
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> We found a test hanging on an IW#forceMerge on elastics CI on an innocent
> looking test:
> {noformat}
> 14:52:06[junit4] 2> at
> java.base@11.0.2/java.lang.Object.wait(Native Method)
> 14:52:06[junit4] 2> at
> app//org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4722)
> 14:52:06[junit4] 2> at
> app//org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:2034)
> 14:52:06[junit4] 2> at
> app//org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1960)
> 14:52:06[junit4] 2> at
> app//org.apache.lucene.index.RandomIndexWriter.forceMerge(RandomIndexWriter.java:500)
> 14:52:06[junit4] 2> at
> app//org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestNumericsVsStoredFields(BaseDocValuesFormatTestCase.java:1301)
> 14:52:06[junit4] 2> at
> app//org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestNumericsVsStoredFields(BaseDocValuesFormatTestCase.java:1258)
> 14:52:06[junit4] 2> at
> app//org.apache.lucene.index.BaseDocValuesFormatTestCase.testZeroOrMin(BaseDocValuesFormatTestCase.java:2423)
> {noformat}
> after spending quite some time trying to reproduce without any luck I tried
> to review all involved code again to understand possible threading issues.
> What I found is that if maxMergeCount gets changed on CMS while there are
> merges running and the forceMerge gets kicked off at the same time the
> running merges return we might miss to pick up the final pending merges which
> causes the forceMerge to hang. I was able to build a test-case that is very
> likely to fail on every run without the fix. While I think this is not a
> critical bug from how likely it is to happen in practice, if it happens it's
> basically a deadlock unless the IW sees any other change that kicks off a
> merge.
> Lemme walk through the issue. Lets say we have 1 pending merge and 2 merge
> threads running on CMS. The forceMerge is already waiting for merges to
> finish. Once the first merge thread finishes we try to check if we need to
> stall it
> [here|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.1/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java#L580]
> but since it's a merge thread we return
> [here|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.1/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java#L596]
> and don't pick up another merge
> [here|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.1/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java#L526].
>
> Now the second running merge thread checks the condition
> [here|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.1/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java#L580]
> while the first one is finishing up. But before it can actually update the
> internal datastructures
> [here|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.1/lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java#L688]
> it releases the CMS lock and the calculation in the stall method on how many
> threads are running is off causing the second thread also to step out of the
> maybeStall method not picking up the pending merge.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9339) Only call MergeScheduler when we actually found new merges
Simon Willnauer created LUCENE-9339: --- Summary: Only call MergeScheduler when we actually found new merges Key: LUCENE-9339 URL: https://issues.apache.org/jira/browse/LUCENE-9339 Project: Lucene - Core Issue Type: Improvement Affects Versions: master (9.0), 8.6 Reporter: Simon Willnauer IW#maybeMerge calls the MergeScheduler even if it didn't find any merges we should instead only do this if there is in-fact anything there to merge and safe the call into a sync'd method. In addition to this we never respect the _newMergesFound_ parameter. It seems unnecessary and can be removed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9339) Only call MergeScheduler when we actually found new merges
[ https://issues.apache.org/jira/browse/LUCENE-9339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer resolved LUCENE-9339. - Fix Version/s: 8.6 master (9.0) Lucene Fields: New,Patch Available (was: New) Assignee: Simon Willnauer Resolution: Fixed > Only call MergeScheduler when we actually found new merges > -- > > Key: LUCENE-9339 > URL: https://issues.apache.org/jira/browse/LUCENE-9339 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: master (9.0), 8.6 >Reporter: Simon Willnauer >Assignee: Simon Willnauer >Priority: Major > Fix For: master (9.0), 8.6 > > Time Spent: 20m > Remaining Estimate: 0h > > IW#maybeMerge calls the MergeScheduler even if it didn't find any merges we > should instead only do this if there is in-fact anything there to merge and > safe the call into a sync'd method. In addition to this we never respect the > _newMergesFound_ parameter. It seems unnecessary and can be removed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9363) TestIndexWriterDelete.testDeleteAllNoDeadLock failed on CI
Simon Willnauer created LUCENE-9363:
---
Summary: TestIndexWriterDelete.testDeleteAllNoDeadLock failed on CI
Key: LUCENE-9363
URL: https://issues.apache.org/jira/browse/LUCENE-9363
Project: Lucene - Core
Issue Type: Improvement
Reporter: Simon Willnauer
{noformat}
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestIndexWriterDelete
-Dtests.method=testDeleteAllNoDeadLock -Dtests.seed=7C9973DFB2835976
-Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ro-MD
-Dtests.timezone=SystemV/AST4ADT -Dtests.asserts=true
-Dtests.file.encoding=US-ASCII
[junit4] FAILURE 0.25s J2 | TestIndexWriterDelete.testDeleteAllNoDeadLock <<<
[junit4]> Throwable #1: java.lang.AssertionError
[junit4]>at
__randomizedtesting.SeedInfo.seed([7C9973DFB2835976:4F809CC13240E320]:0)
[junit4]>at
org.apache.lucene.index.IndexWriter.abortMerges(IndexWriter.java:2497)
[junit4]>at
org.apache.lucene.index.IndexWriter.deleteAll(IndexWriter.java:2424)
[junit4]>at
org.apache.lucene.index.RandomIndexWriter.deleteAll(RandomIndexWriter.java:373)
[junit4]>at
org.apache.lucene.index.TestIndexWriterDelete.testDeleteAllNoDeadLock(TestIndexWriterDelete.java:348)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4]>at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4]>at
java.base/java.lang.reflect.Method.invoke(Method.java:564)
[junit4]>at java.base/java.lang.Thread.run(Thread.java:832)
[junit4] 2> NOTE: leaving temporary files on disk at:
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/build/core/test/J2/temp/lucene.index.TestIndexWriterDelete_7C9973DFB2835976-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene84):
{field=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene84)),
city=Lucene84,
contents=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene84)),
id=Lucene84, value=PostingsFormat(name=LuceneVarGapDocFreqInterval),
content=PostingsFormat(name=LuceneVarGapDocFreqInterval)},
docValues:{dv=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=2033,
maxMBSortInHeap=6.1560152382287825,
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@c9cd3c9),
locale=ro-MD, timezone=SystemV/AST4ADT
[junit4] 2> NOTE: Linux 5.3.0-46-generic amd64/Oracle Corporation 15-ea
(64-bit)/cpus=16,threads=1,free=389663704,total=518979584
[junit4] 2> NOTE: All tests run in this JVM: [TestIntroSorter,
TestNGramPhraseQuery, TestUpgradeIndexMergePolicy, TestBasicModelIne,
TestTragicIndexWriterDeadlock, TestSimpleExplanations, TestDirectory,
TestSegmentToThreadMapping, TestNIOFSDirectory, TestAxiomaticF3LOG,
TestIndexOptions, TestSpanNotQuery, TestBinaryDocument,
TestFixedLengthBytesRefArray, TestForUtil, TestParallelLeafReader,
TestDirectoryReader, TestBytesRef, TestLongRange, TestByteSlices,
TestRectangle2D, TestBufferedChecksum, TestIndexOrDocValuesQuery,
TestFilterDirectoryReader, TestSleepingLockWrapper, TestDocIdSetIterator,
TestMultiPhraseQuery, TestSearchWithThreads, TestPointQueries,
TestForceMergeForever, TestNewestSegment, TestCrash, TestIntSet,
Test4GBStoredFields, TestDirectPacked, TestLiveFieldValues,
TestTopDocsCollector, Test2BPositions, TestAtomicUpdate,
TestFSTDirectAddressing, TestXYPointQueries, TestLatLonLineShapeQueries,
TestStressAdvance, TestBasics, TestLucene50StoredFieldsFormat,
TestQueryRescorer, TestConstantScoreScorer, TestAssertions, TestDemo,
TestExternalCodecs, TestMergeSchedulerExternal, TestAnalyzerWrapper,
TestCachingTokenFilter, TestCharArrayMap, TestCharArraySet, TestCharFilter,
TestCharacterUtils, TestDelegatingAnalyzerWrapper, TestGraphTokenFilter,
TestGraphTokenizers, TestReusableStringReader, TestWordlistLoader,
TestStandardAnalyzer, TestBytesRefAttImpl, TestCharTermAttributeImpl,
TestCodecUtil, TestCompetitiveFreqNormAccumulator, TestFastCompressionMode,
TestFastDecompressionMode, TestHighCompressionMode,
TestLucene60FieldInfoFormat, TestLucene60PointsFormat,
TestLucene70SegmentInfoFormat, TestDocument, TestDoubleRange,
TestFeatureDoubleValues, TestFeatureField, TestFeatureSort, TestField,
TestFieldType, TestFloatRange, TestLatLonMultiPolygonShapeQueries,
TestXYMultiLineShapeQueries, TestXYPolygonShapeQueries, TestLine2D,
TestTessellator, TestXYCircle, TestXYPolygon, Test2BDocs,
TestAllFilesCheckIndexHeader, TestDocInverterPerFieldErrorInfo,
TestDocsAndPositions, TestExceedMaxTermLength, TestFieldInfos,
TestFieldInvertState, TestFieldUpdatesBuffer, TestFlex, TestIndexWriterCommit,
[jira] [Updated] (LUCENE-9363) TestIndexWriterDelete.testDeleteAllNoDeadLock failed on CI
[
https://issues.apache.org/jira/browse/LUCENE-9363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer updated LUCENE-9363:
Issue Type: Test (was: Improvement)
> TestIndexWriterDelete.testDeleteAllNoDeadLock failed on CI
> --
>
> Key: LUCENE-9363
> URL: https://issues.apache.org/jira/browse/LUCENE-9363
> Project: Lucene - Core
> Issue Type: Test
>Reporter: Simon Willnauer
>Priority: Major
>
> {noformat}
> [junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestIndexWriterDelete -Dtests.method=testDeleteAllNoDeadLock
> -Dtests.seed=7C9973DFB2835976 -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.locale=ro-MD -Dtests.timezone=SystemV/AST4ADT -Dtests.asserts=true
> -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 0.25s J2 | TestIndexWriterDelete.testDeleteAllNoDeadLock
> <<<
>[junit4]> Throwable #1: java.lang.AssertionError
>[junit4]> at
> __randomizedtesting.SeedInfo.seed([7C9973DFB2835976:4F809CC13240E320]:0)
>[junit4]> at
> org.apache.lucene.index.IndexWriter.abortMerges(IndexWriter.java:2497)
>[junit4]> at
> org.apache.lucene.index.IndexWriter.deleteAll(IndexWriter.java:2424)
>[junit4]> at
> org.apache.lucene.index.RandomIndexWriter.deleteAll(RandomIndexWriter.java:373)
>[junit4]> at
> org.apache.lucene.index.TestIndexWriterDelete.testDeleteAllNoDeadLock(TestIndexWriterDelete.java:348)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:564)
>[junit4]> at java.base/java.lang.Thread.run(Thread.java:832)
>[junit4] 2> NOTE: leaving temporary files on disk at:
> /home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/build/core/test/J2/temp/lucene.index.TestIndexWriterDelete_7C9973DFB2835976-001
>[junit4] 2> NOTE: test params are: codec=Asserting(Lucene84):
> {field=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene84)),
> city=Lucene84,
> contents=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene84)),
> id=Lucene84, value=PostingsFormat(name=LuceneVarGapDocFreqInterval),
> content=PostingsFormat(name=LuceneVarGapDocFreqInterval)},
> docValues:{dv=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=2033,
> maxMBSortInHeap=6.1560152382287825,
> sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@c9cd3c9),
> locale=ro-MD, timezone=SystemV/AST4ADT
>[junit4] 2> NOTE: Linux 5.3.0-46-generic amd64/Oracle Corporation 15-ea
> (64-bit)/cpus=16,threads=1,free=389663704,total=518979584
>[junit4] 2> NOTE: All tests run in this JVM: [TestIntroSorter,
> TestNGramPhraseQuery, TestUpgradeIndexMergePolicy, TestBasicModelIne,
> TestTragicIndexWriterDeadlock, TestSimpleExplanations, TestDirectory,
> TestSegmentToThreadMapping, TestNIOFSDirectory, TestAxiomaticF3LOG,
> TestIndexOptions, TestSpanNotQuery, TestBinaryDocument,
> TestFixedLengthBytesRefArray, TestForUtil, TestParallelLeafReader,
> TestDirectoryReader, TestBytesRef, TestLongRange, TestByteSlices,
> TestRectangle2D, TestBufferedChecksum, TestIndexOrDocValuesQuery,
> TestFilterDirectoryReader, TestSleepingLockWrapper, TestDocIdSetIterator,
> TestMultiPhraseQuery, TestSearchWithThreads, TestPointQueries,
> TestForceMergeForever, TestNewestSegment, TestCrash, TestIntSet,
> Test4GBStoredFields, TestDirectPacked, TestLiveFieldValues,
> TestTopDocsCollector, Test2BPositions, TestAtomicUpdate,
> TestFSTDirectAddressing, TestXYPointQueries, TestLatLonLineShapeQueries,
> TestStressAdvance, TestBasics, TestLucene50StoredFieldsFormat,
> TestQueryRescorer, TestConstantScoreScorer, TestAssertions, TestDemo,
> TestExternalCodecs, TestMergeSchedulerExternal, TestAnalyzerWrapper,
> TestCachingTokenFilter, TestCharArrayMap, TestCharArraySet, TestCharFilter,
> TestCharacterUtils, TestDelegatingAnalyzerWrapper, TestGraphTokenFilter,
> TestGraphTokenizers, TestReusableStringReader, TestWordlistLoader,
> TestStandardAnalyzer, TestBytesRefAttImpl, TestCharTermAttributeImpl,
> TestCodecUtil, TestCompetitiveFreqNormAccumulator, TestFastCompressionMode,
> TestFastDecompressionMode, TestHighCompressionMode,
> TestLucene60FieldInfoFormat, TestLucene60PointsFormat,
> TestLucene70SegmentInfoFormat, TestDocument, TestDoubleRange,
> TestFeatureDoubleValues, TestFeatureField, TestFeatureSort, TestField,
> TestFieldType, TestFloatRange,
[jira] [Updated] (LUCENE-9164) Should not consider ACE a tragedy if IW is closed
[ https://issues.apache.org/jira/browse/LUCENE-9164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer updated LUCENE-9164: Fix Version/s: 8.5 master (9.0) Resolution: Fixed Status: Resolved (was: Patch Available) > Should not consider ACE a tragedy if IW is closed > - > > Key: LUCENE-9164 > URL: https://issues.apache.org/jira/browse/LUCENE-9164 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: master (9.0), 8.5, 8.4.2 >Reporter: Nhat Nguyen >Assignee: Nhat Nguyen >Priority: Major > Fix For: master (9.0), 8.5 > > Attachments: LUCENE-9164.patch, LUCENE-9164.patch > > Time Spent: 7h 20m > Remaining Estimate: 0h > > If IndexWriter is closed or being closed, AlreadyClosedException is expected. > We should not consider it a tragic event in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9363) TestIndexWriterDelete.testDeleteAllNoDeadLock failed on CI
[
https://issues.apache.org/jira/browse/LUCENE-9363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer resolved LUCENE-9363.
-
Assignee: Simon Willnauer
Resolution: Fixed
> TestIndexWriterDelete.testDeleteAllNoDeadLock failed on CI
> --
>
> Key: LUCENE-9363
> URL: https://issues.apache.org/jira/browse/LUCENE-9363
> Project: Lucene - Core
> Issue Type: Test
>Reporter: Simon Willnauer
>Assignee: Simon Willnauer
>Priority: Major
>
> {noformat}
> [junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestIndexWriterDelete -Dtests.method=testDeleteAllNoDeadLock
> -Dtests.seed=7C9973DFB2835976 -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.locale=ro-MD -Dtests.timezone=SystemV/AST4ADT -Dtests.asserts=true
> -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 0.25s J2 | TestIndexWriterDelete.testDeleteAllNoDeadLock
> <<<
>[junit4]> Throwable #1: java.lang.AssertionError
>[junit4]> at
> __randomizedtesting.SeedInfo.seed([7C9973DFB2835976:4F809CC13240E320]:0)
>[junit4]> at
> org.apache.lucene.index.IndexWriter.abortMerges(IndexWriter.java:2497)
>[junit4]> at
> org.apache.lucene.index.IndexWriter.deleteAll(IndexWriter.java:2424)
>[junit4]> at
> org.apache.lucene.index.RandomIndexWriter.deleteAll(RandomIndexWriter.java:373)
>[junit4]> at
> org.apache.lucene.index.TestIndexWriterDelete.testDeleteAllNoDeadLock(TestIndexWriterDelete.java:348)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:564)
>[junit4]> at java.base/java.lang.Thread.run(Thread.java:832)
>[junit4] 2> NOTE: leaving temporary files on disk at:
> /home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/build/core/test/J2/temp/lucene.index.TestIndexWriterDelete_7C9973DFB2835976-001
>[junit4] 2> NOTE: test params are: codec=Asserting(Lucene84):
> {field=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene84)),
> city=Lucene84,
> contents=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene84)),
> id=Lucene84, value=PostingsFormat(name=LuceneVarGapDocFreqInterval),
> content=PostingsFormat(name=LuceneVarGapDocFreqInterval)},
> docValues:{dv=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=2033,
> maxMBSortInHeap=6.1560152382287825,
> sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@c9cd3c9),
> locale=ro-MD, timezone=SystemV/AST4ADT
>[junit4] 2> NOTE: Linux 5.3.0-46-generic amd64/Oracle Corporation 15-ea
> (64-bit)/cpus=16,threads=1,free=389663704,total=518979584
>[junit4] 2> NOTE: All tests run in this JVM: [TestIntroSorter,
> TestNGramPhraseQuery, TestUpgradeIndexMergePolicy, TestBasicModelIne,
> TestTragicIndexWriterDeadlock, TestSimpleExplanations, TestDirectory,
> TestSegmentToThreadMapping, TestNIOFSDirectory, TestAxiomaticF3LOG,
> TestIndexOptions, TestSpanNotQuery, TestBinaryDocument,
> TestFixedLengthBytesRefArray, TestForUtil, TestParallelLeafReader,
> TestDirectoryReader, TestBytesRef, TestLongRange, TestByteSlices,
> TestRectangle2D, TestBufferedChecksum, TestIndexOrDocValuesQuery,
> TestFilterDirectoryReader, TestSleepingLockWrapper, TestDocIdSetIterator,
> TestMultiPhraseQuery, TestSearchWithThreads, TestPointQueries,
> TestForceMergeForever, TestNewestSegment, TestCrash, TestIntSet,
> Test4GBStoredFields, TestDirectPacked, TestLiveFieldValues,
> TestTopDocsCollector, Test2BPositions, TestAtomicUpdate,
> TestFSTDirectAddressing, TestXYPointQueries, TestLatLonLineShapeQueries,
> TestStressAdvance, TestBasics, TestLucene50StoredFieldsFormat,
> TestQueryRescorer, TestConstantScoreScorer, TestAssertions, TestDemo,
> TestExternalCodecs, TestMergeSchedulerExternal, TestAnalyzerWrapper,
> TestCachingTokenFilter, TestCharArrayMap, TestCharArraySet, TestCharFilter,
> TestCharacterUtils, TestDelegatingAnalyzerWrapper, TestGraphTokenFilter,
> TestGraphTokenizers, TestReusableStringReader, TestWordlistLoader,
> TestStandardAnalyzer, TestBytesRefAttImpl, TestCharTermAttributeImpl,
> TestCodecUtil, TestCompetitiveFreqNormAccumulator, TestFastCompressionMode,
> TestFastDecompressionMode, TestHighCompressionMode,
> TestLucene60FieldInfoFormat, TestLucene60PointsFormat,
> TestLucene70SegmentInfoFormat, TestDocument, TestDoubleRange,
> TestFeatureDoubleValues, TestFeatureField, TestFeatureSort, TestField,
>
[jira] [Created] (LUCENE-9511) Include StoredFieldsWriter in DWPT accounting
Simon Willnauer created LUCENE-9511:
---
Summary: Include StoredFieldsWriter in DWPT accounting
Key: LUCENE-9511
URL: https://issues.apache.org/jira/browse/LUCENE-9511
Project: Lucene - Core
Issue Type: Bug
Reporter: Simon Willnauer
StoredFieldsWriter might consume some heap space memory that can have a
significant impact on decisions made in the IW if writers should be stalled or
DWPTs should be flushed if memory settings are small in IWC and flushes are
frequent. We should add some accounting to the StoredFieldsWriter since it's
part of the DWPT lifecycle and not just present during flush.
Our nightly builds ran into some OOMs due to the large chunk size used in the
CompressedStoredFieldsFormat. The reason are very frequent flushes due to small
maxBufferedDocs which causes 300+ DWPTs to be blocked for flush causing
ultimately an OOM exception.
{noformat}
NOTE: reproduce with: ant test -Dtestcase=TestIndexingSequenceNumbers
-Dtests.method=testStressConcurrentCommit -Dtests.seed=A04943A98C8E2954
-Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true
-Dtests.locale=vo-001 -Dtests.timezone=Africa/Ouagadougou -Dtests.asserts=true
-Dtests.file.encoding=UTF8*06:06:15*[junit4] ERROR107s J3 |
TestIndexingSequenceNumbers.testStressConcurrentCommit <<<*06:06:15*
[junit4]> Throwable #1: org.apache.lucene.store.AlreadyClosedException:
this IndexWriter is closed*06:06:15*[junit4]> at
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:876)*06:06:15*
[junit4]> at
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:890)*06:06:15*
[junit4]> at
org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3727)*06:06:15*
[junit4]> at
org.apache.lucene.index.TestIndexingSequenceNumbers.testStressConcurrentCommit(TestIndexingSequenceNumbers.java:228)*06:06:15*
[junit4]> at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)*06:06:15*[junit4]>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)*06:06:15*
[junit4]> at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*06:06:15*
[junit4]> at
java.base/java.lang.reflect.Method.invoke(Method.java:566)*06:06:15*
[junit4]>at
java.base/java.lang.Thread.run(Thread.java:834)*06:06:15*[junit4]>
Caused by: java.lang.OutOfMemoryError: Java heap space*06:06:15*[junit4]
> at __randomizedtesting.SeedInfo.seed([A04943A98C8E2954]:0)*06:06:15*
[junit4]> at
org.apache.lucene.store.GrowableByteArrayDataOutput.(GrowableByteArrayDataOutput.java:46)*06:06:15*
[junit4]> at
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.(CompressingStoredFieldsWriter.java:111)*06:06:15*
[junit4]> at
org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:130)*06:06:15*
[junit4]> at
org.apache.lucene.codecs.lucene87.Lucene87StoredFieldsFormat.fieldsWriter(Lucene87StoredFieldsFormat.java:141)*06:06:15*
[junit4]>at
org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:48)*06:06:15*
[junit4]> at
org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:39)*06:06:15*
[junit4]> at
org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:46)*06:06:15*
[junit4]> at
org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:426)*06:06:15*
[junit4]> at
org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:462)*06:06:15*
[junit4]> at
org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:233)*06:06:15*
[junit4]> at
org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:419)*06:06:15*
[junit4]> at
org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1470)*06:06:15*
[junit4]>at
org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1463)*06:06:15*
[junit4]>at
org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:206)Throwable
#2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
uncaught exception in thread: Thread[id=1140, name=Thread-1095, state=RUNNABLE,
group=TGRP-TestIndexingSequenceNumbers]*06:06:15*[junit4]> Caused by:
java.lang.OutOfMemoryError: Java heap spaceThrowable #3:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught
[jira] [Commented] (LUCENE-9504) DocumentsWriterDeleteQueue.forceApplyGlobalSlice has needless locking
[
https://issues.apache.org/jira/browse/LUCENE-9504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191579#comment-17191579
]
Simon Willnauer commented on LUCENE-9504:
-
there is no need for measuring impact. This is a leftover I think. It's only
used in tests anyway.
> DocumentsWriterDeleteQueue.forceApplyGlobalSlice has needless locking
> -
>
> Key: LUCENE-9504
> URL: https://issues.apache.org/jira/browse/LUCENE-9504
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{DocumentsWriterDeleteQueue.forceApplyGlobalSlice}} is only called from
> {{getBufferedUpdateTermsSize}} so we end up doing double the locking. It's
> not a huge problem because re-entrant locks are cheap, but we can probably
> unwrap the methods a little bit here.
> This may impact HotSpot inlining, I haven't tried measuring that yet. If that
> is a concern, please let me know.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9516) Remove DocConsumer and IndexingChain from Lucene
[ https://issues.apache.org/jira/browse/LUCENE-9516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer resolved LUCENE-9516. - Fix Version/s: master (9.0) Resolution: Fixed > Remove DocConsumer and IndexingChain from Lucene > > > Key: LUCENE-9516 > URL: https://issues.apache.org/jira/browse/LUCENE-9516 > Project: Lucene - Core > Issue Type: Wish >Reporter: Simon Willnauer >Priority: Major > Fix For: master (9.0) > > Time Spent: 20m > Remaining Estimate: 0h > > Disclaimer: This is a breaking change! > We allow today to replace the entire indexing chain which is a fundamental > part of our software. I personally don't know if there are any users of this > API. but given the complexity I personally don't think we should further > support this. If you are willing to implement this entire thing yourself I > really wonder if you are better off building lucene from the source. An > option like this on IWC might make users look into it while I am convinced > they shouldn't. It's too complex and nothing is made for reuse down there. I > wonder what others think but from my perspective it's time to remove it in 9.0 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9515) Detach DWPT from DefaultIndexingChain
[ https://issues.apache.org/jira/browse/LUCENE-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer resolved LUCENE-9515. - Resolution: Fixed > Detach DWPT from DefaultIndexingChain > -- > > Key: LUCENE-9515 > URL: https://issues.apache.org/jira/browse/LUCENE-9515 > Project: Lucene - Core > Issue Type: Improvement >Affects Versions: master (9.0), 8.7 >Reporter: Simon Willnauer >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > DefaultIndexingChain or rather its super class DocConsumer is still tightly > coupled to DocumentsWriterPerThread. In oder to guarantee better code > isolation and hopefully independent testing down the road we can expand the > IndexingChain interface to take all the primitives needed rather than the > complex DWPT class. It's unclear anyway how this API (DocConsumer and > IndexingChain) should be implemented with all the classes it depends on being > package private. I will open a follow-up to discuss if we can remove that > abstraction. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9514) Include TermVectorsWriter in DWPT accounting
[ https://issues.apache.org/jira/browse/LUCENE-9514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer resolved LUCENE-9514. - Fix Version/s: 8.7 master (9.0) Resolution: Fixed > Include TermVectorsWriter in DWPT accounting > > > Key: LUCENE-9514 > URL: https://issues.apache.org/jira/browse/LUCENE-9514 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Simon Willnauer >Priority: Major > Fix For: master (9.0), 8.7 > > Time Spent: 1h > Remaining Estimate: 0h > > As a followup of LUCENE-9511 we should also include the TermVectorsWriter > into the DWPT accounting since it might hold on to RAM in it's byte buffer > recycling etc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9502) Can we remove JaspellLookup?
Simon Willnauer created LUCENE-9502: --- Summary: Can we remove JaspellLookup? Key: LUCENE-9502 URL: https://issues.apache.org/jira/browse/LUCENE-9502 Project: Lucene - Core Issue Type: Wish Affects Versions: master (9.0) Reporter: Simon Willnauer We deprecated JaspellLookup in version 4.9 - I do wonder if it's time to get rid of it? I can see it's still used in solr but I don't see if it's deprecated there. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9514) Include TermVectorsWriter in DWPT accounting
Simon Willnauer created LUCENE-9514: --- Summary: Include TermVectorsWriter in DWPT accounting Key: LUCENE-9514 URL: https://issues.apache.org/jira/browse/LUCENE-9514 Project: Lucene - Core Issue Type: Improvement Reporter: Simon Willnauer As a followup of LUCENE-9511 we should also include the TermVectorsWriter into the DWPT accounting since it might hold on to RAM in it's byte buffer recycling etc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9511) Include StoredFieldsWriter in DWPT accounting
[
https://issues.apache.org/jira/browse/LUCENE-9511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192701#comment-17192701
]
Simon Willnauer commented on LUCENE-9511:
-
{quote}I guess this would mean applications using more threads to index would
have seen RAM consumed but not tracked properly by {{IndexWriter}} until this
fix.
{quote}
That is true. I think it's insignificant for most of our users since it only
really comes into play when you are tons of flushes and a large number of
blocked DWPT that are otherwise flush pending. That situation requires slow
flushes and slow commits as well as the situation that the DWPTs that are
flushing or pending are very small by itself ie. only a couple of small docs.
Otherwise stall control would kick in earlier and prevent adding more docs.
Still lucene is used in so many environments and having large amounts of
threads concurrently might add up here.
> Include StoredFieldsWriter in DWPT accounting
> -
>
> Key: LUCENE-9511
> URL: https://issues.apache.org/jira/browse/LUCENE-9511
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Simon Willnauer
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> StoredFieldsWriter might consume some heap space memory that can have a
> significant impact on decisions made in the IW if writers should be stalled
> or DWPTs should be flushed if memory settings are small in IWC and flushes
> are frequent. We should add some accounting to the StoredFieldsWriter since
> it's part of the DWPT lifecycle and not just present during flush.
> Our nightly builds ran into some OOMs due to the large chunk size used in the
> CompressedStoredFieldsFormat. The reason are very frequent flushes due to
> small maxBufferedDocs which causes 300+ DWPTs to be blocked for flush causing
> ultimately an OOM exception.
> {noformat}
>
> NOTE: reproduce with: ant test -Dtestcase=TestIndexingSequenceNumbers
> -Dtests.method=testStressConcurrentCommit -Dtests.seed=A04943A98C8E2954
> -Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true
> -Dtests.locale=vo-001 -Dtests.timezone=Africa/Ouagadougou
> -Dtests.asserts=true -Dtests.file.encoding=UTF8*06:06:15*[junit4] ERROR
> 107s J3 | TestIndexingSequenceNumbers.testStressConcurrentCommit
> <<<*06:06:15*[junit4]> Throwable #1:
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is
> closed*06:06:15*[junit4]>at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:876)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:890)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3727)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.TestIndexingSequenceNumbers.testStressConcurrentCommit(TestIndexingSequenceNumbers.java:228)*06:06:15*
> [junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)*06:06:15*[junit4]>at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)*06:06:15*
> [junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*06:06:15*
> [junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)*06:06:15*
> [junit4]>at
> java.base/java.lang.Thread.run(Thread.java:834)*06:06:15*[junit4]>
> Caused by: java.lang.OutOfMemoryError: Java heap space*06:06:15*[junit4]
> > at
> __randomizedtesting.SeedInfo.seed([A04943A98C8E2954]:0)*06:06:15*[junit4]
>> at
> org.apache.lucene.store.GrowableByteArrayDataOutput.(GrowableByteArrayDataOutput.java:46)*06:06:15*
> [junit4]> at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.(CompressingStoredFieldsWriter.java:111)*06:06:15*
> [junit4]> at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:130)*06:06:15*
> [junit4]> at
> org.apache.lucene.codecs.lucene87.Lucene87StoredFieldsFormat.fieldsWriter(Lucene87StoredFieldsFormat.java:141)*06:06:15*
> [junit4]>at
> org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:48)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:39)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:46)*06:06:15*
> [junit4]> at
>
[jira] [Comment Edited] (LUCENE-9511) Include StoredFieldsWriter in DWPT accounting
[
https://issues.apache.org/jira/browse/LUCENE-9511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192701#comment-17192701
]
Simon Willnauer edited comment on LUCENE-9511 at 9/9/20, 7:35 AM:
--
{quote}I guess this would mean applications using more threads to index would
have seen RAM consumed but not tracked properly by {{IndexWriter}} until this
fix.
{quote}
That is true. I think it's insignificant for most of our users since it only
really comes into play when you are tons of flushes and a large number of
blocked DWPT that are otherwise flush pending. That situation requires slow
flushes and slow commits as well as the situation that the DWPTs that are
flushing or pending are very small by itself ie. only a couple of small docs.
Otherwise stall control would kick in earlier and prevent adding more docs.
Still lucene is used in so many environments and having large amounts of
threads concurrently might add up here.
What I am more concerned about is that if we have a bug in the code causing
large amounts of memory allocated will likely be undetected unless we see some
slowdowns due to flushing and it will help here to detect this, I hope :)
was (Author: simonw):
{quote}I guess this would mean applications using more threads to index would
have seen RAM consumed but not tracked properly by {{IndexWriter}} until this
fix.
{quote}
That is true. I think it's insignificant for most of our users since it only
really comes into play when you are tons of flushes and a large number of
blocked DWPT that are otherwise flush pending. That situation requires slow
flushes and slow commits as well as the situation that the DWPTs that are
flushing or pending are very small by itself ie. only a couple of small docs.
Otherwise stall control would kick in earlier and prevent adding more docs.
Still lucene is used in so many environments and having large amounts of
threads concurrently might add up here.
> Include StoredFieldsWriter in DWPT accounting
> -
>
> Key: LUCENE-9511
> URL: https://issues.apache.org/jira/browse/LUCENE-9511
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Simon Willnauer
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> StoredFieldsWriter might consume some heap space memory that can have a
> significant impact on decisions made in the IW if writers should be stalled
> or DWPTs should be flushed if memory settings are small in IWC and flushes
> are frequent. We should add some accounting to the StoredFieldsWriter since
> it's part of the DWPT lifecycle and not just present during flush.
> Our nightly builds ran into some OOMs due to the large chunk size used in the
> CompressedStoredFieldsFormat. The reason are very frequent flushes due to
> small maxBufferedDocs which causes 300+ DWPTs to be blocked for flush causing
> ultimately an OOM exception.
> {noformat}
>
> NOTE: reproduce with: ant test -Dtestcase=TestIndexingSequenceNumbers
> -Dtests.method=testStressConcurrentCommit -Dtests.seed=A04943A98C8E2954
> -Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true
> -Dtests.locale=vo-001 -Dtests.timezone=Africa/Ouagadougou
> -Dtests.asserts=true -Dtests.file.encoding=UTF8*06:06:15*[junit4] ERROR
> 107s J3 | TestIndexingSequenceNumbers.testStressConcurrentCommit
> <<<*06:06:15*[junit4]> Throwable #1:
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is
> closed*06:06:15*[junit4]>at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:876)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:890)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3727)*06:06:15*
> [junit4]> at
> org.apache.lucene.index.TestIndexingSequenceNumbers.testStressConcurrentCommit(TestIndexingSequenceNumbers.java:228)*06:06:15*
> [junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)*06:06:15*[junit4]>at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)*06:06:15*
> [junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*06:06:15*
> [junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)*06:06:15*
> [junit4]>at
> java.base/java.lang.Thread.run(Thread.java:834)*06:06:15*[junit4]>
> Caused by: java.lang.OutOfMemoryError: Java heap space*06:06:15*[junit4]
> > at
> __randomizedtesting.SeedInfo.seed([A04943A98C8E2954]:0)*06:06:15*[junit4]
>> at
>
[jira] [Created] (LUCENE-9515) Detach DWPT from DefaultIndexingChain
Simon Willnauer created LUCENE-9515: --- Summary: Detach DWPT from DefaultIndexingChain Key: LUCENE-9515 URL: https://issues.apache.org/jira/browse/LUCENE-9515 Project: Lucene - Core Issue Type: Improvement Affects Versions: master (9.0), 8.7 Reporter: Simon Willnauer DefaultIndexingChain or rather its super class DocConsumer is still tightly coupled to DocumentsWriterPerThread. In oder to guarantee better code isolation and hopefully independent testing down the road we can expand the IndexingChain interface to take all the primitives needed rather than the complex DWPT class. It's unclear anyway how this API (DocConsumer and IndexingChain) should be implemented with all the classes it depends on being package private. I will open a follow-up to discuss if we can remove that abstraction. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9516) Remove DocConsumer and IndexingChain from Lucene
Simon Willnauer created LUCENE-9516: --- Summary: Remove DocConsumer and IndexingChain from Lucene Key: LUCENE-9516 URL: https://issues.apache.org/jira/browse/LUCENE-9516 Project: Lucene - Core Issue Type: Wish Reporter: Simon Willnauer Disclaimer: This is a breaking change! We allow today to replace the entire indexing chain which is a fundamental part of our software. I personally don't know if there are any users of this API. but given the complexity I personally don't think we should further support this. If you are willing to implement this entire thing yourself I really wonder if you are better off building lucene from the source. An option like this on IWC might make users look into it while I am convinced they shouldn't. It's too complex and nothing is made for reuse down there. I wonder what others think but from my perspective it's time to remove it in 9.0 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9484) Allow index sorting to happen after the fact
Simon Willnauer created LUCENE-9484: --- Summary: Allow index sorting to happen after the fact Key: LUCENE-9484 URL: https://issues.apache.org/jira/browse/LUCENE-9484 Project: Lucene - Core Issue Type: Improvement Reporter: Simon Willnauer I did look into sorting an index after it was created and found that with some smallish modifications we can actually allow that by piggibacking on SortingLeafReader and addIndices in a pretty straight-forward and simple way. With some smallish modifications / fixes to SortingLeafReader we can just merge and unsorted index into a sorted index using a fresh index writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9469) Use RandomMockMergePolicy more often
Simon Willnauer created LUCENE-9469:
---
Summary: Use RandomMockMergePolicy more often
Key: LUCENE-9469
URL: https://issues.apache.org/jira/browse/LUCENE-9469
Project: Lucene - Core
Issue Type: Test
Reporter: Simon Willnauer
During the work on LUCENE-8962 we had some discussions about testability of the
feature and how well we can guarantee that it's tested. One argument was that
we can rely to a certain extend on the randomization and enabling these
features by swapping in RandomMockMergePolicy.
I ran a test which throws an assertion error every time the feature is
explicitly used which required MockRandomMergePolicy to be used. Unfortunately
I had two entire test runs without any failure except of the tests that
explicitly enabled it ie. the ones that I wrote for the feature.
I think we are not using this MP often enough in our tests and I want to
propose to use it way more frequently. It's certainly not a replacement for
dedicated unit tests, I wrote a dedicated one for every random failure I found
which should be common practice but it would be great to increase coverage by
swapping in MockRandomMergePolicy more often. Maybe something as simple as this
would do it:
{noformat}
--- a/lucene/test-framework/src/java/org/apache/lucene/util/LuceneTestCase.java
+++ b/lucene/test-framework/src/java/org/apache/lucene/util/LuceneTestCase.java
@@ -1059,7 +1059,7 @@ public abstract class LuceneTestCase extends Assert {
}
public static MergePolicy newMergePolicy(Random r, boolean includeMockMP) {
-if (includeMockMP && rarely(r)) {
+if (includeMockMP && r.nextBoolean()) {
return new MockRandomMergePolicy(r);
} else if (r.nextBoolean()) {
return newTieredMergePolicy(r);
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9470) TestXYMultiPolygonShapeQueries reproducing failure
[
https://issues.apache.org/jira/browse/LUCENE-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17181921#comment-17181921
]
Simon Willnauer commented on LUCENE-9470:
-
I found another one on our CI:
{noformat}2:13:29[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestXYMultiPolygonShapeQueries -Dtests.method=testRandomBig
-Dtests.seed=8B7FC18414411131 -Dtests.nightly=true -Dtests.slow=true
-Dtests.badapples=true -Dtests.locale=en-MU -Dtests.timezone=America/Havana
-Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
12:13:29[junit4] FAILURE 128s J2 |
TestXYMultiPolygonShapeQueries.testRandomBig <<<
12:13:29[junit4]> Throwable #1: java.lang.AssertionError: wrong hit
(first of possibly more):
12:13:29[junit4]> FAIL: id=6812 should match but did not
12:13:29[junit4]> relation=CONTAINS
{noformat}
> TestXYMultiPolygonShapeQueries reproducing failure
> --
>
> Key: LUCENE-9470
> URL: https://issues.apache.org/jira/browse/LUCENE-9470
> Project: Lucene - Core
> Issue Type: Improvement
>Affects Versions: master (9.0)
>Reporter: Michael McCandless
>Priority: Major
>
> I hit this while beasting tests against [~simonw]'s PR to add "merge segments
> during getReader":
> It reproduces for me on current
> ({{77a4d495cc553ec80001346376fd87d6b73a6059}}) mainline:
> {noformat}
> [junit4:pickseed] Seed property 'tests.seed' already defined:
> 50C3891F4E0641A4
> [junit4] says g'day! Master seed: 50C3891F4E0641A4
> [junit4] Executing 1 suite with 1 JVM.
> [junit4]
> [junit4] Started J0 PID(1993475@localhost).
> [junit4] Suite: org.apache.lucene.document.TestXYMultiPolygonShapeQueries
> [junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestXYMultiPolygonShapeQueries -Dtests.method=testRandomBig
> -Dtests.seed=50C3891F4E0641A4 -Dtests.nightly=true -Dtests.slow=true
> -Dtests.badap\
> ples=true
> -Dtests.linedocsfile=/l/simon/lucene/test-framework/src/resources/org/apache/lucene/util/2000mb.txt.gz
> -Dtests.locale=en-IM -Dtests.timezone=America/Lima -Dtests.asserts=true
> -Dtests.file.enco\
> ding=UTF-8
> [junit4] FAILURE 51.3s | TestXYMultiPolygonShapeQueries.testRandomBig <<<
> [junit4] > Throwable #1: java.lang.AssertionError: wrong hit (first of
> possibly more):
> [junit4] > FAIL: id=8233 should match but did not
> [junit4] > relation=CONTAINS
> [junit4] > query=XYShapeQuery: field=shape:[XYLINE([3.402823E38,
> 3.4028235E38][3.402823E38, 3.4028235E38][3.402823E38,
> 3.4028235E38][3.402823E38, 3.4028235E38][3.402823E38, 3.4028235E38][3.40282\
> 3E38, 3.4028235E38][3.402823E38, 3.4028235E38][3.402823E38,
> 3.4028235E38][3.402823E38, 3.4028235E38][3.402823E38,
> 3.4028235E38][3.402823E38, 3.4028235E38][3.402823E38,
> 3.4028235E38][3.402823E38, 3.40282\
> 35E38][3.402823E38, 3.4028235E38][3.402823E38, 3.4028235E38][3.402823E38,
> 3.4028235E38][3.402823E38, 3.4028235E38][3.4028229E38,
> 3.4028235E38][3.4028229E38, 3.4028235E38][3.4028229E38, 3.4028235E38][3.4\
> 028229E38, 3.4028235E38][3.4028229E38, 3.4028235E38][3.4028229E38,
> 3.4028235E38][3.4028229E38, 3.4028235E38][3.4028229E38,
> 3.4028235E38][3.4028229E38, 3.4028235E38][3.4028229E38,
> 3.4028235E38][3.4028229\
> E38, 3.4028235E38][3.4028229E38, 3.4028235E38][3.4028229E38,
> 3.4028235E38][3.4028229E38, 3.4028235E38][3.4028229E38,
> 3.4028235E38][3.4028229E38, 3.4028235E38][3.4028229E38,
> 3.4028235E38][3.4028229E38, 3\
> .4028235E38][3.4028229E38, 3.4028235E38][3.4028229E38,
> 3.4028235E38][3.4028229E38, 3.4028235E38][3.4028229E38,
> 3.4028235E38][3.4028229E38, 3.4028235E38][3.4028229E38,
> 3.4028235E38][3.4028229E38, 3.40282\
> 35E38][3.4028229E38, 3.4028235E38][3.4028229E38, 3.4028235E38][3.4028229E38,
> 3.4028235E38][3.4028229E38, 3.4028235E38][3.4028229E38,
> 3.4028235E38][3.4028229E38, 3.4028233E38][3.4028229E38, 3.4028233E38]\
> [3.4028229E38, 3.4028233E38][3.4028229E38, 3.4028233E38][3.4028229E38,
> 3.4028233E38][3.4028229E38, 3.4028233E38][3.4028229E38,
> 3.4028233E38][3.4028229E38, 3.4028233E38][3.4028229E38, 3.4028233E38][3.402\
> 8227E38, 3.4028233E38][3.4028227E38, 3.4028233E38][3.4028227E38,
> 3.4028233E38][3.4028227E38, 3.4028233E38][3.4028227E38,
> 3.4028233E38][3.4028227E38, 3.4028233E38][3.4028227E38,
> 3.4028233E38][3.4028227E3\
> 8, 3.4028233E38][3.4028227E38, 3.4028233E38][3.4028227E38,
> 3.4028233E38][3.4028227E38, 3.4028233E38][3.4028227E38,
> 3.4028233E38][3.4028227E38, 3.4028233E38][3.4028227E38,
> 3.4028233E38][3.4028227E38, 3.4\
> 028233E38][3.4028227E38, 3.4028233E38][3.4028227E38,
> 3.4028233E38][3.4028227E38, 3.4028233E38][3.4028227E38,
> 3.4028233E38][3.4028227E38, 3.4028233E38][3.4028227E38,
> 3.4028233E38][3.4028227E38, 3.4028233\
> ...
> 3E38][3.4028235E38, 3.4028233E38][3.4028235E38, 3.4028233E38][3.4028235E38,
>
[jira] [Created] (LUCENE-9473) TestIndexWriterDelete#testDeleteAllNoDeadLock fails sometimes on nightly
Simon Willnauer created LUCENE-9473:
---
Summary: TestIndexWriterDelete#testDeleteAllNoDeadLock fails
sometimes on nightly
Key: LUCENE-9473
URL: https://issues.apache.org/jira/browse/LUCENE-9473
Project: Lucene - Core
Issue Type: Test
Reporter: Simon Willnauer
{noformat}
2> NOTE: reproduce with: ant test -Dtestcase=TestIndexWriterDelete
-Dtests.method=testDeleteAllNoDeadLock -Dtests.seed=E595F5BCCD45156B
-Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true
-Dtests.linedocsfile=/l/simon/lucene/test-framework/src/resources/org/apache/lucene/util/2000mb.txt.gz
-Dtests.locale=da-DK -Dtests.timezone=SST -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
2> aug. 19, 2020 8:24:20 PM
com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
uncaughtException
2> WARNING: Uncaught exception in thread:
Thread[Thread-8,5,TGRP-TestIndexWriterDelete]
2> java.lang.RuntimeException:
org.apache.lucene.store.AlreadyClosedException: this DocumentsWriter is closed
2>at __randomizedtesting.SeedInfo.seed([E595F5BCCD45156B]:0)
2>at
org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:332)
2> Caused by: org.apache.lucene.store.AlreadyClosedException: this
DocumentsWriter is closed
2>at
org.apache.lucene.index.DocumentsWriter.ensureOpen(DocumentsWriter.java:202)
2>at
org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:416)
2>at
org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1470)
2>at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1756)
2>at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1399)
2>at
org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:193)
2>at
org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:326)
2>
2> aug. 19, 2020 8:24:20 PM
com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
uncaughtException
2> WARNING: Uncaught exception in thread:
Thread[Thread-7,5,TGRP-TestIndexWriterDelete]
2> java.lang.RuntimeException:
org.apache.lucene.store.AlreadyClosedException: DWPTPool is already closed
2>at __randomizedtesting.SeedInfo.seed([E595F5BCCD45156B]:0)
2>at
org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:332)
2> Caused by: org.apache.lucene.store.AlreadyClosedException: DWPTPool is
already closed
2>at
org.apache.lucene.index.DocumentsWriterPerThreadPool.ensureOpen(DocumentsWriterPerThreadPool.java:134)
2>at
org.apache.lucene.index.DocumentsWriterPerThreadPool.newWriter(DocumentsWriterPerThreadPool.java:102)
2>at
org.apache.lucene.index.DocumentsWriterPerThreadPool.getAndLock(DocumentsWriterPerThreadPool.java:128)
2>at
org.apache.lucene.index.DocumentsWriterFlushControl.obtainAndLock(DocumentsWriterFlushControl.java:445)
2>at
org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:409)
2>at
org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1470)
2>at
org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1443)
2>at
org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:163)
2>at
org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:326)
2>
2> aug. 19, 2020 8:24:20 PM
com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
uncaughtException
2> WARNING: Uncaught exception in thread: Thread[Lucene Merge Thread
#9,5,TGRP-TestIndexWriterDelete]
2> org.apache.lucene.index.MergePolicy$MergeException:
org.apache.lucene.index.CorruptIndexException: Problem reading index.
(resource=_di.cfe)
2>at __randomizedtesting.SeedInfo.seed([E595F5BCCD45156B]:0)
2>at
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
2>at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:694)
2> Caused by: org.apache.lucene.index.CorruptIndexException: Problem reading
index. (resource=_di.cfe)
2>at
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:144)
2>at org.apache.lucene.index.SegmentReader.(SegmentReader.java:83)
2>at
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:171)
2>at
org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:714)
2>at
org.apache.lucene.index.IndexWriter.lambda$mergeMiddle$16(IndexWriter.java:4672)
2>at
org.apache.lucene.index.MergePolicy$OneMerge.initMergeReaders(MergePolicy.java:438)
2>at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4669)
2>at
[jira] [Created] (LUCENE-9477) IndexWriter might leave broken segments file behind on exception during rollback
Simon Willnauer created LUCENE-9477:
---
Summary: IndexWriter might leave broken segments file behind on
exception during rollback
Key: LUCENE-9477
URL: https://issues.apache.org/jira/browse/LUCENE-9477
Project: Lucene - Core
Issue Type: Bug
Reporter: Simon Willnauer
Mike ran some beasty tests while I was working on LUCENE-8962. This test caused
some headaches since it only rarely also fails on master:
{noformat}
org.apache.lucene.index.TestIndexWriterOnVMError > testUnknownError FAILED
org.apache.lucene.index.CorruptIndexException: Unexpected file read error
while reading index.
(resource=BufferedChecksumIndexInput(MockIndexInputWrapper((clone of)
ByteBuffersIndexInput (file=pending_segments_2, buffers\
=258 bytes, block size: 1, blocks: 1, position: 0
at
__randomizedtesting.SeedInfo.seed([587A104EFE0C57E1:B32CCFCEFC8BC1D1]:0)
at
org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:300)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:521)
at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:301)
at
org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:836)
at
org.apache.lucene.index.TestIndexWriterOnVMError.doTest(TestIndexWriterOnVMError.java:89)
at
org.apache.lucene.index.TestIndexWriterOnVMError.testUnknownError(TestIndexWriterOnVMError.java:251)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at
[jira] [Resolved] (LUCENE-9473) TestIndexWriterDelete#testDeleteAllNoDeadLock fails sometimes on nightly
[
https://issues.apache.org/jira/browse/LUCENE-9473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer resolved LUCENE-9473.
-
Fix Version/s: 8.7
master (9.0)
Resolution: Fixed
> TestIndexWriterDelete#testDeleteAllNoDeadLock fails sometimes on nightly
>
>
> Key: LUCENE-9473
> URL: https://issues.apache.org/jira/browse/LUCENE-9473
> Project: Lucene - Core
> Issue Type: Test
>Reporter: Simon Willnauer
>Priority: Major
> Fix For: master (9.0), 8.7
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {noformat}
> 2> NOTE: reproduce with: ant test -Dtestcase=TestIndexWriterDelete
> -Dtests.method=testDeleteAllNoDeadLock -Dtests.seed=E595F5BCCD45156B
> -Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true
> -Dtests.linedocsfile=/l/simon/lucene/test-framework/src/resources/org/apache/lucene/util/2000mb.txt.gz
> -Dtests.locale=da-DK -Dtests.timezone=SST -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> 2> aug. 19, 2020 8:24:20 PM
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
> uncaughtException
> 2> WARNING: Uncaught exception in thread:
> Thread[Thread-8,5,TGRP-TestIndexWriterDelete]
> 2> java.lang.RuntimeException:
> org.apache.lucene.store.AlreadyClosedException: this DocumentsWriter is closed
> 2> at __randomizedtesting.SeedInfo.seed([E595F5BCCD45156B]:0)
> 2> at
> org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:332)
> 2> Caused by: org.apache.lucene.store.AlreadyClosedException: this
> DocumentsWriter is closed
> 2> at
> org.apache.lucene.index.DocumentsWriter.ensureOpen(DocumentsWriter.java:202)
> 2> at
> org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:416)
> 2> at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1470)
> 2> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1756)
> 2> at
> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1399)
> 2> at
> org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:193)
> 2> at
> org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:326)
> 2>
> 2> aug. 19, 2020 8:24:20 PM
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
> uncaughtException
> 2> WARNING: Uncaught exception in thread:
> Thread[Thread-7,5,TGRP-TestIndexWriterDelete]
> 2> java.lang.RuntimeException:
> org.apache.lucene.store.AlreadyClosedException: DWPTPool is already closed
> 2> at __randomizedtesting.SeedInfo.seed([E595F5BCCD45156B]:0)
> 2> at
> org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:332)
> 2> Caused by: org.apache.lucene.store.AlreadyClosedException: DWPTPool is
> already closed
> 2> at
> org.apache.lucene.index.DocumentsWriterPerThreadPool.ensureOpen(DocumentsWriterPerThreadPool.java:134)
> 2> at
> org.apache.lucene.index.DocumentsWriterPerThreadPool.newWriter(DocumentsWriterPerThreadPool.java:102)
> 2> at
> org.apache.lucene.index.DocumentsWriterPerThreadPool.getAndLock(DocumentsWriterPerThreadPool.java:128)
> 2> at
> org.apache.lucene.index.DocumentsWriterFlushControl.obtainAndLock(DocumentsWriterFlushControl.java:445)
> 2> at
> org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:409)
> 2> at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1470)
> 2> at
> org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1443)
> 2> at
> org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:163)
> 2> at
> org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:326)
> 2>
> 2> aug. 19, 2020 8:24:20 PM
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
> uncaughtException
> 2> WARNING: Uncaught exception in thread: Thread[Lucene Merge Thread
> #9,5,TGRP-TestIndexWriterDelete]
> 2> org.apache.lucene.index.MergePolicy$MergeException:
> org.apache.lucene.index.CorruptIndexException: Problem reading index.
> (resource=_di.cfe)
> 2> at __randomizedtesting.SeedInfo.seed([E595F5BCCD45156B]:0)
> 2> at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
> 2> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:694)
> 2> Caused by: org.apache.lucene.index.CorruptIndexException: Problem
> reading index. (resource=_di.cfe)
> 2> at
> org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:144)
> 2> at org.apache.lucene.index.SegmentReader.(SegmentReader.java:83)
> 2>
[jira] [Created] (LUCENE-9478) DWPTDeleteQueue leaks memory
Simon Willnauer created LUCENE-9478: --- Summary: DWPTDeleteQueue leaks memory Key: LUCENE-9478 URL: https://issues.apache.org/jira/browse/LUCENE-9478 Project: Lucene - Core Issue Type: Bug Affects Versions: 8.6.1, master (9.0), 8.7 Reporter: Simon Willnauer We introduced a broken behavior that leaks memory in DWPTDeleteQueue recently (LUCENE-9304) The line [here|https://github.com/apache/lucene-solr/blob/18af6325ed98b50c23a1f026ab9d05f17039c731/lucene/core/src/java/org/apache/lucene/index/DocumentsWriterDeleteQueue.java#L594-L596] tries to be smart and even adds a comment that the current impl is safe but it isn't if you reference a final member in a lambda it references _this_ which holds on to the object. I think we need to do a quick 8.6.2 for this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9477) IndexWriter might leave broken segments file behind on exception during rollback
[
https://issues.apache.org/jira/browse/LUCENE-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer resolved LUCENE-9477.
-
Fix Version/s: 8.7
master (9.0)
Resolution: Fixed
> IndexWriter might leave broken segments file behind on exception during
> rollback
>
>
> Key: LUCENE-9477
> URL: https://issues.apache.org/jira/browse/LUCENE-9477
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Simon Willnauer
>Priority: Major
> Fix For: master (9.0), 8.7
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Mike ran some beasty tests while I was working on LUCENE-8962. This test
> caused some headaches since it only rarely also fails on master:
> {noformat}
> org.apache.lucene.index.TestIndexWriterOnVMError > testUnknownError FAILED
> org.apache.lucene.index.CorruptIndexException: Unexpected file read error
> while reading index.
> (resource=BufferedChecksumIndexInput(MockIndexInputWrapper((clone of)
> ByteBuffersIndexInput (file=pending_segments_2, buffers\
> =258 bytes, block size: 1, blocks: 1, position: 0
> at
> __randomizedtesting.SeedInfo.seed([587A104EFE0C57E1:B32CCFCEFC8BC1D1]:0)
> at
> org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:300)
> at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:521)
> at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:301)
> at
> org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:836)
> at
> org.apache.lucene.index.TestIndexWriterOnVMError.doTest(TestIndexWriterOnVMError.java:89)
> at
> org.apache.lucene.index.TestIndexWriterOnVMError.testUnknownError(TestIndexWriterOnVMError.java:251)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
> at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
>
[jira] [Resolved] (LUCENE-9478) DWPTDeleteQueue leaks memory
[ https://issues.apache.org/jira/browse/LUCENE-9478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer resolved LUCENE-9478. - Fix Version/s: 8.6.2 8.7 master (9.0) Lucene Fields: New,Patch Available (was: New) Assignee: Simon Willnauer Resolution: Fixed > DWPTDeleteQueue leaks memory > > > Key: LUCENE-9478 > URL: https://issues.apache.org/jira/browse/LUCENE-9478 > Project: Lucene - Core > Issue Type: Bug >Affects Versions: master (9.0), 8.7, 8.6.1 >Reporter: Simon Willnauer >Assignee: Simon Willnauer >Priority: Blocker > Fix For: master (9.0), 8.7, 8.6.2 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > We introduced a broken behavior that leaks memory in DWPTDeleteQueue recently > (LUCENE-9304) The line > [here|https://github.com/apache/lucene-solr/blob/18af6325ed98b50c23a1f026ab9d05f17039c731/lucene/core/src/java/org/apache/lucene/index/DocumentsWriterDeleteQueue.java#L594-L596] > tries to be smart and even adds a comment that the current impl is safe but > it isn't if you reference a final member in a lambda it references _this_ > which holds on to the object. I think we need to do a quick 8.6.2 for this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9467) TestOfflineSorter fails with combination of NRTCaching and ByteBuffersDirectory
[
https://issues.apache.org/jira/browse/LUCENE-9467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17178876#comment-17178876
]
Simon Willnauer commented on LUCENE-9467:
-
one option to fix this is to use fileLength instead of opening a file.
{noformat}
diff --git
a/lucene/core/src/java/org/apache/lucene/store/NRTCachingDirectory.java
b/lucene/core/src/java/org/apache/lucene/store/NRTCachingDirectory.java
index 597ea9d5f2e..40e798959c1 100644
--- a/lucene/core/src/java/org/apache/lucene/store/NRTCachingDirectory.java
+++ b/lucene/core/src/java/org/apache/lucene/store/NRTCachingDirectory.java
@@ -277,7 +277,7 @@ public class NRTCachingDirectory extends FilterDirectory
implements Accountable
* there's some unexpected error. */
static boolean slowFileExists(Directory dir, String fileName) throws
IOException {
try {
- dir.openInput(fileName, IOContext.DEFAULT).close();
+ dir.fileLength(fileName);
return true;
} catch (NoSuchFileException | FileNotFoundException e) {
return false;
{noformat}
> TestOfflineSorter fails with combination of NRTCaching and
> ByteBuffersDirectory
> ---
>
> Key: LUCENE-9467
> URL: https://issues.apache.org/jira/browse/LUCENE-9467
> Project: Lucene - Core
> Issue Type: Bug
>Affects Versions: 8.7, 8.6.1
>Reporter: Simon Willnauer
>Priority: Major
>
> TestOfflineSorter#testThreadSafety fails on 8.x since NRTCachingDirectory
> uses the delegates openInput method to slowly check if the file exists. This
> is not present anymore on main line. The problem is that ByteBuffersDirectory
> fires an AccessDeniedException when the file is not fully written yet.
> {noformat}
> 00:27:31[junit4] Suite: org.apache.lucene.util.TestOfflineSorter
> 00:27:31[junit4] 2> KMN 14, 2020 6:27:30 AM
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
> uncaughtException
> 00:27:31[junit4] 2> WARNING: Uncaught exception in thread:
> Thread[Thread-7318,5,TGRP-TestOfflineSorter]
> 00:27:31[junit4] 2> java.lang.RuntimeException:
> java.nio.file.AccessDeniedException: Can't open a file still open for
> writing: unsorted_tmp_6.tmp
> 00:27:31[junit4] 2> at
> __randomizedtesting.SeedInfo.seed([B1129484013118F7]:0)
> 00:27:31[junit4] 2> at
> org.apache.lucene.util.TestOfflineSorter$2.run(TestOfflineSorter.java:272)
> 00:27:31[junit4] 2> Caused by: java.nio.file.AccessDeniedException:
> Can't open a file still open for writing: unsorted_tmp_6.tmp
> 00:27:31[junit4] 2> at
> org.apache.lucene.store.ByteBuffersDirectory$FileEntry.openInput(ByteBuffersDirectory.java:251)
> 00:27:31[junit4] 2> at
> org.apache.lucene.store.ByteBuffersDirectory.openInput(ByteBuffersDirectory.java:218)
> 00:27:31[junit4] 2> at
> org.apache.lucene.store.FileSwitchDirectory.openInput(FileSwitchDirectory.java:233)
> 00:27:31[junit4] 2> at
> org.apache.lucene.store.NRTCachingDirectory.slowFileExists(NRTCachingDirectory.java:280)
> 00:27:31[junit4] 2> at
> org.apache.lucene.store.NRTCachingDirectory.createTempOutput(NRTCachingDirectory.java:254)
> 00:27:31[junit4] 2> at
> org.apache.lucene.store.MockDirectoryWrapper.createTempOutput(MockDirectoryWrapper.java:707)
> 00:27:31[junit4] 2> at
> org.apache.lucene.util.TestOfflineSorter.checkSort(TestOfflineSorter.java:184)
> 00:27:31[junit4] 2> at
> org.apache.lucene.util.TestOfflineSorter.access$100(TestOfflineSorter.java:48)
> 00:27:31[junit4] 2> at
> org.apache.lucene.util.TestOfflineSorter$2.run(TestOfflineSorter.java:268)
> 00:27:31[junit4] 2>
> 00:27:31[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestOfflineSorter -Dtests.method=testThreadSafety
> -Dtests.seed=B1129484013118F7 -Dtests.nightly=true -Dtests.slow=true
> -Dtests.badapples=true -Dtests.locale=cgg-UG -Dtests.timezone=Asia/Dacca
> -Dtests.asserts=true -Dtests.file.encoding=UTF8
> 00:27:31[junit4] ERROR 0.04s J3 | TestOfflineSorter.testThreadSafety <<<
> 00:27:31[junit4]> Throwable #1: java.lang.AssertionError
> 00:27:31[junit4]> at
> org.apache.lucene.util.TestOfflineSorter.testThreadSafety(TestOfflineSorter.java:283)
> 00:27:31[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 00:27:31[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 00:27:31[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 00:27:31[junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 00:27:31[junit4]
[jira] [Created] (LUCENE-9467) TestOfflineSorter fails with combination of NRTCaching and ByteBuffersDirectory
Simon Willnauer created LUCENE-9467:
---
Summary: TestOfflineSorter fails with combination of NRTCaching
and ByteBuffersDirectory
Key: LUCENE-9467
URL: https://issues.apache.org/jira/browse/LUCENE-9467
Project: Lucene - Core
Issue Type: Bug
Affects Versions: 8.7, 8.6.1
Reporter: Simon Willnauer
TestOfflineSorter#testThreadSafety fails on 8.x since NRTCachingDirectory uses
the delegates openInput method to slowly check if the file exists. This is not
present anymore on main line. The problem is that ByteBuffersDirectory fires an
AccessDeniedException when the file is not fully written yet.
{noformat}
00:27:31[junit4] Suite: org.apache.lucene.util.TestOfflineSorter
00:27:31[junit4] 2> KMN 14, 2020 6:27:30 AM
com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
uncaughtException
00:27:31[junit4] 2> WARNING: Uncaught exception in thread:
Thread[Thread-7318,5,TGRP-TestOfflineSorter]
00:27:31[junit4] 2> java.lang.RuntimeException:
java.nio.file.AccessDeniedException: Can't open a file still open for writing:
unsorted_tmp_6.tmp
00:27:31[junit4] 2> at
__randomizedtesting.SeedInfo.seed([B1129484013118F7]:0)
00:27:31[junit4] 2> at
org.apache.lucene.util.TestOfflineSorter$2.run(TestOfflineSorter.java:272)
00:27:31[junit4] 2> Caused by: java.nio.file.AccessDeniedException: Can't
open a file still open for writing: unsorted_tmp_6.tmp
00:27:31[junit4] 2> at
org.apache.lucene.store.ByteBuffersDirectory$FileEntry.openInput(ByteBuffersDirectory.java:251)
00:27:31[junit4] 2> at
org.apache.lucene.store.ByteBuffersDirectory.openInput(ByteBuffersDirectory.java:218)
00:27:31[junit4] 2> at
org.apache.lucene.store.FileSwitchDirectory.openInput(FileSwitchDirectory.java:233)
00:27:31[junit4] 2> at
org.apache.lucene.store.NRTCachingDirectory.slowFileExists(NRTCachingDirectory.java:280)
00:27:31[junit4] 2> at
org.apache.lucene.store.NRTCachingDirectory.createTempOutput(NRTCachingDirectory.java:254)
00:27:31[junit4] 2> at
org.apache.lucene.store.MockDirectoryWrapper.createTempOutput(MockDirectoryWrapper.java:707)
00:27:31[junit4] 2> at
org.apache.lucene.util.TestOfflineSorter.checkSort(TestOfflineSorter.java:184)
00:27:31[junit4] 2> at
org.apache.lucene.util.TestOfflineSorter.access$100(TestOfflineSorter.java:48)
00:27:31[junit4] 2> at
org.apache.lucene.util.TestOfflineSorter$2.run(TestOfflineSorter.java:268)
00:27:31[junit4] 2>
00:27:31[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestOfflineSorter -Dtests.method=testThreadSafety
-Dtests.seed=B1129484013118F7 -Dtests.nightly=true -Dtests.slow=true
-Dtests.badapples=true -Dtests.locale=cgg-UG -Dtests.timezone=Asia/Dacca
-Dtests.asserts=true -Dtests.file.encoding=UTF8
00:27:31[junit4] ERROR 0.04s J3 | TestOfflineSorter.testThreadSafety <<<
00:27:31[junit4]> Throwable #1: java.lang.AssertionError
00:27:31[junit4]> at
org.apache.lucene.util.TestOfflineSorter.testThreadSafety(TestOfflineSorter.java:283)
00:27:31[junit4]> at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
00:27:31[junit4]> at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
00:27:31[junit4]> at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
00:27:31[junit4]> at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
00:27:31[junit4]> at
java.base/java.lang.Thread.run(Thread.java:834)Throwable #2:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught
exception in thread: Thread[id=7993, name=Thread-7318, state=RUNNABLE,
group=TGRP-TestOfflineSorter]
00:27:31[junit4]> Caused by: java.lang.RuntimeException:
java.nio.file.AccessDeniedException: Can't open a file still open for writing:
unsorted_tmp_6.tmp
00:27:31[junit4]> at
__randomizedtesting.SeedInfo.seed([B1129484013118F7]:0)
00:27:31[junit4]> at
org.apache.lucene.util.TestOfflineSorter$2.run(TestOfflineSorter.java:272)
00:27:31[junit4]> Caused by: java.nio.file.AccessDeniedException: Can't
open a file still open for writing: unsorted_tmp_6.tmp
00:27:31[junit4]> at
org.apache.lucene.store.ByteBuffersDirectory$FileEntry.openInput(ByteBuffersDirectory.java:251)
00:27:31[junit4]> at
org.apache.lucene.store.ByteBuffersDirectory.openInput(ByteBuffersDirectory.java:218)
00:27:31[junit4]> at
org.apache.lucene.store.FileSwitchDirectory.openInput(FileSwitchDirectory.java:233)
00:27:31[junit4]> at
[jira] [Resolved] (LUCENE-9467) TestOfflineSorter fails with combination of NRTCaching and ByteBuffersDirectory
[
https://issues.apache.org/jira/browse/LUCENE-9467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer resolved LUCENE-9467.
-
Fix Version/s: 8.7
Resolution: Fixed
> TestOfflineSorter fails with combination of NRTCaching and
> ByteBuffersDirectory
> ---
>
> Key: LUCENE-9467
> URL: https://issues.apache.org/jira/browse/LUCENE-9467
> Project: Lucene - Core
> Issue Type: Bug
>Affects Versions: 8.7, 8.6.1
>Reporter: Simon Willnauer
>Priority: Major
> Fix For: 8.7
>
>
> TestOfflineSorter#testThreadSafety fails on 8.x since NRTCachingDirectory
> uses the delegates openInput method to slowly check if the file exists. This
> is not present anymore on main line. The problem is that ByteBuffersDirectory
> fires an AccessDeniedException when the file is not fully written yet.
> {noformat}
> 00:27:31[junit4] Suite: org.apache.lucene.util.TestOfflineSorter
> 00:27:31[junit4] 2> KMN 14, 2020 6:27:30 AM
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
> uncaughtException
> 00:27:31[junit4] 2> WARNING: Uncaught exception in thread:
> Thread[Thread-7318,5,TGRP-TestOfflineSorter]
> 00:27:31[junit4] 2> java.lang.RuntimeException:
> java.nio.file.AccessDeniedException: Can't open a file still open for
> writing: unsorted_tmp_6.tmp
> 00:27:31[junit4] 2> at
> __randomizedtesting.SeedInfo.seed([B1129484013118F7]:0)
> 00:27:31[junit4] 2> at
> org.apache.lucene.util.TestOfflineSorter$2.run(TestOfflineSorter.java:272)
> 00:27:31[junit4] 2> Caused by: java.nio.file.AccessDeniedException:
> Can't open a file still open for writing: unsorted_tmp_6.tmp
> 00:27:31[junit4] 2> at
> org.apache.lucene.store.ByteBuffersDirectory$FileEntry.openInput(ByteBuffersDirectory.java:251)
> 00:27:31[junit4] 2> at
> org.apache.lucene.store.ByteBuffersDirectory.openInput(ByteBuffersDirectory.java:218)
> 00:27:31[junit4] 2> at
> org.apache.lucene.store.FileSwitchDirectory.openInput(FileSwitchDirectory.java:233)
> 00:27:31[junit4] 2> at
> org.apache.lucene.store.NRTCachingDirectory.slowFileExists(NRTCachingDirectory.java:280)
> 00:27:31[junit4] 2> at
> org.apache.lucene.store.NRTCachingDirectory.createTempOutput(NRTCachingDirectory.java:254)
> 00:27:31[junit4] 2> at
> org.apache.lucene.store.MockDirectoryWrapper.createTempOutput(MockDirectoryWrapper.java:707)
> 00:27:31[junit4] 2> at
> org.apache.lucene.util.TestOfflineSorter.checkSort(TestOfflineSorter.java:184)
> 00:27:31[junit4] 2> at
> org.apache.lucene.util.TestOfflineSorter.access$100(TestOfflineSorter.java:48)
> 00:27:31[junit4] 2> at
> org.apache.lucene.util.TestOfflineSorter$2.run(TestOfflineSorter.java:268)
> 00:27:31[junit4] 2>
> 00:27:31[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestOfflineSorter -Dtests.method=testThreadSafety
> -Dtests.seed=B1129484013118F7 -Dtests.nightly=true -Dtests.slow=true
> -Dtests.badapples=true -Dtests.locale=cgg-UG -Dtests.timezone=Asia/Dacca
> -Dtests.asserts=true -Dtests.file.encoding=UTF8
> 00:27:31[junit4] ERROR 0.04s J3 | TestOfflineSorter.testThreadSafety <<<
> 00:27:31[junit4]> Throwable #1: java.lang.AssertionError
> 00:27:31[junit4]> at
> org.apache.lucene.util.TestOfflineSorter.testThreadSafety(TestOfflineSorter.java:283)
> 00:27:31[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 00:27:31[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 00:27:31[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 00:27:31[junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 00:27:31[junit4]> at
> java.base/java.lang.Thread.run(Thread.java:834)Throwable #2:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=7993, name=Thread-7318,
> state=RUNNABLE, group=TGRP-TestOfflineSorter]
> 00:27:31[junit4]> Caused by: java.lang.RuntimeException:
> java.nio.file.AccessDeniedException: Can't open a file still open for
> writing: unsorted_tmp_6.tmp
> 00:27:31[junit4]> at
> __randomizedtesting.SeedInfo.seed([B1129484013118F7]:0)
> 00:27:31[junit4]> at
> org.apache.lucene.util.TestOfflineSorter$2.run(TestOfflineSorter.java:272)
> 00:27:31[junit4]> Caused by: java.nio.file.AccessDeniedException:
> Can't open a file still open for writing: unsorted_tmp_6.tmp
> 00:27:31[junit4]> at
>
[jira] [Commented] (LUCENE-9467) TestOfflineSorter fails with combination of NRTCaching and ByteBuffersDirectory
[
https://issues.apache.org/jira/browse/LUCENE-9467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17179096#comment-17179096
]
Simon Willnauer commented on LUCENE-9467:
-
{quote}ByteBuffersDirectory actually has a dedicated fileExists method but
fileLength should work too, I guess.{quote}
correct but it's not part of the interface in 8.x unfortunately. I will go
ahead and push the file length fix
> TestOfflineSorter fails with combination of NRTCaching and
> ByteBuffersDirectory
> ---
>
> Key: LUCENE-9467
> URL: https://issues.apache.org/jira/browse/LUCENE-9467
> Project: Lucene - Core
> Issue Type: Bug
>Affects Versions: 8.7, 8.6.1
>Reporter: Simon Willnauer
>Priority: Major
>
> TestOfflineSorter#testThreadSafety fails on 8.x since NRTCachingDirectory
> uses the delegates openInput method to slowly check if the file exists. This
> is not present anymore on main line. The problem is that ByteBuffersDirectory
> fires an AccessDeniedException when the file is not fully written yet.
> {noformat}
> 00:27:31[junit4] Suite: org.apache.lucene.util.TestOfflineSorter
> 00:27:31[junit4] 2> KMN 14, 2020 6:27:30 AM
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
> uncaughtException
> 00:27:31[junit4] 2> WARNING: Uncaught exception in thread:
> Thread[Thread-7318,5,TGRP-TestOfflineSorter]
> 00:27:31[junit4] 2> java.lang.RuntimeException:
> java.nio.file.AccessDeniedException: Can't open a file still open for
> writing: unsorted_tmp_6.tmp
> 00:27:31[junit4] 2> at
> __randomizedtesting.SeedInfo.seed([B1129484013118F7]:0)
> 00:27:31[junit4] 2> at
> org.apache.lucene.util.TestOfflineSorter$2.run(TestOfflineSorter.java:272)
> 00:27:31[junit4] 2> Caused by: java.nio.file.AccessDeniedException:
> Can't open a file still open for writing: unsorted_tmp_6.tmp
> 00:27:31[junit4] 2> at
> org.apache.lucene.store.ByteBuffersDirectory$FileEntry.openInput(ByteBuffersDirectory.java:251)
> 00:27:31[junit4] 2> at
> org.apache.lucene.store.ByteBuffersDirectory.openInput(ByteBuffersDirectory.java:218)
> 00:27:31[junit4] 2> at
> org.apache.lucene.store.FileSwitchDirectory.openInput(FileSwitchDirectory.java:233)
> 00:27:31[junit4] 2> at
> org.apache.lucene.store.NRTCachingDirectory.slowFileExists(NRTCachingDirectory.java:280)
> 00:27:31[junit4] 2> at
> org.apache.lucene.store.NRTCachingDirectory.createTempOutput(NRTCachingDirectory.java:254)
> 00:27:31[junit4] 2> at
> org.apache.lucene.store.MockDirectoryWrapper.createTempOutput(MockDirectoryWrapper.java:707)
> 00:27:31[junit4] 2> at
> org.apache.lucene.util.TestOfflineSorter.checkSort(TestOfflineSorter.java:184)
> 00:27:31[junit4] 2> at
> org.apache.lucene.util.TestOfflineSorter.access$100(TestOfflineSorter.java:48)
> 00:27:31[junit4] 2> at
> org.apache.lucene.util.TestOfflineSorter$2.run(TestOfflineSorter.java:268)
> 00:27:31[junit4] 2>
> 00:27:31[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestOfflineSorter -Dtests.method=testThreadSafety
> -Dtests.seed=B1129484013118F7 -Dtests.nightly=true -Dtests.slow=true
> -Dtests.badapples=true -Dtests.locale=cgg-UG -Dtests.timezone=Asia/Dacca
> -Dtests.asserts=true -Dtests.file.encoding=UTF8
> 00:27:31[junit4] ERROR 0.04s J3 | TestOfflineSorter.testThreadSafety <<<
> 00:27:31[junit4]> Throwable #1: java.lang.AssertionError
> 00:27:31[junit4]> at
> org.apache.lucene.util.TestOfflineSorter.testThreadSafety(TestOfflineSorter.java:283)
> 00:27:31[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 00:27:31[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 00:27:31[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 00:27:31[junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 00:27:31[junit4]> at
> java.base/java.lang.Thread.run(Thread.java:834)Throwable #2:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=7993, name=Thread-7318,
> state=RUNNABLE, group=TGRP-TestOfflineSorter]
> 00:27:31[junit4]> Caused by: java.lang.RuntimeException:
> java.nio.file.AccessDeniedException: Can't open a file still open for
> writing: unsorted_tmp_6.tmp
> 00:27:31[junit4]> at
> __randomizedtesting.SeedInfo.seed([B1129484013118F7]:0)
> 00:27:31[junit4]> at
> org.apache.lucene.util.TestOfflineSorter$2.run(TestOfflineSorter.java:272)
>
[jira] [Created] (LUCENE-9539) Improve memory footprint of SortingCodecReader
Simon Willnauer created LUCENE-9539: --- Summary: Improve memory footprint of SortingCodecReader Key: LUCENE-9539 URL: https://issues.apache.org/jira/browse/LUCENE-9539 Project: Lucene - Core Issue Type: Improvement Reporter: Simon Willnauer SortingCodecReader is a very memory heavy since it needs to re-sort and load large parts of the index into memory. We can try to make it more efficient by using more compact internal data-structures, remove the caches it uses provided we define it's usage as a merge only reader wrapper. Ultimately we need to find a way to allow the reader or some other structure to minimize its heap memory. One way is to slice existing readers and merge them in multiple steps. There will be multiple steps towards a more useable version of this class. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (LUCENE-9534) DocumentsWriterFlushControl expects IndexingChain#ramBytesUsed to be thread-safe but it's not
[
https://issues.apache.org/jira/browse/LUCENE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer reassigned LUCENE-9534:
---
Assignee: Simon Willnauer
> DocumentsWriterFlushControl expects IndexingChain#ramBytesUsed to be
> thread-safe but it's not
> -
>
> Key: LUCENE-9534
> URL: https://issues.apache.org/jira/browse/LUCENE-9534
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Adrien Grand
>Assignee: Simon Willnauer
>Priority: Minor
>
> I found this after seeing the following test failure:
> {noformat}
> 17:43:57[junit4] 1> java.lang.NullPointerException
> 17:43:57[junit4] 1> at
> org.apache.lucene.store.ByteBuffersDataOutput.lambda$ramBytesUsed$2(ByteBuffersDataOutput.java:405)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.ReferencePipeline$5$1.accept(ReferencePipeline.java:229)
> 17:43:57[junit4] 1> at
> java.base/java.util.ArrayDeque$DeqSpliterator.forEachRemaining(ArrayDeque.java:849)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.LongPipeline.reduce(LongPipeline.java:474)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.LongPipeline.sum(LongPipeline.java:432)
> 17:43:57[junit4] 1> at
> org.apache.lucene.store.ByteBuffersDataOutput.ramBytesUsed(ByteBuffersDataOutput.java:405)
> 17:43:57[junit4] 1> at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.ramBytesUsed(CompressingStoredFieldsWriter.java:699)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.DefaultIndexingChain.ramBytesUsed(DefaultIndexingChain.java:801)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.DocumentsWriterPerThread.ramBytesUsed(DocumentsWriterPerThread.java:521)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.DocumentsWriterFlushControl.findLargestNonPendingWriter(DocumentsWriterFlushControl.java:672)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.FlushPolicy.findLargestNonPendingWriter(FlushPolicy.java:104)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.FlushByRamOrCountsPolicy.markLargestWriterPending(FlushByRamOrCountsPolicy.java:93)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.FlushByRamOrCountsPolicy.onInsert(FlushByRamOrCountsPolicy.java:82)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.FlushPolicy.onUpdate(FlushPolicy.java:71)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.DocumentsWriterFlushControl.doAfterDocument(DocumentsWriterFlushControl.java:178)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:430)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1471)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1757)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.TestIndexWriterExceptions$IndexerThread.run(TestIndexWriterExceptions.java:216)
> 17:43:57[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestIndexWriterExceptions
> -Dtests.method=testRandomExceptionsThreads -Dtests.seed=3C0F2600B3EEA596
> -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=brx-IN
> -Dtests.timezone=Africa/Mbabane -Dtests.asserts=true
> -Dtests.file.encoding=US-ASCII
> 17:43:57[junit4] FAILURE 0.52s J3 |
> TestIndexWriterExceptions.testRandomExceptionsThreads <<<
> 17:43:57[junit4]> Throwable #1: java.lang.AssertionError: thread
> Indexer 3: hit unexpected failure
> 17:43:57[junit4]> at
> __randomizedtesting.SeedInfo.seed([3C0F2600B3EEA596:83E16C1DBE01C597]:0)
> 17:43:57[junit4]> at
> org.apache.lucene.index.TestIndexWriterExceptions.testRandomExceptionsThreads(TestIndexWriterExceptions.java:316)
> 17:43:57[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 17:43:57[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 17:43:57[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 17:43:57[junit4]>
[jira] [Commented] (LUCENE-9534) DocumentsWriterFlushControl expects IndexingChain#ramBytesUsed to be thread-safe but it's not
[
https://issues.apache.org/jira/browse/LUCENE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198409#comment-17198409
]
Simon Willnauer commented on LUCENE-9534:
-
I am working on a fix. Thanks for opening this.
> DocumentsWriterFlushControl expects IndexingChain#ramBytesUsed to be
> thread-safe but it's not
> -
>
> Key: LUCENE-9534
> URL: https://issues.apache.org/jira/browse/LUCENE-9534
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Adrien Grand
>Assignee: Simon Willnauer
>Priority: Minor
>
> I found this after seeing the following test failure:
> {noformat}
> 17:43:57[junit4] 1> java.lang.NullPointerException
> 17:43:57[junit4] 1> at
> org.apache.lucene.store.ByteBuffersDataOutput.lambda$ramBytesUsed$2(ByteBuffersDataOutput.java:405)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.ReferencePipeline$5$1.accept(ReferencePipeline.java:229)
> 17:43:57[junit4] 1> at
> java.base/java.util.ArrayDeque$DeqSpliterator.forEachRemaining(ArrayDeque.java:849)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.LongPipeline.reduce(LongPipeline.java:474)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.LongPipeline.sum(LongPipeline.java:432)
> 17:43:57[junit4] 1> at
> org.apache.lucene.store.ByteBuffersDataOutput.ramBytesUsed(ByteBuffersDataOutput.java:405)
> 17:43:57[junit4] 1> at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.ramBytesUsed(CompressingStoredFieldsWriter.java:699)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.DefaultIndexingChain.ramBytesUsed(DefaultIndexingChain.java:801)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.DocumentsWriterPerThread.ramBytesUsed(DocumentsWriterPerThread.java:521)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.DocumentsWriterFlushControl.findLargestNonPendingWriter(DocumentsWriterFlushControl.java:672)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.FlushPolicy.findLargestNonPendingWriter(FlushPolicy.java:104)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.FlushByRamOrCountsPolicy.markLargestWriterPending(FlushByRamOrCountsPolicy.java:93)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.FlushByRamOrCountsPolicy.onInsert(FlushByRamOrCountsPolicy.java:82)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.FlushPolicy.onUpdate(FlushPolicy.java:71)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.DocumentsWriterFlushControl.doAfterDocument(DocumentsWriterFlushControl.java:178)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:430)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1471)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1757)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.TestIndexWriterExceptions$IndexerThread.run(TestIndexWriterExceptions.java:216)
> 17:43:57[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestIndexWriterExceptions
> -Dtests.method=testRandomExceptionsThreads -Dtests.seed=3C0F2600B3EEA596
> -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=brx-IN
> -Dtests.timezone=Africa/Mbabane -Dtests.asserts=true
> -Dtests.file.encoding=US-ASCII
> 17:43:57[junit4] FAILURE 0.52s J3 |
> TestIndexWriterExceptions.testRandomExceptionsThreads <<<
> 17:43:57[junit4]> Throwable #1: java.lang.AssertionError: thread
> Indexer 3: hit unexpected failure
> 17:43:57[junit4]> at
> __randomizedtesting.SeedInfo.seed([3C0F2600B3EEA596:83E16C1DBE01C597]:0)
> 17:43:57[junit4]> at
> org.apache.lucene.index.TestIndexWriterExceptions.testRandomExceptionsThreads(TestIndexWriterExceptions.java:316)
> 17:43:57[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 17:43:57[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 17:43:57[junit4]> at
>
[jira] [Resolved] (LUCENE-9534) DocumentsWriterFlushControl expects IndexingChain#ramBytesUsed to be thread-safe but it's not
[
https://issues.apache.org/jira/browse/LUCENE-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer resolved LUCENE-9534.
-
Resolution: Fixed
> DocumentsWriterFlushControl expects IndexingChain#ramBytesUsed to be
> thread-safe but it's not
> -
>
> Key: LUCENE-9534
> URL: https://issues.apache.org/jira/browse/LUCENE-9534
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Adrien Grand
>Assignee: Simon Willnauer
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I found this after seeing the following test failure:
> {noformat}
> 17:43:57[junit4] 1> java.lang.NullPointerException
> 17:43:57[junit4] 1> at
> org.apache.lucene.store.ByteBuffersDataOutput.lambda$ramBytesUsed$2(ByteBuffersDataOutput.java:405)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.ReferencePipeline$5$1.accept(ReferencePipeline.java:229)
> 17:43:57[junit4] 1> at
> java.base/java.util.ArrayDeque$DeqSpliterator.forEachRemaining(ArrayDeque.java:849)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.LongPipeline.reduce(LongPipeline.java:474)
> 17:43:57[junit4] 1> at
> java.base/java.util.stream.LongPipeline.sum(LongPipeline.java:432)
> 17:43:57[junit4] 1> at
> org.apache.lucene.store.ByteBuffersDataOutput.ramBytesUsed(ByteBuffersDataOutput.java:405)
> 17:43:57[junit4] 1> at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.ramBytesUsed(CompressingStoredFieldsWriter.java:699)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.DefaultIndexingChain.ramBytesUsed(DefaultIndexingChain.java:801)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.DocumentsWriterPerThread.ramBytesUsed(DocumentsWriterPerThread.java:521)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.DocumentsWriterFlushControl.findLargestNonPendingWriter(DocumentsWriterFlushControl.java:672)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.FlushPolicy.findLargestNonPendingWriter(FlushPolicy.java:104)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.FlushByRamOrCountsPolicy.markLargestWriterPending(FlushByRamOrCountsPolicy.java:93)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.FlushByRamOrCountsPolicy.onInsert(FlushByRamOrCountsPolicy.java:82)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.FlushPolicy.onUpdate(FlushPolicy.java:71)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.DocumentsWriterFlushControl.doAfterDocument(DocumentsWriterFlushControl.java:178)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:430)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1471)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1757)
> 17:43:57[junit4] 1> at
> org.apache.lucene.index.TestIndexWriterExceptions$IndexerThread.run(TestIndexWriterExceptions.java:216)
> 17:43:57[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestIndexWriterExceptions
> -Dtests.method=testRandomExceptionsThreads -Dtests.seed=3C0F2600B3EEA596
> -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=brx-IN
> -Dtests.timezone=Africa/Mbabane -Dtests.asserts=true
> -Dtests.file.encoding=US-ASCII
> 17:43:57[junit4] FAILURE 0.52s J3 |
> TestIndexWriterExceptions.testRandomExceptionsThreads <<<
> 17:43:57[junit4]> Throwable #1: java.lang.AssertionError: thread
> Indexer 3: hit unexpected failure
> 17:43:57[junit4]> at
> __randomizedtesting.SeedInfo.seed([3C0F2600B3EEA596:83E16C1DBE01C597]:0)
> 17:43:57[junit4]> at
> org.apache.lucene.index.TestIndexWriterExceptions.testRandomExceptionsThreads(TestIndexWriterExceptions.java:316)
> 17:43:57[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 17:43:57[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 17:43:57[junit4]> at
>
[jira] [Commented] (LUCENE-8776) Start offset going backwards has a legitimate purpose
[
https://issues.apache.org/jira/browse/LUCENE-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176184#comment-17176184
]
Simon Willnauer commented on LUCENE-8776:
-
I think preventing negative offsets is more than just allowing people to pay
the price of 5 bytes per negative offset. There are many more things than
compression based on this behavior. Just a few that come to my mind right away:
* Correctness verification of indices, if we see a negative offset the index
must be broken.
* Catch broken analyzers early
* have clear bounds of what an offset can be and in what direction it can
grow. This helps tons if you want to implement future features that can rely on
this behavior.
{quote}
So if we just add a simple mechanism to instruct Lucene which DocConsumer to
use, then all could be happy and not have to resort to dirty hacks or forks.
The most efficient impl will be the default, yet will allow us us - dirty
bastards - shoot ourselves in foot if we so desire. SOLR as well as
ElasticSearch devs might not mind having the option in the future - it can come
in handy. Wouldn't that be wonderful? Well, wonderful certainly not, just
useful... could I do it?
{quote}
if you feel like you wanna implement this, go ahead. I am sure there will be
more issues like Check Index will not work anymore etc. There might be future
features you will break with doing this. But again it's all open source, nobody
forces you to upgrade or use the code we package directly. You can download the
source and modify it that' is all up to you.
We as a community need to make decisions that sometimes don't work for
everybody. We have a great responsibility with a project like Lucene being used
in an unbelievable wide range of applications and that sometimes means to add
restrictions. We don't take this easily, it's almost always a hard decisions.
Having offsets going forward only will be a win for a large majority and that's
why we keep on having this restrictions. I am totally sorry for you struggle.
Talking about extending DocConsumer, I am torn it should be a extension point.
I know that there is an implementation that uses it out there but if I could
pick I'd remove this extension since it's, as you say, way to hard to get it
right.
> Start offset going backwards has a legitimate purpose
> -
>
> Key: LUCENE-8776
> URL: https://issues.apache.org/jira/browse/LUCENE-8776
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
>Affects Versions: 7.6
>Reporter: Ram Venkat
>Priority: Major
>
> Here is the use case where startOffset can go backwards:
> Say there is a line "Organic light-emitting-diode glows", and I want to run
> span queries and highlight them properly.
> During index time, light-emitting-diode is split into three words, which
> allows me to search for 'light', 'emitting' and 'diode' individually. The
> three words occupy adjacent positions in the index, as 'light' adjacent to
> 'emitting' and 'light' at a distance of two words from 'diode' need to match
> this word. So, the order of words after splitting are: Organic, light,
> emitting, diode, glows.
> But, I also want to search for 'organic' being adjacent to
> 'light-emitting-diode' or 'light-emitting-diode' being adjacent to 'glows'.
> The way I solved this was to also generate 'light-emitting-diode' at two
> positions: (a) In the same position as 'light' and (b) in the same position
> as 'glows', like below:
> ||organic||light||emitting||diode||glows||
> | |light-emitting-diode| |light-emitting-diode| |
> |0|1|2|3|4|
> The positions of the two 'light-emitting-diode' are 1 and 3, but the offsets
> are obviously the same. This works beautifully in Lucene 5.x in both
> searching and highlighting with span queries.
> But when I try this in Lucene 7.6, it hits the condition "Offsets must not go
> backwards" at DefaultIndexingChain:818. This IllegalArgumentException is
> being thrown without any comments on why this check is needed. As I explained
> above, startOffset going backwards is perfectly valid, to deal with word
> splitting and span operations on these specialized use cases. On the other
> hand, it is not clear what value is added by this check and which highlighter
> code is affected by offsets going backwards. This same check is done at
> BaseTokenStreamTestCase:245.
> I see others talk about how this check found bugs in WordDelimiter etc. but
> it also prevents legitimate use cases. Can this check be removed?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (LUCENE-9405) IndexWriter incorrectly calls closeMergeReaders twice when the merged segment is 100% deleted
[
https://issues.apache.org/jira/browse/LUCENE-9405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer reassigned LUCENE-9405:
---
Assignee: Simon Willnauer
I will try to look at this
> IndexWriter incorrectly calls closeMergeReaders twice when the merged segment
> is 100% deleted
> -
>
> Key: LUCENE-9405
> URL: https://issues.apache.org/jira/browse/LUCENE-9405
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Reporter: Michael McCandless
>Assignee: Simon Willnauer
>Priority: Minor
>
> This is the first spinoff from a [controversial PR to add a new index-time
> feature to Lucene to merge small segments during
> commit|https://github.com/apache/lucene-solr/pull/1552]. This can
> substantially reduce the number of small index segments to search.
> See specifically [this discussion
> there|https://github.com/apache/lucene-solr/pull/1552#discussion_r440298695].
> {{IndexWriter}} seems to be missing a {{success = true}} inside
> {{mergeMiddle}} in the case where all segments being merged have 100%
> deletions and the segments will simply be dropped.
> In this case, in master today, I think we are incorrectly calling
> {{closeMergedReaders}} twice, first with {{suppressExceptions = false}} and
> second time with {{true}}.
> There is a [dedicated test case here showing the
> issue|https://github.com/apache/lucene-solr/commit/cab5ef5e6f2bdcda59fd669a298ec137af9d],
> but that test case relies on changes in the controversial feature (added
> {{MergePolicy.findFullFlushMerges}}). I think it should be possible to make
> another test case show the bug without that controversial feature, and I am
> unsure why our existing randomized tests have not uncovered this yet ...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9405) IndexWriter incorrectly calls closeMergeReaders twice when the merged segment is 100% deleted
[
https://issues.apache.org/jira/browse/LUCENE-9405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer resolved LUCENE-9405.
-
Fix Version/s: 8.6
master (9.0)
Resolution: Fixed
> IndexWriter incorrectly calls closeMergeReaders twice when the merged segment
> is 100% deleted
> -
>
> Key: LUCENE-9405
> URL: https://issues.apache.org/jira/browse/LUCENE-9405
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Reporter: Michael McCandless
>Assignee: Simon Willnauer
>Priority: Minor
> Fix For: master (9.0), 8.6
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> This is the first spinoff from a [controversial PR to add a new index-time
> feature to Lucene to merge small segments during
> commit|https://github.com/apache/lucene-solr/pull/1552]. This can
> substantially reduce the number of small index segments to search.
> See specifically [this discussion
> there|https://github.com/apache/lucene-solr/pull/1552#discussion_r440298695].
> {{IndexWriter}} seems to be missing a {{success = true}} inside
> {{mergeMiddle}} in the case where all segments being merged have 100%
> deletions and the segments will simply be dropped.
> In this case, in master today, I think we are incorrectly calling
> {{closeMergedReaders}} twice, first with {{suppressExceptions = false}} and
> second time with {{true}}.
> There is a [dedicated test case here showing the
> issue|https://github.com/apache/lucene-solr/commit/cab5ef5e6f2bdcda59fd669a298ec137af9d],
> but that test case relies on changes in the controversial feature (added
> {{MergePolicy.findFullFlushMerges}}). I think it should be possible to make
> another test case show the bug without that controversial feature, and I am
> unsure why our existing randomized tests have not uncovered this yet ...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9408) OneMerge#mergeFinish is called multiple times
Simon Willnauer created LUCENE-9408:
---
Summary: OneMerge#mergeFinish is called multiple times
Key: LUCENE-9408
URL: https://issues.apache.org/jira/browse/LUCENE-9408
Project: Lucene - Core
Issue Type: Bug
Reporter: Simon Willnauer
After enforcing calling this method only once a random test caused
OneMerge#mergeFinished to be called multiple times.
{noformat}
21:06:59[junit4] Suite: org.apache.lucene.index.TestIndexFileDeleter
21:06:59[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestIndexFileDeleter -Dtests.method=testExcInDeleteFile
-Dtests.seed=BCFF67862FF6529B -Dtests.slow=true -Dtests.badapples=true
-Dtests.locale=haw-US -Dtests.timezone=Etc/Zulu -Dtests.asserts=true
-Dtests.file.encoding=ISO-8859-1
21:06:59[junit4] FAILURE 0.04s J0 |
TestIndexFileDeleter.testExcInDeleteFile <<<
21:06:59[junit4]> Throwable #1: java.lang.AssertionError
21:06:59[junit4]> at
__randomizedtesting.SeedInfo.seed([BCFF67862FF6529B:518F81E5ACB0444E]:0)
21:06:59[junit4]> at
org.apache.lucene.index.TestIndexFileDeleter.testExcInDeleteFile(TestIndexFileDeleter.java:525)
21:06:59[junit4]> at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
21:06:59[junit4]> at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
21:06:59[junit4]> at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
21:06:59[junit4]> at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
21:06:59[junit4]> at
java.base/java.lang.Thread.run(Thread.java:834)
{noformat}
mergeFinished should only be called once.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142389#comment-17142389
]
Simon Willnauer commented on LUCENE-8962:
-
This would also explain the failure I am looking into right now on elastic CI:
{noformat}
12:20:39[junit4] Suite: org.apache.lucene.index.TestIndexingSequenceNumbers
12:20:39[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestIndexingSequenceNumbers
-Dtests.method=testStressConcurrentCommit -Dtests.seed=FA92259FC8239E7
-Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true
-Dtests.locale=en-TC -Dtests.timezone=Pacific/Pago_Pago -Dtests.asserts=true
-Dtests.file.encoding=UTF8
12:20:39[junit4] FAILURE 85.1s J2 |
TestIndexingSequenceNumbers.testStressConcurrentCommit <<<
12:20:39[junit4]> Throwable #1: java.lang.AssertionError: expected:<1>
but was:<0>
12:20:39[junit4]> at
__randomizedtesting.SeedInfo.seed([FA92259FC8239E7:743DB8E67E421850]:0)
12:20:39[junit4]> at
org.apache.lucene.index.TestIndexingSequenceNumbers.testStressConcurrentCommit(TestIndexingSequenceNumbers.java:273)
12:20:39[junit4]> at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
12:20:39[junit4]> at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
12:20:39[junit4]> at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
12:20:39[junit4]> at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
12:20:39[junit4]> at
java.base/java.lang.Thread.run(Thread.java:834)
12:20:39[junit4] 2> NOTE: leaving temporary files on disk at:
/var/lib/jenkins/workspace/apache+lucene-solr+nightly+branch_8x/lucene/build/core/test/J2/temp/lucene.index.TestIndexingSequenceNumbers_FA92259FC8239E7-001
12:20:39[junit4] 2> NOTE: test params are: codec=Asserting(Lucene86):
{id=Lucene84}, docValues:{thread=DocValuesFormat(name=Asserting),
___soft_deletes=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=279,
maxMBSortInHeap=6.3565394726088424,
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@1c08a55f),
locale=en-TC, timezone=Pacific/Pago_Pago
12:20:39[junit4] 2> NOTE: Linux 4.18.0-193.6.3.el8_2.x86_64 amd64/Oracle
Corporation 11.0.2 (64-bit)/cpus=32,threads=1,free=339006432,total=400556032
12:20:39[junit4] 2> NOTE: All tests run in this JVM:
[TestMultiTermConstantScore, TestSparseFixedBitSet, TestSpanBoostQuery,
TestOfflineSorter, TestRadixSelector, TestBagOfPositions,
TestPhrasePrefixQuery, TestAxiomaticF2LOG, TestLucene50CompoundFormat,
TestLongBitSet, TestBinaryDocument, TestBooleanOr,
TestComplexExplanationsOfNonMatches, TestTransactions, TestTermQuery,
TestUTF32ToUTF8, TestWildcard, TestIndexWriterForceMerge, TestMinShouldMatch2,
TestCodecs, TestBytesStore, Test2BNumericDocValues, TestCharFilter,
TestXYMultiPolygonShapeQueries, TestFeatureDoubleValues, TestPackedInts,
TestGraphTokenizers, TestFilterCodecReader, TestSetOnce, TestLatLonPoint,
TestLongRange, TestQueryRescorer, TestNRTThreads, TestMergedIterator,
TestLucene86SegmentInfoFormat, TestPackedTokenAttributeImpl, TestBasicModelIne,
TestDocCount, TestLMDirichletSimilarity, TestAttributeSource,
TestPositiveScoresOnlyCollector, TestXYPointQueries, TestElevationComparator,
TestIndexWriterMergePolicy, TestUpgradeIndexMergePolicy,
TestIntRangeFieldQueries, TestIndexingSequenceNumbers]
12:20:39[junit4] Completed [369/567 (1!)] on J2 in 168.09s, 8 tests, 1
failure <<< FAILURES!
{noformat}
it's missing a document, pretty scary but explains what we see here.
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: 8.6
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch, failure_log.txt
>
> Time Spent: 19h 40m
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally
[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142388#comment-17142388
]
Simon Willnauer commented on LUCENE-8962:
-
[~msoko...@gmail.com] I think I have bad news for you. I think we forgot about
an important guarantee that we have from updateDocument that is now with this
feature enabled not holding up anymore. During a merge we carry over deletes
that are made concurrently to the source segments. This would mean that if we
continue including deleted that are carried over in a commit we are currently
making we don't carry over the corresponding added document which means that
there are documents not visible in the comment that should be visible. I think
this is a pretty significant problem and we should roll back this feature for
now. I am sorry that it took me a while to understand the implications here but
these IW logs are not the most intuitive thing to read and stare at.
We need to look into commitMergedDeletes if we can make it work or if we need
to do the merge again since we now need point in time semantics for merges too.
I am not sure if that's actually fixable, but lets look into it. [~mikemccand]
WDYT I think we should roll it back for now.
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: 8.6
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch, failure_log.txt
>
> Time Spent: 19h 40m
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer updated LUCENE-8962:
Attachment: test.diff
Status: Reopened (was: Reopened)
here is a test that shows the issue and fails every time.
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: 8.6
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch,
> failure_log.txt, test.diff
>
> Time Spent: 19h 40m
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142303#comment-17142303
]
Simon Willnauer commented on LUCENE-8962:
-
I opened a PR [here|https://github.com/apache/lucene-solr/pull/1601]
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: 8.6
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch, failure_log.txt
>
> Time Spent: 19.5h
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143881#comment-17143881
]
Simon Willnauer commented on LUCENE-8962:
-
I agree mike! I looked into it and it seems doable with some modifications. I
already pushed a prototype of what I think can work
[here|https://github.com/apache/lucene-solr/pull/1601/commits/de7677f1c98ffd38e9febce74f265fb44484c1d1]
[~sokolov] do you have a branch we can work on for this?
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: 8.6
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch,
> failure_log.txt, test.diff
>
> Time Spent: 19h 40m
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144366#comment-17144366
]
Simon Willnauer commented on LUCENE-8962:
-
[~sokolov] thanks, I updated the branch with the approach that I think can
work. I added quite some changes that you folks should catch up on. I think we
need to do more on the testing end with this feature being explicitly enabled.
I already pushed some tests for it. I would appreciate if you and [~mikemccand]
could spend some time looking that the work that I did. I am happy to open a PR
do do the reviews but maybe take a look at it first?
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: 8.6
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch,
> failure_log.txt, test.diff
>
> Time Spent: 20h 20m
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9408) OneMerge#mergeFinish is called multiple times
[
https://issues.apache.org/jira/browse/LUCENE-9408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer resolved LUCENE-9408.
-
Resolution: Fixed
> OneMerge#mergeFinish is called multiple times
> -
>
> Key: LUCENE-9408
> URL: https://issues.apache.org/jira/browse/LUCENE-9408
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Simon Willnauer
>Priority: Minor
> Time Spent: 50m
> Remaining Estimate: 0h
>
> After enforcing calling this method only once a random test caused
> OneMerge#mergeFinished to be called multiple times.
> {noformat}
> 21:06:59[junit4] Suite: org.apache.lucene.index.TestIndexFileDeleter
> 21:06:59[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestIndexFileDeleter -Dtests.method=testExcInDeleteFile
> -Dtests.seed=BCFF67862FF6529B -Dtests.slow=true -Dtests.badapples=true
> -Dtests.locale=haw-US -Dtests.timezone=Etc/Zulu -Dtests.asserts=true
> -Dtests.file.encoding=ISO-8859-1
> 21:06:59[junit4] FAILURE 0.04s J0 |
> TestIndexFileDeleter.testExcInDeleteFile <<<
> 21:06:59[junit4]> Throwable #1: java.lang.AssertionError
> 21:06:59[junit4]> at
> __randomizedtesting.SeedInfo.seed([BCFF67862FF6529B:518F81E5ACB0444E]:0)
> 21:06:59[junit4]> at
> org.apache.lucene.index.TestIndexFileDeleter.testExcInDeleteFile(TestIndexFileDeleter.java:525)
> 21:06:59[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 21:06:59[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 21:06:59[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 21:06:59[junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 21:06:59[junit4]> at
> java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
> mergeFinished should only be called once.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9408) OneMerge#mergeFinish is called multiple times
[
https://issues.apache.org/jira/browse/LUCENE-9408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer updated LUCENE-9408:
Fix Version/s: 8.6
master (9.0)
> OneMerge#mergeFinish is called multiple times
> -
>
> Key: LUCENE-9408
> URL: https://issues.apache.org/jira/browse/LUCENE-9408
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Simon Willnauer
>Priority: Minor
> Fix For: master (9.0), 8.6
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> After enforcing calling this method only once a random test caused
> OneMerge#mergeFinished to be called multiple times.
> {noformat}
> 21:06:59[junit4] Suite: org.apache.lucene.index.TestIndexFileDeleter
> 21:06:59[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestIndexFileDeleter -Dtests.method=testExcInDeleteFile
> -Dtests.seed=BCFF67862FF6529B -Dtests.slow=true -Dtests.badapples=true
> -Dtests.locale=haw-US -Dtests.timezone=Etc/Zulu -Dtests.asserts=true
> -Dtests.file.encoding=ISO-8859-1
> 21:06:59[junit4] FAILURE 0.04s J0 |
> TestIndexFileDeleter.testExcInDeleteFile <<<
> 21:06:59[junit4]> Throwable #1: java.lang.AssertionError
> 21:06:59[junit4]> at
> __randomizedtesting.SeedInfo.seed([BCFF67862FF6529B:518F81E5ACB0444E]:0)
> 21:06:59[junit4]> at
> org.apache.lucene.index.TestIndexFileDeleter.testExcInDeleteFile(TestIndexFileDeleter.java:525)
> 21:06:59[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 21:06:59[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 21:06:59[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 21:06:59[junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 21:06:59[junit4]> at
> java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
> mergeFinished should only be called once.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146286#comment-17146286
]
Simon Willnauer commented on LUCENE-8962:
-
[~mikemccand] I pushed fixes for both failures
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: 8.6
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch,
> failure_log.txt, test.diff
>
> Time Spent: 20h 20m
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146635#comment-17146635
]
Simon Willnauer commented on LUCENE-8962:
-
I did push a fix. it's a test bug, this test relies on 2 segments that got
merged away by this feature :)
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: 8.6
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch,
> failure_log.txt, test.diff
>
> Time Spent: 21h 50m
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9419) TestLongBitSet.testHugeCapacity sometimes fails on nightly
[
https://issues.apache.org/jira/browse/LUCENE-9419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer updated LUCENE-9419:
Environment: (was: I see this once in a while on our CI with
-Dtests.nightly=true
{noformat}
00:05:30[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestLongBitSet -Dtests.method=testHugeCapacity
-Dtests.seed=FB56CEB40315E44E -Dtests.nightly=true -Dtests.slow=true
-Dtests.badapples=true -Dtests.locale=lu -Dtests.timezone=Africa/Casablanca
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
00:05:30[junit4] ERROR 0.08s J1 | TestLongBitSet.testHugeCapacity <<<
00:05:30[junit4]> Throwable #1: java.lang.OutOfMemoryError: Java heap
space
00:05:30[junit4]> at
__randomizedtesting.SeedInfo.seed([FB56CEB40315E44E:8116844FEAE5C24A]:0)
00:05:30[junit4]> at
org.apache.lucene.util.ArrayUtil.growExact(ArrayUtil.java:323)
00:05:30[junit4]> at
org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:332)
00:05:30[junit4]> at
org.apache.lucene.util.LongBitSet.ensureCapacity(LongBitSet.java:54)
00:05:30[junit4]> at
org.apache.lucene.util.TestLongBitSet.testHugeCapacity(TestLongBitSet.java:343)
00:05:30[junit4]> at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
00:05:30[junit4]> at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
00:05:30[junit4]> at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
00:05:30[junit4]> at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
00:05:30[junit4] 2> NOTE: test params are: codec=Asserting(Lucene86): {},
docValues:{}, maxPointsInLeafNode=246, maxMBSortInHeap=6.097958740443044,
sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=lu,
timezone=Africa/Casablanca
00:05:30[junit4] 2> NOTE: Linux 4.12.14-122.26-default amd64/Oracle
Corporation 11.0.2 (64-bit)/cpus=32,threads=1,free=346124224,total=362807296
00:05:30[junit4] 2> NOTE: All tests run in this JVM: [TestSynonymQuery,
TestCachingCollector, TestLucene86PointsFormat, TestSizeBoundedForceMerge,
TestSimpleWKTShapeParsing, TestMergeRateLimiter, TestAxiomaticSimilarity,
TestReaderClosed, TestSpanExplanations, TestSingleInstanceLockFactory,
TestIndexSorting, TestIndexInput, TestBasicModelIn, TestPoint2D,
TestAtomicUpdate, TestOneMergeWrappingMergePolicy,
TestTragicIndexWriterDeadlock, TestRollingUpdates, TestCodecHoldsOpenFiles,
TestLucene50StoredFieldsFormatHighCompression, TestFieldsReader,
TestAxiomaticF2EXP, TestPointValues, TestIndependenceStandardized,
TestTwoPhaseCommitTool, TestLongBitSet]
00:05:30[junit4] Completed [87/566 (1!)] on J1 in 3.30s, 11 tests, 1 error
<<< FAILURES!
{noformat}
my suspicion is that some other test is holding on to memory here. I am trying
to record all failures here maybe we find some tests in common. The test itself
needs about 300MB to run so that should be ok with a 512M JVM
)
> TestLongBitSet.testHugeCapacity sometimes fails on nightly
> ---
>
> Key: LUCENE-9419
> URL: https://issues.apache.org/jira/browse/LUCENE-9419
> Project: Lucene - Core
> Issue Type: Test
>Reporter: Simon Willnauer
>Priority: Minor
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9419) TestLongBitSet.testHugeCapacity sometimes fails on nightly
[
https://issues.apache.org/jira/browse/LUCENE-9419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer updated LUCENE-9419:
Description:
I see this once in a while on our CI with -Dtests.nightly=true
{noformat}
00:05:30[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestLongBitSet -Dtests.method=testHugeCapacity
-Dtests.seed=FB56CEB40315E44E -Dtests.nightly=true -Dtests.slow=true
-Dtests.badapples=true -Dtests.locale=lu -Dtests.timezone=Africa/Casablanca
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
00:05:30[junit4] ERROR 0.08s J1 | TestLongBitSet.testHugeCapacity <<<
00:05:30[junit4]> Throwable #1: java.lang.OutOfMemoryError: Java heap
space
00:05:30[junit4]> at
__randomizedtesting.SeedInfo.seed([FB56CEB40315E44E:8116844FEAE5C24A]:0)
00:05:30[junit4]> at
org.apache.lucene.util.ArrayUtil.growExact(ArrayUtil.java:323)
00:05:30[junit4]> at
org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:332)
00:05:30[junit4]> at
org.apache.lucene.util.LongBitSet.ensureCapacity(LongBitSet.java:54)
00:05:30[junit4]> at
org.apache.lucene.util.TestLongBitSet.testHugeCapacity(TestLongBitSet.java:343)
00:05:30[junit4]> at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
00:05:30[junit4]> at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
00:05:30[junit4]> at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
00:05:30[junit4]> at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
00:05:30[junit4] 2> NOTE: test params are: codec=Asserting(Lucene86): {},
docValues:{}, maxPointsInLeafNode=246, maxMBSortInHeap=6.097958740443044,
sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=lu,
timezone=Africa/Casablanca
00:05:30[junit4] 2> NOTE: Linux 4.12.14-122.26-default amd64/Oracle
Corporation 11.0.2 (64-bit)/cpus=32,threads=1,free=346124224,total=362807296
00:05:30[junit4] 2> NOTE: All tests run in this JVM: [TestSynonymQuery,
TestCachingCollector, TestLucene86PointsFormat, TestSizeBoundedForceMerge,
TestSimpleWKTShapeParsing, TestMergeRateLimiter, TestAxiomaticSimilarity,
TestReaderClosed, TestSpanExplanations, TestSingleInstanceLockFactory,
TestIndexSorting, TestIndexInput, TestBasicModelIn, TestPoint2D,
TestAtomicUpdate, TestOneMergeWrappingMergePolicy,
TestTragicIndexWriterDeadlock, TestRollingUpdates, TestCodecHoldsOpenFiles,
TestLucene50StoredFieldsFormatHighCompression, TestFieldsReader,
TestAxiomaticF2EXP, TestPointValues, TestIndependenceStandardized,
TestTwoPhaseCommitTool, TestLongBitSet]
00:05:30[junit4] Completed [87/566 (1!)] on J1 in 3.30s, 11 tests, 1 error
<<< FAILURES!
{noformat}
my suspicion is that some other test is holding on to memory here. I am trying
to record all failures here maybe we find some tests in common. The test itself
needs about 300MB to run so that should be ok with a 512M JVM
> TestLongBitSet.testHugeCapacity sometimes fails on nightly
> ---
>
> Key: LUCENE-9419
> URL: https://issues.apache.org/jira/browse/LUCENE-9419
> Project: Lucene - Core
> Issue Type: Test
>Reporter: Simon Willnauer
>Priority: Minor
>
> I see this once in a while on our CI with -Dtests.nightly=true
> {noformat}
> 00:05:30[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestLongBitSet -Dtests.method=testHugeCapacity
> -Dtests.seed=FB56CEB40315E44E -Dtests.nightly=true -Dtests.slow=true
> -Dtests.badapples=true -Dtests.locale=lu -Dtests.timezone=Africa/Casablanca
> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
> 00:05:30[junit4] ERROR 0.08s J1 | TestLongBitSet.testHugeCapacity <<<
> 00:05:30[junit4]> Throwable #1: java.lang.OutOfMemoryError: Java heap
> space
> 00:05:30[junit4]> at
> __randomizedtesting.SeedInfo.seed([FB56CEB40315E44E:8116844FEAE5C24A]:0)
> 00:05:30[junit4]> at
> org.apache.lucene.util.ArrayUtil.growExact(ArrayUtil.java:323)
> 00:05:30[junit4]> at
> org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:332)
> 00:05:30[junit4]> at
> org.apache.lucene.util.LongBitSet.ensureCapacity(LongBitSet.java:54)
> 00:05:30[junit4]> at
> org.apache.lucene.util.TestLongBitSet.testHugeCapacity(TestLongBitSet.java:343)
> 00:05:30[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 00:05:30[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 00:05:30[junit4]> at
>
[jira] [Created] (LUCENE-9419) TestLongBitSet.testHugeCapacity sometimes fails on nightly
Simon Willnauer created LUCENE-9419:
---
Summary: TestLongBitSet.testHugeCapacity sometimes fails on
nightly
Key: LUCENE-9419
URL: https://issues.apache.org/jira/browse/LUCENE-9419
Project: Lucene - Core
Issue Type: Test
Environment: I see this once in a while on our CI with
-Dtests.nightly=true
{noformat}
00:05:30[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestLongBitSet -Dtests.method=testHugeCapacity
-Dtests.seed=FB56CEB40315E44E -Dtests.nightly=true -Dtests.slow=true
-Dtests.badapples=true -Dtests.locale=lu -Dtests.timezone=Africa/Casablanca
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
00:05:30[junit4] ERROR 0.08s J1 | TestLongBitSet.testHugeCapacity <<<
00:05:30[junit4]> Throwable #1: java.lang.OutOfMemoryError: Java heap
space
00:05:30[junit4]> at
__randomizedtesting.SeedInfo.seed([FB56CEB40315E44E:8116844FEAE5C24A]:0)
00:05:30[junit4]> at
org.apache.lucene.util.ArrayUtil.growExact(ArrayUtil.java:323)
00:05:30[junit4]> at
org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:332)
00:05:30[junit4]> at
org.apache.lucene.util.LongBitSet.ensureCapacity(LongBitSet.java:54)
00:05:30[junit4]> at
org.apache.lucene.util.TestLongBitSet.testHugeCapacity(TestLongBitSet.java:343)
00:05:30[junit4]> at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
00:05:30[junit4]> at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
00:05:30[junit4]> at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
00:05:30[junit4]> at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
00:05:30[junit4] 2> NOTE: test params are: codec=Asserting(Lucene86): {},
docValues:{}, maxPointsInLeafNode=246, maxMBSortInHeap=6.097958740443044,
sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=lu,
timezone=Africa/Casablanca
00:05:30[junit4] 2> NOTE: Linux 4.12.14-122.26-default amd64/Oracle
Corporation 11.0.2 (64-bit)/cpus=32,threads=1,free=346124224,total=362807296
00:05:30[junit4] 2> NOTE: All tests run in this JVM: [TestSynonymQuery,
TestCachingCollector, TestLucene86PointsFormat, TestSizeBoundedForceMerge,
TestSimpleWKTShapeParsing, TestMergeRateLimiter, TestAxiomaticSimilarity,
TestReaderClosed, TestSpanExplanations, TestSingleInstanceLockFactory,
TestIndexSorting, TestIndexInput, TestBasicModelIn, TestPoint2D,
TestAtomicUpdate, TestOneMergeWrappingMergePolicy,
TestTragicIndexWriterDeadlock, TestRollingUpdates, TestCodecHoldsOpenFiles,
TestLucene50StoredFieldsFormatHighCompression, TestFieldsReader,
TestAxiomaticF2EXP, TestPointValues, TestIndependenceStandardized,
TestTwoPhaseCommitTool, TestLongBitSet]
00:05:30[junit4] Completed [87/566 (1!)] on J1 in 3.30s, 11 tests, 1 error
<<< FAILURES!
{noformat}
my suspicion is that some other test is holding on to memory here. I am trying
to record all failures here maybe we find some tests in common. The test itself
needs about 300MB to run so that should be ok with a 512M JVM
Reporter: Simon Willnauer
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer resolved LUCENE-8962.
-
Fix Version/s: master (9.0)
Resolution: Fixed
added to 8.6 and master. Thanks everyone here!
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: master (9.0), 8.6
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch,
> failure_log.txt, test.diff
>
> Time Spent: 22h 10m
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17141809#comment-17141809
]
Simon Willnauer commented on LUCENE-8962:
-
It seems like this caused this test failure:
{noformat}
18:06:29[junit4] Suite: org.apache.lucene.index.TestIndexWriter
18:06:29[junit4] 2> Mod 21, 2020 3:06:19 EBONGI
com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
uncaughtException
18:06:29[junit4] 2> WARNING: Uncaught exception in thread:
Thread[Thread-1905,5,TGRP-TestIndexWriter]
18:06:29[junit4] 2> java.lang.AssertionError:
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
18:06:29[junit4] 2> at
__randomizedtesting.SeedInfo.seed([83A78A6E34594E08]:0)
18:06:29[junit4] 2> at
org.apache.lucene.index.TestIndexWriter.lambda$testRandomOperations$48(TestIndexWriter.java:3886)
18:06:29[junit4] 2> at
java.base/java.lang.Thread.run(Thread.java:834)
18:06:29[junit4] 2> Caused by:
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
18:06:29[junit4] 2> at
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:739)
18:06:29[junit4] 2> at
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:753)
18:06:29[junit4] 2> at
org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1330)
18:06:29[junit4] 2> at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1619)
18:06:29[junit4] 2> at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1262)
18:06:29[junit4] 2> at
org.apache.lucene.index.TestIndexWriter.lambda$testRandomOperations$48(TestIndexWriter.java:3873)
18:06:29[junit4] 2> ... 1 more
18:06:29[junit4] 2> Caused by: java.nio.file.NoSuchFileException: _7.cfs
18:06:29[junit4] 2> at
org.apache.lucene.store.ByteBuffersDirectory.deleteFile(ByteBuffersDirectory.java:148)
18:06:29[junit4] 2> at
org.apache.lucene.store.MockDirectoryWrapper.deleteFile(MockDirectoryWrapper.java:607)
18:06:29[junit4] 2> at
org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:38)
18:06:29[junit4] 2> at
org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:696)
18:06:29[junit4] 2> at
org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:690)
18:06:29[junit4] 2> at
org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:589)
18:06:29[junit4] 2> at
org.apache.lucene.index.IndexFileDeleter.deleteCommits(IndexFileDeleter.java:382)
18:06:29[junit4] 2> at
org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:527)
18:06:29[junit4] 2> at
org.apache.lucene.index.IndexWriter.finishCommit(IndexWriter.java:3546)
18:06:29[junit4] 2> at
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3502)
18:06:29[junit4] 2> at
org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3452)
18:06:29[junit4] 2> at
org.apache.lucene.index.TestIndexWriter.lambda$testRandomOperations$48(TestIndexWriter.java:3879)
18:06:29[junit4] 2> ... 1 more
18:06:29[junit4] 2>
18:06:29[junit4] 2> Mod 21, 2020 3:06:19 EBONGI
com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
uncaughtException
18:06:29[junit4] 2> WARNING: Uncaught exception in thread: Thread[Lucene
Merge Thread #3,5,TGRP-TestIndexWriter]
18:06:29[junit4] 2> org.apache.lucene.index.MergePolicy$MergeException:
java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot
merge
18:06:29[junit4] 2> at
__randomizedtesting.SeedInfo.seed([83A78A6E34594E08]:0)
18:06:29[junit4] 2> at
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
18:06:29[junit4] 2> at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:694)
18:06:29[junit4] 2> Caused by: java.lang.IllegalStateException: this
writer hit an unrecoverable error; cannot merge
18:06:29[junit4] 2> at
org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:4250)
18:06:29[junit4] 2> at
org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:4230)
18:06:29[junit4] 2> at
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083)
18:06:29[junit4] 2> at
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:5650)
18:06:29[junit4] 2> at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
18:06:29[junit4] 2>
[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17141811#comment-17141811
]
Simon Willnauer commented on LUCENE-8962:
-
{quote}
I'm surprised to see that
IndexWriterConfig.DEFAULT_MAX_COMMIT_MERGE_WAIT_SECONDS is 0. After all, the
default is not to merge on commit so if someone configured a merge policy to do
this, shouldn't we wait? I think by default we should wait indefinitely. It's
up to the developer/configure-er of the merge policy to find cheap merges (if
any). Keeping it at 0 creates a gotcha for yet another setting that's required
to merge on commit.
{quote}
I don't think we should. It's such an expert setting and unless we implement it
ourself I think we should keep it that way. I'd assume if we make it non-zero
by default that our MPs implement it and I can use it right away. If we decide
to do this in the future we can keep the default and implement that method by
default without enabling the feature. If we set it to non-zero that option goes
away.
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: 8.6
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch
>
> Time Spent: 18.5h
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146321#comment-17146321
]
Simon Willnauer commented on LUCENE-8962:
-
I think we are ready for review I opened a PR here
https://github.com/apache/lucene-solr/pull/1617
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: 8.6
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch,
> failure_log.txt, test.diff
>
> Time Spent: 20h 40m
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142197#comment-17142197
]
Simon Willnauer commented on LUCENE-8962:
-
I attached a log file with a failure and index writer logging enabled. I think
I have a smoking gun here:
{noformat}
[junit4] 1> IW 2 [2020-06-22T15:33:08.479370Z; Lucene Merge Thread #31]:
merged segment _2f(9.0.0):c1:[diagnostics={os.version=10.14.6,
mergeMaxNumSegments=-1, java.version=12.0.1, java.vm.version=12.0.1+12,
lucene.version=9.0.0, timestamp=1592839988473, os=Mac OS X,
java.runtime.version=12.0.1+12, mergeFactor=1, os.arch=x86_64, source=merge,
java.vendor=Oracle
Corporation}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]
:id=70ut1o7n9kgx798rria9sma3n is 100% deleted; skipping insert
[junit4] 1> SM 2 [2020-06-22T15:33:08.479615Z; Lucene Merge Thread #32]: 0
msec to write field infos [29 docs]
[junit4] 1> IFD 2 [2020-06-22T15:33:08.479662Z; Lucene Merge Thread #31]:
will delete new file "_2f.cfe"
[junit4] 1> IFD 2 [2020-06-22T15:33:08.479712Z; Lucene Merge Thread #31]:
will delete new file "_2f.cfs"
[junit4] 1> IFD 2 [2020-06-22T15:33:08.479740Z; Lucene Merge Thread #31]:
will delete new file "_2f.si"
[junit4] 1> IFD 2 [2020-06-22T15:33:08.479770Z; Lucene Merge Thread #31]:
delete [_2f.cfe, _2f.cfs, _2f.si]
{noformat}
and further down
{noformat}
[junit4] 2> Caused by: java.nio.file.NoSuchFileException: _2f.cfs
[junit4] 2>at
org.apache.lucene.store.ByteBuffersDirectory.deleteFile(ByteBuffersDirectory.java:148)
[junit4] 2>at
org.apache.lucene.store.MockDirectoryWrapper.deleteFile(MockDirectoryWrapper.java:607)
[junit4] 2>at
org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:38)
[junit4] 2>at
org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:696)
[junit4] 2>at
org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:690)
[junit4] 2>at
org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:589)
[junit4] 2>at
org.apache.lucene.index.IndexFileDeleter.deleteCommits(IndexFileDeleter.java:382)
[junit4] 2>at
org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:527)
[junit4] 2>at
org.apache.lucene.index.IndexWriter.finishCommit(IndexWriter.java:3546)
[junit4] 2>at
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3502)
[junit4] 2>at
org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3452)
[junit4] 2>at
org.apache.lucene.index.TestIndexWriter.lambda$testRandomOperations$48(TestIndexWriter.java:3879)
[junit4] 2>... 1 more
[junit4] 2>
{noformat}
I think we are not respecting the fact that it's fully deleted and dropped
because of this before we get a chance to incRef the segment. I will work on a
patch for this.
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: 8.6
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch, failure_log.txt
>
> Time Spent: 18h 50m
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best
[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142184#comment-17142184
]
Simon Willnauer commented on LUCENE-8962:
-
I can reproduce this every 10 iterations. Funny enough I can't in my IDE only
when I repeatedly run it on the CMD. This line reproduces for me quite
frequently:
{noformat}
ant test -Dtestcase=TestIndexWriter -Dtests.method=testRandomOperations
-Dtests.seed=83A78A6E34594E08 -Dtests.nightly=true -Dtests.slow=true
-Dtests.badapples=true -Dtests.locale=teo-KE
-Dtests.timezone=Antarctica/Rothera -Dtests.asserts=true
-Dtests.file.encoding=ISO-8859-1
{noformat}
I also checked if it's related to calling mergeFinished multiple times but it
doesn't
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: 8.6
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch
>
> Time Spent: 18h 50m
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer updated LUCENE-8962:
Attachment: failure_log.txt
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: 8.6
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch, failure_log.txt
>
> Time Spent: 18h 50m
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9508) DocumentsWriter doesn't check for BlockedFlushes in stall mode``
[ https://issues.apache.org/jira/browse/LUCENE-9508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer resolved LUCENE-9508. - Fix Version/s: 8.8 master (9.0) Lucene Fields: New,Patch Available (was: New) Resolution: Fixed > DocumentsWriter doesn't check for BlockedFlushes in stall mode`` > > > Key: LUCENE-9508 > URL: https://issues.apache.org/jira/browse/LUCENE-9508 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: 8.5.1 >Reporter: Sorabh Hamirwasia >Priority: Major > Labels: IndexWriter > Fix For: master (9.0), 8.8 > > Time Spent: 1h > Remaining Estimate: 0h > > Hi, > I was investigating an issue where the memory usage by a single Lucene > IndexWriter went up to ~23GB. Lucene has a concept of stalling in case the > memory used by each index breaches the 2 X ramBuffer limit (10% of JVM heap, > this case ~3GB). So ideally memory usage should not go above that limit. I > looked into the heap dump and found that the fullFlush thread when enters > *markForFullFlush* method, it tries to take lock on the ThreadStates of all > the DWPT thread sequentially. If lock on one of the ThreadState is blocked > then it will block indefinitely. This is what happened in my case, where one > of the DWPT thread was stuck in indexing process. Due to this fullFlush > thread was unable to populate the flush queue even though the stall mode was > detected. This caused the new indexing request which came on indexing thread > to continue after sleeping for a second, and continue with indexing. In > **preUpdate()** method it looks for the stalled case and see if there is any > pending flushes (based on flush queue), if not then sleep and continue. > Question: > 1) Should **preUpdate** look into the blocked flushes information as well > instead of just flush queue ? > 2) Should the fullFlush thread wait indefinitely for the lock on ThreadStates > ? Since single blocking writing thread can block the full flush here. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15013) Reproducing test failure TestFieldCacheSort 8x
[
https://issues.apache.org/jira/browse/SOLR-15013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer updated SOLR-15013:
---
Issue Type: Test (was: Bug)
> Reproducing test failure TestFieldCacheSort 8x
> --
>
> Key: SOLR-15013
> URL: https://issues.apache.org/jira/browse/SOLR-15013
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Erick Erickson
>Assignee: Simon Willnauer
>Priority: Major
>
> ant test -Dtestcase=TestFieldCacheSort
> -Dtests.method=testEmptyStringVsNullStringSort -Dtests.seed=2E14D932C133811F
> -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=hr-BA
> -Dtests.timezone=America/Scoresbysund -Dtests.asserts=true
> -Dtests.file.encoding=ISO-8859-1
>
>
> [junit4] Started J0 PID(96093@localhost).
> [junit4] Suite: org.apache.solr.uninverting.TestFieldCacheSort
> [junit4] 2> 1334 INFO
> (SUITE-TestFieldCacheSort-seed#[2E14D932C133811F]-worker) [ ]
> o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to
> test-framework derived value of
> '/Users/Erick/apache/solr/solrtest8/solr/server/solr/configsets/_default/conf'
> [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestFieldCacheSort
> -Dtests.method=testEmptyStringVsNullStringSort -Dtests.seed=2E14D932C133811F
> -Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true
> -Dtests.locale=hr-BA -Dtests.timezone=America/Scoresbysund
> -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
> [junit4] FAILURE 0.40s | TestFieldCacheSort.testEmptyStringVsNullStringSort
> <<<
> [junit4] > Throwable #1: java.lang.AssertionError: expected:<1> but was:<0>
> [junit4] > at
> __randomizedtesting.SeedInfo.seed([2E14D932C133811F:4FF7EBE5B95287AF]:0)
> [junit4] > at
> org.apache.solr.uninverting.TestFieldCacheSort.testEmptyStringVsNullStringSort(TestFieldCacheSort.java:1610)
> [junit4] > at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit4] > at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [junit4] > at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit4] > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> [junit4] > at java.base/java.lang.Thread.run(Thread.java:834)
> [junit4] 2> NOTE: test params are: codec=CheapBastard,
> sim=Asserting(RandomSimilarity(queryNorm=false): \{t=DFR I(F)BZ(0.3)}),
> locale=hr-BA, timezone=America/Scoresbysund
> [junit4] 2> NOTE: Mac OS X 10.16 x86_64/AdoptOpenJDK 11.0.5
> (64-bit)/cpus=12,threads=1,free=468375472,total=536870912
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-15013) Reproducing test failure TestFieldCacheSort 8x
[
https://issues.apache.org/jira/browse/SOLR-15013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer resolved SOLR-15013.
Assignee: Simon Willnauer
Resolution: Fixed
I pushed a fix...
> Reproducing test failure TestFieldCacheSort 8x
> --
>
> Key: SOLR-15013
> URL: https://issues.apache.org/jira/browse/SOLR-15013
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Erick Erickson
>Assignee: Simon Willnauer
>Priority: Major
>
> ant test -Dtestcase=TestFieldCacheSort
> -Dtests.method=testEmptyStringVsNullStringSort -Dtests.seed=2E14D932C133811F
> -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=hr-BA
> -Dtests.timezone=America/Scoresbysund -Dtests.asserts=true
> -Dtests.file.encoding=ISO-8859-1
>
>
> [junit4] Started J0 PID(96093@localhost).
> [junit4] Suite: org.apache.solr.uninverting.TestFieldCacheSort
> [junit4] 2> 1334 INFO
> (SUITE-TestFieldCacheSort-seed#[2E14D932C133811F]-worker) [ ]
> o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to
> test-framework derived value of
> '/Users/Erick/apache/solr/solrtest8/solr/server/solr/configsets/_default/conf'
> [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestFieldCacheSort
> -Dtests.method=testEmptyStringVsNullStringSort -Dtests.seed=2E14D932C133811F
> -Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true
> -Dtests.locale=hr-BA -Dtests.timezone=America/Scoresbysund
> -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
> [junit4] FAILURE 0.40s | TestFieldCacheSort.testEmptyStringVsNullStringSort
> <<<
> [junit4] > Throwable #1: java.lang.AssertionError: expected:<1> but was:<0>
> [junit4] > at
> __randomizedtesting.SeedInfo.seed([2E14D932C133811F:4FF7EBE5B95287AF]:0)
> [junit4] > at
> org.apache.solr.uninverting.TestFieldCacheSort.testEmptyStringVsNullStringSort(TestFieldCacheSort.java:1610)
> [junit4] > at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit4] > at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [junit4] > at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit4] > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> [junit4] > at java.base/java.lang.Thread.run(Thread.java:834)
> [junit4] 2> NOTE: test params are: codec=CheapBastard,
> sim=Asserting(RandomSimilarity(queryNorm=false): \{t=DFR I(F)BZ(0.3)}),
> locale=hr-BA, timezone=America/Scoresbysund
> [junit4] 2> NOTE: Mac OS X 10.16 x86_64/AdoptOpenJDK 11.0.5
> (64-bit)/cpus=12,threads=1,free=468375472,total=536870912
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9621) pendingNumDocs doesn't match totalMaxDoc if tragedy on flush()
[
https://issues.apache.org/jira/browse/LUCENE-9621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247965#comment-17247965
]
Simon Willnauer commented on LUCENE-9621:
-
do we have a stacktrace of the OOM as well where it happened?
> pendingNumDocs doesn't match totalMaxDoc if tragedy on flush()
> --
>
> Key: LUCENE-9621
> URL: https://issues.apache.org/jira/browse/LUCENE-9621
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Affects Versions: 8.6.3
>Reporter: Michael Froh
>Priority: Major
>
> While implementing a test to trigger an OutOfMemoryError on flush() in
> https://github.com/apache/lucene-solr/pull/2088, I noticed that the OOME was
> followed by an assertion failure on rollback with the following stacktrace:
> {code:java}
> java.lang.AssertionError: pendingNumDocs 1 != 0 totalMaxDoc
> at
> __randomizedtesting.SeedInfo.seed([ABBF17C4E0FCDEE5:DDC8E99910AFC8FF]:0)
> at
> org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2398)
> at
> org.apache.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:5196)
> at
> org.apache.lucene.index.IndexWriter.tragicEvent(IndexWriter.java:5186)
> at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3932)
> at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3874)
> at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3853)
> at
> org.apache.lucene.index.TestIndexWriterDelete.testDeleteAllRepeated(TestIndexWriterDelete.java:496)
> {code}
> We should probably look into how exactly we behave with this kind of tragedy
> on flush().
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9508) DocumentsWriter doesn't check for BlockedFlushes in stall mode``
[ https://issues.apache.org/jira/browse/LUCENE-9508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231249#comment-17231249 ] Simon Willnauer commented on LUCENE-9508: - uups sorry! I meant [~shamirwasia] thanks for the headsup > DocumentsWriter doesn't check for BlockedFlushes in stall mode`` > > > Key: LUCENE-9508 > URL: https://issues.apache.org/jira/browse/LUCENE-9508 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: 8.5.1 >Reporter: Sorabh Hamirwasia >Priority: Major > Labels: IndexWriter > > Hi, > I was investigating an issue where the memory usage by a single Lucene > IndexWriter went up to ~23GB. Lucene has a concept of stalling in case the > memory used by each index breaches the 2 X ramBuffer limit (10% of JVM heap, > this case ~3GB). So ideally memory usage should not go above that limit. I > looked into the heap dump and found that the fullFlush thread when enters > *markForFullFlush* method, it tries to take lock on the ThreadStates of all > the DWPT thread sequentially. If lock on one of the ThreadState is blocked > then it will block indefinitely. This is what happened in my case, where one > of the DWPT thread was stuck in indexing process. Due to this fullFlush > thread was unable to populate the flush queue even though the stall mode was > detected. This caused the new indexing request which came on indexing thread > to continue after sleeping for a second, and continue with indexing. In > **preUpdate()** method it looks for the stalled case and see if there is any > pending flushes (based on flush queue), if not then sleep and continue. > Question: > 1) Should **preUpdate** look into the blocked flushes information as well > instead of just flush queue ? > 2) Should the fullFlush thread wait indefinitely for the lock on ThreadStates > ? Since single blocking writing thread can block the full flush here. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9508) DocumentsWriter doesn't check for BlockedFlushes in stall mode``
[
https://issues.apache.org/jira/browse/LUCENE-9508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230497#comment-17230497
]
Simon Willnauer commented on LUCENE-9508:
-
Hey Zach, thanks for opening this. Lemme ask some question and clarify what is
going on here first:
{quote} 2) Should the fullFlush thread wait indefinitely for the lock on
ThreadStates ? Since single blocking writing thread can block the full flush
here.
{quote}
yes we have to block on the threadstates here since this is the contract of
full flush in order to atomically commit changes and establish a happens before
relationship.
{quote}
1) Should *preUpdate* look into the blocked flushes information as well instead
of just flush queue ?
{quote}
I am not sure what is would do with the information in blocked flushes? Can you
elaborate on this? we can't let blocked flushes go unless the full flush is
over otherwise we will have inconsistent commits.
Can you share your IndexWriter config and how you configured the 10% heap?
Can you also share what thread holds the ThreadState that the full flush is
waiting for? I wonder what causes this situation.
> DocumentsWriter doesn't check for BlockedFlushes in stall mode``
>
>
> Key: LUCENE-9508
> URL: https://issues.apache.org/jira/browse/LUCENE-9508
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Affects Versions: 8.5.1
>Reporter: Sorabh Hamirwasia
>Priority: Major
> Labels: IndexWriter
>
> Hi,
> I was investigating an issue where the memory usage by a single Lucene
> IndexWriter went up to ~23GB. Lucene has a concept of stalling in case the
> memory used by each index breaches the 2 X ramBuffer limit (10% of JVM heap,
> this case ~3GB). So ideally memory usage should not go above that limit. I
> looked into the heap dump and found that the fullFlush thread when enters
> *markForFullFlush* method, it tries to take lock on the ThreadStates of all
> the DWPT thread sequentially. If lock on one of the ThreadState is blocked
> then it will block indefinitely. This is what happened in my case, where one
> of the DWPT thread was stuck in indexing process. Due to this fullFlush
> thread was unable to populate the flush queue even though the stall mode was
> detected. This caused the new indexing request which came on indexing thread
> to continue after sleeping for a second, and continue with indexing. In
> **preUpdate()** method it looks for the stalled case and see if there is any
> pending flushes (based on flush queue), if not then sleep and continue.
> Question:
> 1) Should **preUpdate** look into the blocked flushes information as well
> instead of just flush queue ?
> 2) Should the fullFlush thread wait indefinitely for the lock on ThreadStates
> ? Since single blocking writing thread can block the full flush here.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (LUCENE-9669) N-2 compatibility for file formats
[ https://issues.apache.org/jira/browse/LUCENE-9669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer reassigned LUCENE-9669: --- Fix Version/s: master (9.0) Assignee: Simon Willnauer > N-2 compatibility for file formats > -- > > Key: LUCENE-9669 > URL: https://issues.apache.org/jira/browse/LUCENE-9669 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Simon Willnauer >Assignee: Simon Willnauer >Priority: Major > Fix For: master (9.0) > > > Currently Lucene supports reading and writing indices that have been > created with the current or previous (N-1) version of Lucene. Lucene > refuses to open an index created by N-2 or earlier versions. > I would like to propose that Lucene adds support for opening indices > created by version N-2 in read-only mode. Here's what I have in mind: > - Read-only support. You can open a reader on an index created by > version N-2, but you cannot open an IndexWriter on it, meaning that > you can neither delete, update, add documents or force-merge N-2 > indices. > - File-format compatibility only. File-format compatibility enables > reading the content of old indices, but not more. Everything that is > done on top of file formats like analysis or the encoding of length > normalization factors is not guaranteed and only supported on a > best-effort basis. > The reason I came up with these limitations is because I wanted to > make the scope minimal in order to retain Lucene's ability to move > forward. If there is consensus to move forward with this, I would like > to target Lucene 9.0 with this change. > This is a follow-up of the mailinglist thread > [here|http://mail-archives.apache.org/mod_mbox/lucene-dev/202101.mbox/%3ccaahmpkg1bd9rtmvfr4jweygyaxwz5osy1hv9ubzks0fhsfn...@mail.gmail.com%3E] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9669) N-2 compatibility for file formats
Simon Willnauer created LUCENE-9669: --- Summary: N-2 compatibility for file formats Key: LUCENE-9669 URL: https://issues.apache.org/jira/browse/LUCENE-9669 Project: Lucene - Core Issue Type: Improvement Reporter: Simon Willnauer Currently Lucene supports reading and writing indices that have been created with the current or previous (N-1) version of Lucene. Lucene refuses to open an index created by N-2 or earlier versions. I would like to propose that Lucene adds support for opening indices created by version N-2 in read-only mode. Here's what I have in mind: - Read-only support. You can open a reader on an index created by version N-2, but you cannot open an IndexWriter on it, meaning that you can neither delete, update, add documents or force-merge N-2 indices. - File-format compatibility only. File-format compatibility enables reading the content of old indices, but not more. Everything that is done on top of file formats like analysis or the encoding of length normalization factors is not guaranteed and only supported on a best-effort basis. The reason I came up with these limitations is because I wanted to make the scope minimal in order to retain Lucene's ability to move forward. If there is consensus to move forward with this, I would like to target Lucene 9.0 with this change. This is a follow-up of the mailinglist thread [here|http://mail-archives.apache.org/mod_mbox/lucene-dev/202101.mbox/%3ccaahmpkg1bd9rtmvfr4jweygyaxwz5osy1hv9ubzks0fhsfn...@mail.gmail.com%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9669) N-2 compatibility for file formats
[ https://issues.apache.org/jira/browse/LUCENE-9669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer resolved LUCENE-9669. - Resolution: Fixed > N-2 compatibility for file formats > -- > > Key: LUCENE-9669 > URL: https://issues.apache.org/jira/browse/LUCENE-9669 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Simon Willnauer >Assignee: Simon Willnauer >Priority: Major > Fix For: master (9.0) > > Time Spent: 3.5h > Remaining Estimate: 0h > > Currently Lucene supports reading and writing indices that have been > created with the current or previous (N-1) version of Lucene. Lucene > refuses to open an index created by N-2 or earlier versions. > I would like to propose that Lucene adds support for opening indices > created by version N-2 in read-only mode. Here's what I have in mind: > - Read-only support. You can open a reader on an index created by > version N-2, but you cannot open an IndexWriter on it, meaning that > you can neither delete, update, add documents or force-merge N-2 > indices. > - File-format compatibility only. File-format compatibility enables > reading the content of old indices, but not more. Everything that is > done on top of file formats like analysis or the encoding of length > normalization factors is not guaranteed and only supported on a > best-effort basis. > The reason I came up with these limitations is because I wanted to > make the scope minimal in order to retain Lucene's ability to move > forward. If there is consensus to move forward with this, I would like > to target Lucene 9.0 with this change. > This is a follow-up of the mailinglist thread > [here|http://mail-archives.apache.org/mod_mbox/lucene-dev/202101.mbox/%3ccaahmpkg1bd9rtmvfr4jweygyaxwz5osy1hv9ubzks0fhsfn...@mail.gmail.com%3E] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9817) pathological test fixes
[ https://issues.apache.org/jira/browse/LUCENE-9817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302443#comment-17302443 ] Simon Willnauer commented on LUCENE-9817: - sure we can totally make this a base class and split it up by versions somehow. > pathological test fixes > --- > > Key: LUCENE-9817 > URL: https://issues.apache.org/jira/browse/LUCENE-9817 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Robert Muir >Priority: Major > Attachments: LUCENE-9817.patch, LUCENE-9817.patch, LUCENE-9817.patch > > > There are now 13,000+ tests in lucene, and if you don't have dozens of cores > the situation is slow (around 7 minutes here, with everything tuned as fast > as i can get it, running on tmpfs). > It is tricky to keep the situation sustainable: so many tests that usually > just take a few seconds but they all add up. To put it in perspective, > imagine if all 13000 tests only took 1s each, that's 3.5 hours of cpu time. > From my inspection, there are a few cases of inefficiency: > * tests with bad random parameters: they might normally be semi-well-behaved, > but "rarely" take 30 seconds. That's maybe like a 1% chance but keep in mind > 1% equates to 130 wild-west tests every run. > * tests spinning up too many threads and indexing too many docs > unnecessarily: there might literally be thousands of these, so that's a hard > problem to fix... and developers love to use lots of threads and docs in > tests. > * tests just being inefficient: stuff like creating indexes in setup/teardown > when they have many methods that may not even use them (hey, why did > testEqualsHashcode take 30 seconds, what is it doing?) > I only worked on the first case here, if i fixed anything involving the other > two, it was just because I noticed them while I was there. I temporarily > overrode methods like LuceneTestCase.rarely(), atLeast(), and so on to > present more pathological/worst-case conditions and tried to address them all. > So here's a patch to give ~ 80 seconds of cpu-time in tests back. YMMV, maybe > it helps you more if you are actually using hard disks and stuff! > Fixing the other issues here will require some more creativity/work, I will > followup. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9817) pathological test fixes
[ https://issues.apache.org/jira/browse/LUCENE-9817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302410#comment-17302410 ] Simon Willnauer commented on LUCENE-9817: - thanks rob for taking the time to do all this analysis. I do wonder if some tests should be @nightly only for N-2 indices or if we can take a random list of versions we test in each of these tests to make sure we have more reliable times even with more versions released? > pathological test fixes > --- > > Key: LUCENE-9817 > URL: https://issues.apache.org/jira/browse/LUCENE-9817 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Robert Muir >Priority: Major > Attachments: LUCENE-9817.patch, LUCENE-9817.patch, LUCENE-9817.patch > > > There are now 13,000+ tests in lucene, and if you don't have dozens of cores > the situation is slow (around 7 minutes here, with everything tuned as fast > as i can get it, running on tmpfs). > It is tricky to keep the situation sustainable: so many tests that usually > just take a few seconds but they all add up. To put it in perspective, > imagine if all 13000 tests only took 1s each, that's 3.5 hours of cpu time. > From my inspection, there are a few cases of inefficiency: > * tests with bad random parameters: they might normally be semi-well-behaved, > but "rarely" take 30 seconds. That's maybe like a 1% chance but keep in mind > 1% equates to 130 wild-west tests every run. > * tests spinning up too many threads and indexing too many docs > unnecessarily: there might literally be thousands of these, so that's a hard > problem to fix... and developers love to use lots of threads and docs in > tests. > * tests just being inefficient: stuff like creating indexes in setup/teardown > when they have many methods that may not even use them (hey, why did > testEqualsHashcode take 30 seconds, what is it doing?) > I only worked on the first case here, if i fixed anything involving the other > two, it was just because I noticed them while I was there. I temporarily > overrode methods like LuceneTestCase.rarely(), atLeast(), and so on to > present more pathological/worst-case conditions and tried to address them all. > So here's a patch to give ~ 80 seconds of cpu-time in tests back. YMMV, maybe > it helps you more if you are actually using hard disks and stuff! > Fixing the other issues here will require some more creativity/work, I will > followup. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10088) Too many open files in TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader
[ https://issues.apache.org/jira/browse/LUCENE-10088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412399#comment-17412399 ] Simon Willnauer commented on LUCENE-10088: -- Hey [~jpountz] I think your analysis is right and makes sense to me. Unfortunately this is somewhat expected and an existing limitation of the throttle mechanics in IW. I also agree it's an edge case and very unlikely to happen anywhere outside of a randomized test. Due to the fact that we are doing a full flush we can't finish flushing the buffered segments that pile up caused by randomly set buffered docs. I think the TooManyOpenFilesExc is only one possible issue we could hit here, OOM would be another. I think we can improve the protection IW offers here and either add a non-aborting exception that rejects the document before it gets our of hand with too many buffered segments which would be a awkward to the user or we improve our stalling option which is currently solely based on [how big our flush buffer|https://github.com/apache/lucene/blob/ee0695eda8bc45611cf0312ea3eb6a8819537537/lucene/core/src/java/org/apache/lucene/index/DocumentsWriterFlushControl.java#L283] is. I think we can certainly find a good default that would prevent 600+ blocked segments without it having any impact in the real world. If I am not mistaken I think we can simply change this one line [here|https://github.com/apache/lucene/blob/ee0695eda8bc45611cf0312ea3eb6a8819537537/lucene/core/src/java/org/apache/lucene/index/DocumentsWriterFlushControl.java#L294] to add some upper-bound to make it work. > Too many open files in > TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader > -- > > Key: LUCENE-10088 > URL: https://issues.apache.org/jira/browse/LUCENE-10088 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Michael McCandless >Priority: Major > > [This build > failure|https://ci-builds.apache.org/job/Lucene/job/Lucene-NightlyTests-main/386/] > reproduces for me. I'll try to dig. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10088) Too many open files in TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader
[ https://issues.apache.org/jira/browse/LUCENE-10088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17413103#comment-17413103 ] Simon Willnauer commented on LUCENE-10088: -- Oh don’t get me wrong Adrien, we do that. We have that protection in place. If we use 2x the configured ram buffer for flushing we stall. The ram buffer is not an upperbound the iw will use but rather a marker that triggers a flush. Yet, we still use a function of it to make sure we don’t go OOM when flushes are slow. > Too many open files in > TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader > -- > > Key: LUCENE-10088 > URL: https://issues.apache.org/jira/browse/LUCENE-10088 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Michael McCandless >Priority: Major > > [This build > failure|https://ci-builds.apache.org/job/Lucene/job/Lucene-NightlyTests-main/386/] > reproduces for me. I'll try to dig. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
