[
https://issues.apache.org/jira/browse/LUCENE-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14392344#comment-14392344
]
Michael McCandless commented on LUCENE-6386:
--------------------------------------------
This is hard to think about :)
In your example I'm assuming "result" is just the same as "level2", but
converted into CFS (ie, s14 really should have been s13.cfs, but I'll just call
it s14 here to try not to lose my sanity).
So, at the exact moment when IW finishes building the final (level3) CFS [s14],
you have:
* original source segments -> 1X
* level 1 segments -> 1X
* level 2 segment [s13] non-CFS -> 1X
* result segment [s14] CFS -> 1X
We do NOT delete level 1 segments after merging to the level 2 non-CFS segment
before creating the result s14 (we used to do this, but it caused
complexity/problems because a non-CFS file can unexpectedly "sneak into" an IW
commit or an NRT reader even when you demanded CFS).
So at that moment, the peak temp disk usage is 3X?
> TestIndexWriterForceMerge still unreliable in NIGHTLY
> -----------------------------------------------------
>
> Key: LUCENE-6386
> URL: https://issues.apache.org/jira/browse/LUCENE-6386
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Robert Muir
> Assignee: Michael McCandless
> Fix For: Trunk, 5.1
>
> Attachments: LUCENE-6386.patch
>
>
> Discovered by ryan beasting (trunk):
> ant test -Dtestcase=TestIndexWriterForceMerge
> -Dtests.method=testForceMergeTempSpaceUsage -Dtests.seed=DC9ADB74850A581B
> -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true
> -Dtests.locale=sr__#Latn -Dtests.timezone=Indian/Chagos -Dtests.asserts=true
> -Dtests.file.encoding=US-ASCII
> {noformat}
> [junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestIndexWriterForceMerge
> -Dtests.method=testForceMergeTempSpaceUsage -Dtests.seed=DC9ADB74850A581B
> -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true
> -Dtests.locale=sr__#Latn -Dtests.timezone=Indian/Chagos -Dtests.asserts=true
> -Dtests.file.encoding=US-ASCII
> [junit4] FAILURE 1.20s |
> TestIndexWriterForceMerge.testForceMergeTempSpaceUsage <<<
> [junit4] > Throwable #1: java.lang.AssertionError: forceMerge used too
> much temporary space: starting usage was 291570 bytes; final usage was 262469
> bytes; max temp usage was 1079501 but should have been 874710 (= 3X starting
> usage), BEFORE=
> [junit4] > _u.scf 146329
> [junit4] > _u.si 635
> [junit4] > |- (inside compound file) _u.fld 2214
> [junit4] > |- (inside compound file) _u.inf 392
> [junit4] > |- (inside compound file) _u.len 2381
> [junit4] > |- (inside compound file) _u.pst 36758
> [junit4] > |- (inside compound file) _u.vec 104144
> [junit4] > _s.pst 1338
> [junit4] > _s.inf 392
> [junit4] > _s.fld 94
> [junit4] > _s.len 221
> [junit4] > _s.vec 3744
> [junit4] > _s.si 624
> [junit4] > _t.fld 94
> [junit4] > _t.len 221
> [junit4] > _t.pst 1338
> [junit4] > _t.inf 392
> [junit4] > _t.vec 3744
> [junit4] > _t.si 624
> [junit4] > _v.fld 94
> [junit4] > _v.pst 1338
> [junit4] > _v.inf 392
> [junit4] > _v.vec 3744
> [junit4] > _v.si 624
> [junit4] > _v.len 221
> [junit4] > _w.len 221
> [junit4] > _w.pst 1338
> [junit4] > _w.inf 392
> [junit4] > _w.fld 94
> [junit4] > _w.si 624
> [junit4] > _w.vec 3744
> [junit4] > _x.vec 3744
> [junit4] > _x.inf 392
> [junit4] > _x.pst 1338
> [junit4] > _x.fld 94
> [junit4] > _x.si 624
> [junit4] > _x.len 221
> [junit4] > _y.fld 94
> [junit4] > _y.pst 1338
> [junit4] > _y.inf 392
> [junit4] > _y.si 624
> [junit4] > _y.vec 3744
> [junit4] > _y.len 221
> [junit4] > _z.fld 94
> [junit4] > _z.pst 1338
> [junit4] > _z.inf 392
> [junit4] > _z.len 221
> [junit4] > _z.vec 3744
> [junit4] > _z.si 624
> [junit4] > _10.si 630
> [junit4] > _10.fld 94
> [junit4] > _10.pst 1338
> [junit4] > _10.inf 392
> [junit4] > _10.vec 3744
> [junit4] > _10.len 221
> [junit4] > _11.len 221
> [junit4] > _11.si 630
> [junit4] > _11.vec 3744
> [junit4] > _11.pst 1338
> [junit4] > _11.inf 392
> [junit4] > _11.fld 94
> [junit4] > _12.vec 3744
> [junit4] > _12.si 630
> [junit4] > _12.len 221
> [junit4] > _12.fld 94
> [junit4] > _12.pst 1338
> [junit4] > _12.inf 392
> [junit4] > _13.fld 94
> [junit4] > _13.vec 3744
> [junit4] > _13.si 630
> [junit4] > _13.pst 1338
> [junit4] > _13.inf 392
> [junit4] > _13.len 221
> [junit4] > _14.fld 94
> [junit4] > _14.pst 1338
> [junit4] > _14.inf 392
> [junit4] > _14.si 630
> [junit4] > _14.vec 3744
> [junit4] > _14.len 221
> [junit4] > _15.len 221
> [junit4] > _15.vec 3744
> [junit4] > _15.si 630
> [junit4] > _15.pst 1338
> [junit4] > _15.inf 392
> [junit4] > _15.fld 94
> [junit4] > _16.vec 3744
> [junit4] > _16.len 221
> [junit4] > _16.fld 94
> [junit4] > _16.si 630
> [junit4] > _16.pst 1338
> [junit4] > _16.inf 392
> [junit4] > _17.vec 3744
> [junit4] > _17.pst 1338
> [junit4] > _17.inf 392
> [junit4] > _17.len 221
> [junit4] > _17.si 630
> [junit4] > _17.fld 94
> [junit4] > _18.pst 1338
> [junit4] > _18.inf 392
> [junit4] > _18.len 221
> [junit4] > _18.vec 3744
> [junit4] > _18.si 630
> [junit4] > _18.fld 94
> [junit4] > _19.fld 94
> [junit4] > _19.si 630
> [junit4] > _19.len 221
> [junit4] > _19.vec 3744
> [junit4] > _19.pst 1338
> [junit4] > _19.inf 392
> [junit4] > _1a.fld 94
> [junit4] > _1a.pst 1338
> [junit4] > _1a.inf 392
> [junit4] > _1a.len 221
> [junit4] > _1a.vec 3744
> [junit4] > _1a.si 630
> [junit4] > _1b.fld 94
> [junit4] > _1b.si 630
> [junit4] > _1b.pst 1338
> [junit4] > _1b.inf 392
> [junit4] > _1b.vec 3744
> [junit4] > _1b.len 221
> [junit4] > _1c.vec 3744
> [junit4] > _1c.pst 1338
> [junit4] > _1c.inf 392
> [junit4] > _1c.fld 94
> [junit4] > _1c.len 221
> [junit4] > _1c.si 630
> [junit4] > _1d.pst 1338
> [junit4] > _1d.inf 392
> [junit4] > _1d.fld 94
> [junit4] > _1d.len 221
> [junit4] > _1d.vec 3744
> [junit4] > _1d.si 630
> [junit4] > _1e.fld 94
> [junit4] > _1e.pst 1338
> [junit4] > _1e.inf 392
> [junit4] > _1e.len 221
> [junit4] > _1e.vec 3744
> [junit4] > _1e.si 630
> [junit4] > _1f.fld 40
> [junit4] > _1f.pst 195
> [junit4] > _1f.inf 392
> [junit4] > _1f.len 149
> [junit4] > _1f.vec 405
> [junit4] > _1f.si 629
> [junit4] > AFTER=
> [junit4] > _1j.si 635
> [junit4] > _1j.scf 261701
> [junit4] > |- (inside compound file) _1j.fld 3982
> [junit4] > |- (inside compound file) _1j.inf 392
> [junit4] > |- (inside compound file) _1j.len 4149
> [junit4] > |- (inside compound file) _1j.pst 66151
> [junit4] > |- (inside compound file) _1j.vec 186577
> [junit4] > at
> __randomizedtesting.SeedInfo.seed([DC9ADB74850A581B:C6581887EB1ABBDC]:0)
> [junit4] > at
> org.apache.lucene.index.TestIndexWriterForceMerge.testForceMergeTempSpaceUsage(TestIndexWriterForceMerge.java:199)
> [junit4] > at java.lang.Thread.run(Thread.java:745)
> [junit4] 2> NOTE: test params are: codec=SimpleText,
> sim=RandomSimilarityProvider(queryNorm=false,coord=no): {id=IB SPL-D2,
> content=DFR I(F)B1}, locale=sr__#Latn, timezone=Indian/Chagos
> [junit4] 2> NOTE: Linux 3.13.0-45-generic amd64/Oracle Corporation
> 1.8.0_40-ea (64-bit)/cpus=8,threads=1,free=208092000,total=253231104
> [junit4] 2> NOTE: All tests run in this JVM: [TestIndexWriterForceMerge]
> [junit4] Completed in 1.54s, 1 test, 1 failure <<< FAILURES!
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
