[
https://issues.apache.org/jira/browse/LUCENE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475666#comment-13475666
]
Gil Tene edited comment on LUCENE-4482 at 10/13/12 5:20 PM:
------------------------------------------------------------
{quote}
Maybe it would be a good idea to provide both C4 -> Memory Management layers,
so also for plain kernels (as configuration option to the JVM like huge pages
in Oracle's). Or is your VM then only as fast as Oracle's?
{quote}
It's not so much a matter of speed as it is a matter of pause time. Zing is not
faster than Oracle's JVM, it's just as fast but without those pesky pauses.
It's those pauses that keep people from using anything more than a tiny amount
of memory in Java these days (to me "tiny" means "a small fraction of a
commodity, $4K server). With the ability to practically (i.e. without
completely stopping for many seconds at a time once is a while) use the nice,
cheap memory we now have in servers comes another form of speed - the kind that
comes from not repeating work.
A 4 to 6 order of magnitude difference in pause time and in sustainable
allocation rate is so big that a C4 that uses the vanilla memory management api
would be unusable at this point. Think of the difference between a 20usec phase
shift and a 20 second pause...
{quote}
...As VirtualBOX's module has similar use-cases like yours for virtualization,
I hope yours does not conflict with that one.
{quote}
We don't test on VirtualBOX, so I don't know for sure. In general, Zing works
fine when run on top of hypervisors that fully support things like 2MB page
mappings (the same sort of support needed for hugetlb feature to work).
Unfortunately, there are some hypervisors out there (e.g. some versions of Xen
for paravirt guests) that don't support that, and will crash a vanilla linux
kernel trying to use hugetlb. Zing won't work in such cases either, and for the
same reasons...
was (Author: giltene):
bq.
Maybe it would be a good idea to provide both C4 -> Memory Management layers,
so also for plain kernels (as configuration option to the JVM like huge pages
in Oracle's). Or is your VM then only as fast as Oracle's?
It's not so much a matter of speed as it is a matter of pause time. Zing is not
faster than Oracle's JVM, it's just as fast but without those pesky pauses.
It's those pauses that keep people from using anything more than a tiny amount
of memory in Java these days (to me "tiny" means "a small fraction of a
commodity, $4K server). With the ability to practically (i.e. without
completely stopping for many seconds at a time once is a while) use the nice,
cheap memory we now have in servers comes another form of speed - the kind that
comes from not repeating work.
A 4 to 6 order of magnitude difference in pause time and in sustainable
allocation rate is so big that a C4 that uses the vanilla memory management api
would be unusable at this point. Think of the difference between a 20usec phase
shift and a 20 second pause...
bq.
...As VirtualBOX's module has similar use-cases like yours for virtualization,
I hope yours does not conflict with that one.
We don't test on VirtualBOX, so I don't know for sure. In general, Zing works
fine when run on top of hypervisors that fully support things like 2MB page
mappings (the same sort of support needed for hugetlb feature to work).
Unfortunately, there are some hypervisors out there (e.g. some versions of Xen
for paravirt guests) that don't support that, and will crash a vanilla linux
kernel trying to use hugetlb. Zing won't work in such cases either, and for the
same reasons...
> Likely Zing JVM bug causes failures in TestPayloadNearQuery
> -----------------------------------------------------------
>
> Key: LUCENE-4482
> URL: https://issues.apache.org/jira/browse/LUCENE-4482
> Project: Lucene - Core
> Issue Type: Bug
> Environment: Lucene trunk, rev 1397735
> Zing:
> {noformat}
> java version "1.6.0_31"
> Java(TM) SE Runtime Environment (build 1.6.0_31-6)
> Java HotSpot(TM) 64-Bit Tiered VM (build
> 1.6.0_31-ZVM_5.2.3.0-b6-product-azlinuxM-X86_64, mixed mode)
> {noformat}
> Ubuntu 12.04 LTS 3.2.0-23-generic kernel
> Reporter: Michael McCandless
> Attachments: LUCENE-4482.patch
>
>
> I dug into one of the Lucene test failures when running with Zing JVM
> (available free for open source devs...). At least one other test
> sometimes fails but I haven't dug into that yet.
> I managed to get the failure easily reproduced: with the attached
> patch, on rev 1397735 checkout, if you cd to lucene/core and run:
> {noformat}
> ant test -Dtests.jvms=1 -Dtests.seed=C3802435F5FB39D0
> -Dtests.showSuccess=true
> {noformat}
> Then you'll hit several failures in TestPayloadNearQuery, eg:
> {noformat}
> Suite: org.apache.lucene.search.payloads.TestPayloadNearQuery
> 1> FAILED
> 2> NOTE: reproduce with: ant test -Dtestcase=TestPayloadNearQuery
> -Dtests.method=test -Dtests.seed=C3802435F5FB39D0 -Dtests.slow=true
> -Dtests.locale=ga -Dtests.timezone=America/Adak -Dtests.file.encoding=US-ASCII
> ERROR 0.01s | TestPayloadNearQuery.test <<<
> > Throwable #1: java.lang.RuntimeException: overridden idfExplain method
> in TestPayloadNearQuery.BoostingSimilarity was not called
> > at
> __randomizedtesting.SeedInfo.seed([C3802435F5FB39D0:4BD41BEF5B075428]:0)
> > at
> org.apache.lucene.search.similarities.TFIDFSimilarity.computeWeight(TFIDFSimilarity.java:740)
> > at org.apache.lucene.search.spans.SpanWeight.<init>(SpanWeight.java:62)
> > at
> org.apache.lucene.search.payloads.PayloadNearQuery$PayloadNearSpanWeight.<init>(PayloadNearQuery.java:147)
> > at
> org.apache.lucene.search.payloads.PayloadNearQuery.createWeight(PayloadNearQuery.java:75)
> > at
> org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:648)
> > at
> org.apache.lucene.search.AssertingIndexSearcher.createNormalizedWeight(AssertingIndexSearcher.java:60)
> > at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:265)
> > at
> org.apache.lucene.search.payloads.TestPayloadNearQuery.test(TestPayloadNearQuery.java:146)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> > at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> > at
> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> > at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> > at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> > at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
> > at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> > at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> > at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> > at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> > at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> > at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> > at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> > at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> > at java.lang.Thread.run(Thread.java:661)
> {noformat}
> The patch at least isolates the JVM bug even if it's not exactly a
> minimal test :) Somehow the idfExplain method, which
> is overridden in this test's BoostingSimilarity, fails to be called
> (the super.idfExplain is called instead), which leads to the test
> failures.
> The failure does not happen if you run this test in isolation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
