Re: Exceptions in TestConcurrentMergeScheduler
"Chris Hostetter" <[EMAIL PROTECTED]> wrote: > > : But it'd be nice to do this across the board, ie, for any junit test > : if one of CMS's threads (or, threads launched elsewhere) hits an > : unhandled exception, fail the testcase that's currently running. > : I'll dig and see if there's some central way to do this with junit... > > FYI: i did some casual investigation of this and the only thing that > jumped out at me is the static > Thread.setDefaultUncaughtExceptionHandler(...) added in 1.5. for 1.4 > there doesn't seem to be a generic way to notice an uncaught exception > from any thread. Thanks Hoss. Catching the exception is actually not the hard part because I "own" all the threads spawned by ConcurrentMergeScheduler. What's tricky is finding a way to force the currently running JUnit testcase or suite to fail. I'm digging through JUnit and ant's JUnitTestRunner sources to see if there's some hook somewhere where we could insert a check, just before the suite finishes, to assert that no exceptions were hit. Or, if I can somehow "look up" the current Test that's running, I could add an error to it. If there are any JUnit and ant experts out there (I'm not!) please chime in! Mike - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Exceptions in TestConcurrentMergeScheduler
OK, I think I found one possibility here. With ant's junit task, you
can define a custom formatter implementing this interface:
org.apache.tools.ant.taskdefs.optional.junit.JUnitResultFormatter
That interface has a method endTestSuite that is invoked once at the
end of all the test cases. So I can define a customer formatter, and
in this method, I can check with ConcurrentMergeScheduler and if any
unhandled exceptions has occurred, I can throw an exception and the
suite/testcase is marked as failed. It seems to work.
This is a nice solution in that we don't have to modify every unit
test to do its own checking. However, it's not really a "normal" use
case because formatters are supposed to just "format" the test result
output. It also adds a dependence from Lucene's unit test sources to
ant. But at least it does work ("progress not perfection").
And objections to this approach? Is there a better approach?
Mike
"Michael McCandless" <[EMAIL PROTECTED]> wrote:
>
> "Chris Hostetter" <[EMAIL PROTECTED]> wrote:
> >
> > : But it'd be nice to do this across the board, ie, for any junit test
> > : if one of CMS's threads (or, threads launched elsewhere) hits an
> > : unhandled exception, fail the testcase that's currently running.
> > : I'll dig and see if there's some central way to do this with junit...
> >
> > FYI: i did some casual investigation of this and the only thing that
> > jumped out at me is the static
> > Thread.setDefaultUncaughtExceptionHandler(...) added in 1.5. for 1.4
> > there doesn't seem to be a generic way to notice an uncaught exception
> > from any thread.
>
> Thanks Hoss.
>
> Catching the exception is actually not the hard part because I "own"
> all the threads spawned by ConcurrentMergeScheduler. What's tricky is
> finding a way to force the currently running JUnit testcase or suite
> to fail. I'm digging through JUnit and ant's JUnitTestRunner sources
> to see if there's some hook somewhere where we could insert a check,
> just before the suite finishes, to assert that no exceptions were hit.
> Or, if I can somehow "look up" the current Test that's running, I
> could add an error to it.
>
> If there are any JUnit and ant experts out there (I'm not!) please
> chime in!
>
> Mike
>
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Exceptions in TestConcurrentMergeScheduler
Mike,
Would it work to have a common LuceneTestCase base class that could
do that check and fail() in tearDown?
Erik
On Oct 4, 2007, at 5:31 AM, Michael McCandless wrote:
OK, I think I found one possibility here. With ant's junit task, you
can define a custom formatter implementing this interface:
org.apache.tools.ant.taskdefs.optional.junit.JUnitResultFormatter
That interface has a method endTestSuite that is invoked once at the
end of all the test cases. So I can define a customer formatter, and
in this method, I can check with ConcurrentMergeScheduler and if any
unhandled exceptions has occurred, I can throw an exception and the
suite/testcase is marked as failed. It seems to work.
This is a nice solution in that we don't have to modify every unit
test to do its own checking. However, it's not really a "normal" use
case because formatters are supposed to just "format" the test result
output. It also adds a dependence from Lucene's unit test sources to
ant. But at least it does work ("progress not perfection").
And objections to this approach? Is there a better approach?
Mike
"Michael McCandless" <[EMAIL PROTECTED]> wrote:
"Chris Hostetter" <[EMAIL PROTECTED]> wrote:
: But it'd be nice to do this across the board, ie, for any junit
test
: if one of CMS's threads (or, threads launched elsewhere) hits an
: unhandled exception, fail the testcase that's currently running.
: I'll dig and see if there's some central way to do this with
junit...
FYI: i did some casual investigation of this and the only thing that
jumped out at me is the static
Thread.setDefaultUncaughtExceptionHandler(...) added in 1.5. for
1.4
there doesn't seem to be a generic way to notice an uncaught
exception
from any thread.
Thanks Hoss.
Catching the exception is actually not the hard part because I "own"
all the threads spawned by ConcurrentMergeScheduler. What's
tricky is
finding a way to force the currently running JUnit testcase or suite
to fail. I'm digging through JUnit and ant's JUnitTestRunner sources
to see if there's some hook somewhere where we could insert a check,
just before the suite finishes, to assert that no exceptions were
hit.
Or, if I can somehow "look up" the current Test that's running, I
could add an error to it.
If there are any JUnit and ant experts out there (I'm not!) please
chime in!
Mike
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Exceptions in TestConcurrentMergeScheduler
Indeed this will work! I was unsure whether you could fail() a test
inside tearDown() but it seems to work fine -- I just tested it.
Only downside is a mass replacement of all "extends TestCase" with
"extends LuceneTestCase" (but that's a one time cost, now), and making
sure tearDown() always calls super.tearDown().
OK I like this solution better; I plan to go with that. I'll open a
separate issue for it. Thanks Erik :)
Mike
"Erik Hatcher" <[EMAIL PROTECTED]> wrote:
> Mike,
>
> Would it work to have a common LuceneTestCase base class that could
> do that check and fail() in tearDown?
>
> Erik
>
>
> On Oct 4, 2007, at 5:31 AM, Michael McCandless wrote:
>
> >
> > OK, I think I found one possibility here. With ant's junit task, you
> > can define a custom formatter implementing this interface:
> >
> > org.apache.tools.ant.taskdefs.optional.junit.JUnitResultFormatter
> >
> > That interface has a method endTestSuite that is invoked once at the
> > end of all the test cases. So I can define a customer formatter, and
> > in this method, I can check with ConcurrentMergeScheduler and if any
> > unhandled exceptions has occurred, I can throw an exception and the
> > suite/testcase is marked as failed. It seems to work.
> >
> > This is a nice solution in that we don't have to modify every unit
> > test to do its own checking. However, it's not really a "normal" use
> > case because formatters are supposed to just "format" the test result
> > output. It also adds a dependence from Lucene's unit test sources to
> > ant. But at least it does work ("progress not perfection").
> >
> > And objections to this approach? Is there a better approach?
> >
> > Mike
> >
> > "Michael McCandless" <[EMAIL PROTECTED]> wrote:
> >>
> >> "Chris Hostetter" <[EMAIL PROTECTED]> wrote:
> >>>
> >>> : But it'd be nice to do this across the board, ie, for any junit
> >>> test
> >>> : if one of CMS's threads (or, threads launched elsewhere) hits an
> >>> : unhandled exception, fail the testcase that's currently running.
> >>> : I'll dig and see if there's some central way to do this with
> >>> junit...
> >>>
> >>> FYI: i did some casual investigation of this and the only thing that
> >>> jumped out at me is the static
> >>> Thread.setDefaultUncaughtExceptionHandler(...) added in 1.5. for
> >>> 1.4
> >>> there doesn't seem to be a generic way to notice an uncaught
> >>> exception
> >>> from any thread.
> >>
> >> Thanks Hoss.
> >>
> >> Catching the exception is actually not the hard part because I "own"
> >> all the threads spawned by ConcurrentMergeScheduler. What's
> >> tricky is
> >> finding a way to force the currently running JUnit testcase or suite
> >> to fail. I'm digging through JUnit and ant's JUnitTestRunner sources
> >> to see if there's some hook somewhere where we could insert a check,
> >> just before the suite finishes, to assert that no exceptions were
> >> hit.
> >> Or, if I can somehow "look up" the current Test that's running, I
> >> could add an error to it.
> >>
> >> If there are any JUnit and ant experts out there (I'm not!) please
> >> chime in!
> >>
> >> Mike
> >>
> >> -
> >> To unsubscribe, e-mail: [EMAIL PROTECTED]
> >> For additional commands, e-mail: [EMAIL PROTECTED]
> >>
> >
> > -
> > To unsubscribe, e-mail: [EMAIL PROTECTED]
> > For additional commands, e-mail: [EMAIL PROTECTED]
>
>
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Created: (LUCENE-1018) intermittant exceptions in TestConcurrentMergeScheduler
intermittant exceptions in TestConcurrentMergeScheduler --- Key: LUCENE-1018 URL: https://issues.apache.org/jira/browse/LUCENE-1018 Project: Lucene - Java Issue Type: Bug Components: Index Affects Versions: 2.3 Reporter: Michael McCandless Assignee: Michael McCandless Priority: Minor Fix For: 2.3 Attachments: LUCENE-1018.patch The TestConcurrentMergeScheduler throws intermittant exceptions that do not result in a test failure. The exception happens in the "testNoWaitClose()" test, which repeated tests closing an IndexWriter with "false", meaning abort any still-running merges. When a merge is aborted it can hit various exceptions because the files it is reading and/or writing have been deleted, so we ignore these exceptions. The bug was just that we were failing to properly check whether the running merge was actually aborted because of a scoping issue of the "merge" variable in ConcurrentMergeScheduler. So the exceptions are actually "harmless". Thanks to Ning for spotting it! -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-1018) intermittant exceptions in TestConcurrentMergeScheduler
[ https://issues.apache.org/jira/browse/LUCENE-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-1018: --- Attachment: LUCENE-1018.patch Attached patch, fixing the scoping issue Ning found and also a few other small issues and adding more verbosity when infoStream is set. I plan to commit in a few days. > intermittant exceptions in TestConcurrentMergeScheduler > --- > > Key: LUCENE-1018 > URL: https://issues.apache.org/jira/browse/LUCENE-1018 > Project: Lucene - Java > Issue Type: Bug > Components: Index >Affects Versions: 2.3 >Reporter: Michael McCandless >Assignee: Michael McCandless >Priority: Minor > Fix For: 2.3 > > Attachments: LUCENE-1018.patch > > > The TestConcurrentMergeScheduler throws intermittant exceptions that > do not result in a test failure. > The exception happens in the "testNoWaitClose()" test, which repeated > tests closing an IndexWriter with "false", meaning abort any > still-running merges. When a merge is aborted it can hit various > exceptions because the files it is reading and/or writing have been > deleted, so we ignore these exceptions. > The bug was just that we were failing to properly check whether the > running merge was actually aborted because of a scoping issue of the > "merge" variable in ConcurrentMergeScheduler. So the exceptions are > actually "harmless". Thanks to Ning for spotting it! -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-1017) BoostingTermQuery performance
[ https://issues.apache.org/jira/browse/LUCENE-1017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Keegan updated LUCENE-1017: - Attachment: BoostingTermQuery.patch Here is new version that traverses the term positions properly, I believe. It's only about 5% faster than the spans version, though. It's quite possible that this could be improved. I'm using TermScorer to get me to the current doc and then setting TermPositions to the same doc. Do you see any inefficiencies with the 'next' and 'skipTo' calls, particularly in 'getPayloads'? > BoostingTermQuery performance > - > > Key: LUCENE-1017 > URL: https://issues.apache.org/jira/browse/LUCENE-1017 > Project: Lucene - Java > Issue Type: Improvement > Components: Search >Affects Versions: 2.2 > Environment: all >Reporter: Peter Keegan > Attachments: BoostingTermQuery.java, BoostingTermQuery.patch, > termquery.patch > > > I have been experimenting with payloads and BoostingTermQuery, which I think > are excellent additions to Lucene core. Currently, BoostingTermQuery extends > SpanQuery. I would suggest changing this class to extend TermQuery and > refactor the current version to something like 'BoostingSpanQuery'. > The reason is rooted in performance. In my testing, I compared query > throughput using TermQuery against 2 versions of BoostingTermQuery - the > current one that extends SpanQuery and one that extends TermQuery (which I've > included, below). Here are the results (qps = queries per second): > TermQuery:200 qps > BoostingTermQuery (extends SpanQuery): 97 qps > BoostingTermQuery (extends TermQuery): 130 qps > Here is a version of BoostingTermQuery that extends TermQuery. I had to > modify TermQuery and TermScorer to make them public. A code review would be > in order, and I would appreciate your comments on this suggestion. > Peter -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Created: (LUCENE-1019) CustomScoreQuery should support multiple ValueSourceQueries
CustomScoreQuery should support multiple ValueSourceQueries --- Key: LUCENE-1019 URL: https://issues.apache.org/jira/browse/LUCENE-1019 Project: Lucene - Java Issue Type: Improvement Components: Search Affects Versions: 2.2 Reporter: Kyle Maxwell CustomScoreQuery's constructor currently accepts a subQuery, and a ValueSourceQuery. I would like it to accept multiple ValueSourceQueries. The workaround of nested CustomScoreQueries works for simple cases, but it quickly becomes either cumbersome to manage, or impossible to implement the desired function. This patch implements CustomMultiScoreQuery with my desired functionality, and refactors CustomScoreQuery to implement the special case of a CustomMultiScoreQuery with 0 or 1 ValueSourceQueries. This keeps the CustomScoreQuery API intact. This patch includes basic tests, more or less taken from the original implementation, and customized a bit to cover the new cases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Issue Comment Edited: (LUCENE-1019) CustomScoreQuery should support multiple ValueSourceQueries
[ https://issues.apache.org/jira/browse/LUCENE-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12532557 ] fizx edited comment on LUCENE-1019 at 10/4/07 6:08 PM: --- Here's the patch! BTW, I'll edit the docs as soon as someone signs off that this is a good idea! was (Author: fizx): Here's the patch! > CustomScoreQuery should support multiple ValueSourceQueries > --- > > Key: LUCENE-1019 > URL: https://issues.apache.org/jira/browse/LUCENE-1019 > Project: Lucene - Java > Issue Type: Improvement > Components: Search >Affects Versions: 2.2 >Reporter: Kyle Maxwell > Attachments: CustomMultiQuery.v0.diff > > > CustomScoreQuery's constructor currently accepts a subQuery, and a > ValueSourceQuery. I would like it to accept multiple ValueSourceQueries. > The workaround of nested CustomScoreQueries works for simple cases, but it > quickly becomes either cumbersome to manage, or impossible to implement the > desired function. > This patch implements CustomMultiScoreQuery with my desired functionality, > and refactors CustomScoreQuery to implement the special case of a > CustomMultiScoreQuery with 0 or 1 ValueSourceQueries. This keeps the > CustomScoreQuery API intact. > This patch includes basic tests, more or less taken from the original > implementation, and customized a bit to cover the new cases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-1019) CustomScoreQuery should support multiple ValueSourceQueries
[ https://issues.apache.org/jira/browse/LUCENE-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Maxwell updated LUCENE-1019: - Attachment: CustomMultiQuery.v0.diff Here's the patch! > CustomScoreQuery should support multiple ValueSourceQueries > --- > > Key: LUCENE-1019 > URL: https://issues.apache.org/jira/browse/LUCENE-1019 > Project: Lucene - Java > Issue Type: Improvement > Components: Search >Affects Versions: 2.2 >Reporter: Kyle Maxwell > Attachments: CustomMultiQuery.v0.diff > > > CustomScoreQuery's constructor currently accepts a subQuery, and a > ValueSourceQuery. I would like it to accept multiple ValueSourceQueries. > The workaround of nested CustomScoreQueries works for simple cases, but it > quickly becomes either cumbersome to manage, or impossible to implement the > desired function. > This patch implements CustomMultiScoreQuery with my desired functionality, > and refactors CustomScoreQuery to implement the special case of a > CustomMultiScoreQuery with 0 or 1 ValueSourceQueries. This keeps the > CustomScoreQuery API intact. > This patch includes basic tests, more or less taken from the original > implementation, and customized a bit to cover the new cases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1019) CustomScoreQuery should support multiple ValueSourceQueries
[ https://issues.apache.org/jira/browse/LUCENE-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12532560 ] Hoss Man commented on LUCENE-1019: -- this class seems to assume that the ValueSourceQueries should be multipled ... but it would be just as easy to assume the should be added, or averaged. It seems like it might make more sense if instead of a CustomMultiScoreQuery there was just a "ProductValueSource" class that took in a ValueSource[] and multiplied them > CustomScoreQuery should support multiple ValueSourceQueries > --- > > Key: LUCENE-1019 > URL: https://issues.apache.org/jira/browse/LUCENE-1019 > Project: Lucene - Java > Issue Type: Improvement > Components: Search >Affects Versions: 2.2 >Reporter: Kyle Maxwell > Attachments: CustomMultiQuery.v0.diff > > > CustomScoreQuery's constructor currently accepts a subQuery, and a > ValueSourceQuery. I would like it to accept multiple ValueSourceQueries. > The workaround of nested CustomScoreQueries works for simple cases, but it > quickly becomes either cumbersome to manage, or impossible to implement the > desired function. > This patch implements CustomMultiScoreQuery with my desired functionality, > and refactors CustomScoreQuery to implement the special case of a > CustomMultiScoreQuery with 0 or 1 ValueSourceQueries. This keeps the > CustomScoreQuery API intact. > This patch includes basic tests, more or less taken from the original > implementation, and customized a bit to cover the new cases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: svn commit: r582054: typo
Hoss, Thanks for that. There is a typo, it says "llthough" in one of the javadocs. Regards, Paul Elschot - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
