[
https://issues.apache.org/jira/browse/LUCENE-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-1709:
---------------------------------------
Attachment: runLuceneTests.py
I'm attaching the scary python script that I use to use multiple
threads when running the tests.
It's not really generic at all. It's got hardwired paths to my home
dir, it symlinks build/test to /tmp (= tmpfs on my solaris box), it
applies a scary patch to the build xml files (and that patch is
depending on the branch -- only works on 2.9, 3.0, trunk, flex now).
Often the patch fails to apply (as we change the build xml files) so I
have to go and redo them.
Sometimes tests have false positive failure because they temporarily
fill up the tmpfs; if you ctrl+C the test it may leave turd processes
running; etc.
So I wouldn't recommend using this as is.... but maybe someone who is
good-with-the-ant can take the general idea here and make it work more
generally with only ant.
Roughly all that I do is have each thread run its own -Dtestpackage.
The existing search & index packages are too big, so I split them into
2. I also roughly ordered the tests so that they are "balanced", so
that as each thread pulls from the work queue, the "roughly" fininish
at the same time. The test-contrib is run by a single thread.
With all the speedups, this is what the output looks like:
{noformat}
[TRUNK]
0 [0:00:00.310867]: run "ant compile-backwards compile-core compile-demo
jar-core compile-test build-contrib"...
0 [0:00:30.612062]: run "ant test-contrib"...
1 [0:00:30.617522]: run "ant test-core -Dtestpackagea=index"...
2 [0:00:30.623840]: run "ant test-core -Dtestpackageb=index"...
3 [0:00:30.630072]: run "ant test-backwards -Dtestpackagea=index"...
4 [0:00:30.637612]: run "ant test-backwards -Dtestpackageb=index"...
5 [0:00:30.645334]: run "ant test-core -Dtestpackagea=search"...
4 [0:00:58.937803]: run "ant test-core -Dtestpackageb=search"...
2 [0:00:59.052593]: run "ant test-backwards -Dtestpackagea=search"...
5 [0:01:31.313156]: run "ant test-backwards -Dtestpackageb=search"...
2 [0:01:52.617814]: run "ant test-core -Dtestpackage=store"...
4 [0:02:01.698477]: run "ant test-backwards -Dtestpackage=store"...
2 [0:02:10.818047]: run "ant test-core -Dtestpackage=analysis"...
3 [0:02:14.808217]: run "ant test-backwards -Dtestpackage=analysis"...
1 [0:02:15.786972]: run "ant test-core -Dtestpackageroot=lucene"...
4 [0:02:21.937936]: run "ant test-backwards -Dtestpackageroot=lucene"...
1 [0:02:25.898000]: run "ant test-core -Dtestpackage=util"...
2 [0:02:31.037923]: run "ant test-backwards -Dtestpackage=util"...
4 [0:02:31.038090]: run "ant test-core -Dtestpackage=document"...
3 [0:02:32.007975]: run "ant test-backwards -Dtestpackage=document"...
5 [0:02:38.017968]: run "ant test-core -Dtestpackage=queryParser"...
3 [0:02:40.097929]: run "ant test-backwards -Dtestpackage=queryParser"...
4 [0:02:40.151166]: DONE
0 [0:02:41.927786]: DONE
1 [0:02:43.077980]: DONE
2 [0:02:46.198287]: DONE
5 [0:02:49.168172]: DONE
3 [0:02:50.197936]: DONE
DONE: took 0:02:51.046643 [528 tests]
{noformat}
> Parallelize Tests
> -----------------
>
> Key: LUCENE-1709
> URL: https://issues.apache.org/jira/browse/LUCENE-1709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Fix For: 3.1
>
> Attachments: runLuceneTests.py
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The Lucene tests can be parallelized to make for a faster testing system.
> This task from ANT can be used:
> http://ant.apache.org/manual/CoreTasks/parallel.html
> Previous discussion:
> http://www.gossamer-threads.com/lists/lucene/java-dev/69669
> Notes from Mike M.:
> {quote}
> I'd love to see a clean solution here (the tests are embarrassingly
> parallelizable, and we all have machines with good concurrency these
> days)... I have a rather hacked up solution now, that uses
> "-Dtestpackage=XXX" to split the tests up.
> Ideally I would be able to say "use N threads" and it'd do the right
> thing... like the -j flag to make.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
