[jira] [Updated] (SOLR-2548) Multithreaded faceting
[ https://issues.apache.org/jira/browse/SOLR-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley updated SOLR-2548: --- Attachment: SOLR-2548_multithreaded_faceting,_dsmiley.patch The attached patch improves on my previous one a little -- a few more comments, a variable rename for clarity, an assertion. And of course I removed the future.cancel() loop. I think this code is pretty clear as far as multithreaded code goes: One loop that submits tasks, and a follow-on loop that consumes the results of those tasks, and a semaphore to ensure no more than the desired number of threads are computing the facets. It'd be cool to eventually extend multithreading across all the faceting types. I'll look into that next week. Multithreaded faceting -- Key: SOLR-2548 URL: https://issues.apache.org/jira/browse/SOLR-2548 Project: Solr Issue Type: Improvement Components: search Affects Versions: 3.1 Reporter: Janne Majaranta Assignee: Erick Erickson Priority: Minor Labels: facet Fix For: 4.5, 5.0 Attachments: SOLR-2548_4.2.1.patch, SOLR-2548_for_31x.patch, SOLR-2548_multithreaded_faceting,_dsmiley.patch, SOLR-2548_multithreaded_faceting,_dsmiley.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch Add multithreading support for faceting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2548) Multithreaded faceting
[ https://issues.apache.org/jira/browse/SOLR-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley updated SOLR-2548: --- Attachment: SOLR-2548_multithreaded_faceting,_dsmiley.patch This issue just got on my radar; I like working on threading problems. I commend the progress made but I think it can be improved: # I think it's counter-intuitive that if a user supplies facet.threads=2 then 3 cpu cores will actually be used (assuming 2 fields to facet on) # Only the first facet.threads worth of facets are actually done concurrently; the rest are done serially. # Even if the previous problem was solved, the use of the main calling thread to compute facets (beyond facet.threads) means that if by bad luck the main thread is computing the most intensive facets to compute, the other threads will sit idle once they are done when it would be better to have remaining work queued up. # in the event of an exception in one worker; the rest should be cancelled # ExecutionException is a wrapping exception; you should unwrap it and wrap SolrException on its contents, not the ExecutionException itself. The attached patch fixes all these problems, keeps it no more complex and perhaps simpler (IMO), and without increasing the lines-of-code count. Multithreaded faceting -- Key: SOLR-2548 URL: https://issues.apache.org/jira/browse/SOLR-2548 Project: Solr Issue Type: Improvement Components: search Affects Versions: 3.1 Reporter: Janne Majaranta Assignee: Erick Erickson Priority: Minor Labels: facet Fix For: 4.5, 5.0 Attachments: SOLR-2548_4.2.1.patch, SOLR-2548_for_31x.patch, SOLR-2548_multithreaded_faceting,_dsmiley.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch Add multithreading support for faceting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2548) Multithreaded faceting
[ https://issues.apache.org/jira/browse/SOLR-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson updated SOLR-2548: - Attachment: SOLR-2548.patch Final patch, including CHANGES.txt entry. Multithreaded faceting -- Key: SOLR-2548 URL: https://issues.apache.org/jira/browse/SOLR-2548 Project: Solr Issue Type: Improvement Components: search Affects Versions: 3.1 Reporter: Janne Majaranta Assignee: Erick Erickson Priority: Minor Labels: facet Attachments: SOLR-2548_4.2.1.patch, SOLR-2548_for_31x.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch Add multithreading support for faceting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: [jira] [Updated] (SOLR-2548) Multithreaded faceting
OK, got a clean 'ant test'. So I think this is good to go. Big thanks Yonik! Every time I get back into multithreading stuff after not doing it for a while I have to relearn stuff it seems. I'm not entirely enthused about the new c'tor for UnInvertedField, but it's private so at least it's an implementation detail that others won't be tempted to use. On Thu, Sep 5, 2013 at 1:04 PM, Erick Erickson (JIRA) j...@apache.orgwrote: [ https://issues.apache.org/jira/browse/SOLR-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel] Erick Erickson updated SOLR-2548: - Attachment: SOLR-2548.patch bq: See FieldCacheImpl.get() - there is a CreationPlaceHolder object used. Right, but the map that that's placed _in_ is a mapkey, Object. The UnInvertedField cache is mapkey, UnInvertedField. That's what was behind my question about making a dummy UnInvertedField to use similarly to how CreationPlaceHolder is used. I'm really not up for making the underlying UnInvertedField cache take an Object, seems like the tail wagging the dog. Interestingly I think it was what [~yo...@apache.org] pointed out about synching on different objects than I thought I was that was the problem with TestDistributedGrouping, it passes now. The attached patch implements creating a placeholder UnInvertedField, removes the pending map and passes the failing test from yesterday as well as precommit. I'll run the full suite soon. If that passes, I'll let it stew for a bit and commit tomorrow unless there are objections. Multithreaded faceting -- Key: SOLR-2548 URL: https://issues.apache.org/jira/browse/SOLR-2548 Project: Solr Issue Type: Improvement Components: search Affects Versions: 3.1 Reporter: Janne Majaranta Assignee: Erick Erickson Priority: Minor Labels: facet Attachments: SOLR-2548_4.2.1.patch, SOLR-2548_for_31x.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch Add multithreading support for faceting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2548) Multithreaded faceting
[ https://issues.apache.org/jira/browse/SOLR-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson updated SOLR-2548: - Attachment: SOLR-2548.patch bq: See FieldCacheImpl.get() - there is a CreationPlaceHolder object used. Right, but the map that that's placed _in_ is a mapkey, Object. The UnInvertedField cache is mapkey, UnInvertedField. That's what was behind my question about making a dummy UnInvertedField to use similarly to how CreationPlaceHolder is used. I'm really not up for making the underlying UnInvertedField cache take an Object, seems like the tail wagging the dog. Interestingly I think it was what [~yo...@apache.org] pointed out about synching on different objects than I thought I was that was the problem with TestDistributedGrouping, it passes now. The attached patch implements creating a placeholder UnInvertedField, removes the pending map and passes the failing test from yesterday as well as precommit. I'll run the full suite soon. If that passes, I'll let it stew for a bit and commit tomorrow unless there are objections. Multithreaded faceting -- Key: SOLR-2548 URL: https://issues.apache.org/jira/browse/SOLR-2548 Project: Solr Issue Type: Improvement Components: search Affects Versions: 3.1 Reporter: Janne Majaranta Assignee: Erick Erickson Priority: Minor Labels: facet Attachments: SOLR-2548_4.2.1.patch, SOLR-2548_for_31x.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch Add multithreading support for faceting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2548) Multithreaded faceting
[ https://issues.apache.org/jira/browse/SOLR-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson updated SOLR-2548: - Attachment: SOLR-2548.patch Hmm, the whole recording-thread-info is a little more ambitious than I want to be right now. For the nonce, I did some by hand debugging, added in a couple of (temporary) print message in the getUnInvertedField code and insured that when it's called it only executes once per field, so I think I'll call that good now. I did play around with the directExcecutor and now I get to add another bit of knowledge, that it's really kind of cool that it allows one to have code like this. No matter how many times you submit a job, it all just executes in the current thread. Arcane, but kind of cool. As for the rest, I've added at least functional tests and one test that the caching code is working that's non-deterministic but might trip bad conditions at least some of the time. So unless people object I'll be committing this probably tomorrow. It passes precommit and at least all the tests in TestFaceting, I'll be running the full suite in a minute. Multithreaded faceting -- Key: SOLR-2548 URL: https://issues.apache.org/jira/browse/SOLR-2548 Project: Solr Issue Type: Improvement Components: search Affects Versions: 3.1 Reporter: Janne Majaranta Assignee: Erick Erickson Priority: Minor Labels: facet Attachments: SOLR-2548_4.2.1.patch, SOLR-2548_for_31x.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch Add multithreading support for faceting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2548) Multithreaded faceting
[ https://issues.apache.org/jira/browse/SOLR-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson updated SOLR-2548: - Attachment: SOLR-2548.patch OK, maybe this time. 1 put back the passing in base. 2 took out the sleep. 3 changed how exceptions are propagated up past the new threads which fixed another test that this code broke. 4 Added a non-deterministic test that forces parallel uninverting of the fields to make sure we exercise the synchronize/notify code. This test can't _guarantee_ to execute that code every time, but it did manage with some printlns. Running tests again, precommit all that. Won't check in until at least tomorrow. And thank heaven for local history in IntelliJ ;) Multithreaded faceting -- Key: SOLR-2548 URL: https://issues.apache.org/jira/browse/SOLR-2548 Project: Solr Issue Type: Improvement Components: search Affects Versions: 3.1 Reporter: Janne Majaranta Assignee: Erick Erickson Priority: Minor Labels: facet Attachments: SOLR-2548_4.2.1.patch, SOLR-2548_for_31x.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch, SOLR-2548.patch Add multithreading support for faceting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2548) Multithreaded faceting
[ https://issues.apache.org/jira/browse/SOLR-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson updated SOLR-2548: - Attachment: SOLR-2548.patch Latest version that does two things: 1 does the max thread change I commented on earlier. 2 puts in some checking to insure that if multiple threads try to uninvert the same field at the same time, it'll only be loaded once. I used a simple wait/sleep loop here since this method is called from several places and it looks like a real pain to try to do a Future or whatever. Multithreaded faceting -- Key: SOLR-2548 URL: https://issues.apache.org/jira/browse/SOLR-2548 Project: Solr Issue Type: Improvement Components: search Affects Versions: 3.1 Reporter: Janne Majaranta Assignee: Erick Erickson Priority: Minor Labels: facet Attachments: SOLR-2548_4.2.1.patch, SOLR-2548_for_31x.patch, SOLR-2548.patch, SOLR-2548.patch Add multithreading support for faceting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2548) Multithreaded faceting
[ https://issues.apache.org/jira/browse/SOLR-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gun Akkor updated SOLR-2548: Attachment: SOLR-2548_4.2.1.patch Patch against 4.2.1 Multithreaded faceting -- Key: SOLR-2548 URL: https://issues.apache.org/jira/browse/SOLR-2548 Project: Solr Issue Type: Improvement Components: search Affects Versions: 3.1 Reporter: Janne Majaranta Priority: Minor Labels: facet Attachments: SOLR-2548_4.2.1.patch, SOLR-2548_for_31x.patch, SOLR-2548.patch Add multithreading support for faceting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2548) Multithreaded faceting
[ https://issues.apache.org/jira/browse/SOLR-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janne Majaranta updated SOLR-2548: -- Attachment: SOLR-2548_for_31x.patch Patch for 3.1 Multithreaded faceting -- Key: SOLR-2548 URL: https://issues.apache.org/jira/browse/SOLR-2548 Project: Solr Issue Type: Improvement Components: search Affects Versions: 3.1 Reporter: Janne Majaranta Priority: Minor Labels: facet Attachments: SOLR-2548.patch, SOLR-2548_for_31x.patch Add multithreading support for faceting. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
