RE: How do I best detect when my DIH load is done?
Andy, I use an approach similar to yours. There may be something better, however. You might be able to write an onImportEnd listener to tell you when it ends. See http://wiki.apache.org/solr/DataImportHandler#EventListeners for a little documentation See also https://issues.apache.org/jira/browse/SOLR-938 and https://issues.apache.org/jira/browse/SOLR-1081 for the background on this feature. If you do end up using this let us know how it works and if there is anything you could see to improve it. James Dyer E-Commerce Systems Ingram Content Group (615) 213-4311 -Original Message- From: Andy Lester [mailto:a...@petdance.com] Sent: Monday, November 19, 2012 10:29 AM To: solr-user@lucene.apache.org Subject: How do I best detect when my DIH load is done? A little while back, I needed a way to tell if my DIH load was done, so I made up a little Ruby program to query /dih?command=status . The program is here: http://petdance.com/2012/07/a-little-ruby-program-to-monitor-solr-dih-imports/ Is this the best way to do it? Is there some other tool or interface that I should be using instead? Thanks, xoa -- Andy Lester = a...@petdance.com = www.petdance.com = AIM:petdance
Re: How do I best detect when my DIH load is done?
On 11/19/2012 11:52 AM, Dyer, James wrote: Andy, I use an approach similar to yours. There may be something better, however. You might be able to write an onImportEnd listener to tell you when it ends. See http://wiki.apache.org/solr/DataImportHandler#EventListeners for a little documentation See also https://issues.apache.org/jira/browse/SOLR-938 and https://issues.apache.org/jira/browse/SOLR-1081 for the background on this feature. If you do end up using this let us know how it works and if there is anything you could see to improve it. I think it would be a good idea to provide a SolrJ API out of the box (similar to CoreAdminRequest) for gathering the status URL from Solr and obtaining the following information: 1) Determining import status -a) never started (idle) -b) finished successful (idle) -c) finished with error, canceled, etc. (idle) -d) in progress. (busy) 2) Determining how many documents have been added. 3) Determining how long the import took or has taken so far. 4) Any other commonly gathered information. There may be some reluctance to do this simply because DIH is a contrib module. Perhaps there could be a contrib module for SolrJ? Thanks, Shawn
Re: How do I best detect when my DIH load is done?
Hello Andy, i had a similar question on this some time ago. http://lucene.472066.n3.nabble.com/possible-status-codes-from-solr-during-a-DIH-data-import-process-td3987110.html#a3987123 http://lucene.472066.n3.nabble.com/need-input-lessons-learned-or-best-practices-for-data-imports-td3801327.html#a3803658 i ended up writing my own shell based polling application that runs from our *nx batch server that handles all of our Control-M work. +1 on the idea of making this a more formal part of the API. let me know if you want concrete example code. -- View this message in context: http://lucene.472066.n3.nabble.com/How-do-I-best-detect-when-my-DIH-load-is-done-tp4021121p4021148.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: How do I best detect when my DIH load is done?
James, was it you (cannot remember) that replied to one of my queries on this subject and mentioned that there was consideration being given to cleaning up the response codes to remove ambiguity? -- View this message in context: http://lucene.472066.n3.nabble.com/How-do-I-best-detect-when-my-DIH-load-is-done-tp4021121p4021150.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: How do I best detect when my DIH load is done?
I'm not sure. But there are at least a few jira issues open with differing ideas on how to improve this. For instance, SOLR-1554 SOLR-2728 SOLR-2729 James Dyer E-Commerce Systems Ingram Content Group (615) 213-4311 -Original Message- From: geeky2 [mailto:gee...@hotmail.com] Sent: Monday, November 19, 2012 1:52 PM To: solr-user@lucene.apache.org Subject: RE: How do I best detect when my DIH load is done? James, was it you (cannot remember) that replied to one of my queries on this subject and mentioned that there was consideration being given to cleaning up the response codes to remove ambiguity? -- View this message in context: http://lucene.472066.n3.nabble.com/How-do-I-best-detect-when-my-DIH-load-is-done-tp4021121p4021150.html Sent from the Solr - User mailing list archive at Nabble.com.
