Re: How to override rollback behavior in DIH
Following up on this a bit - my main index is updated by a SolrJ client in another process. If the DIH fails, the SolrJ client is never informed of the index rollback, and any pending updates are lost. For now, I've made sure that the DIH processor never throws an exception, but this makes it a bit harder to detect the failure via the admin interface. Thanks, Peter On Tue, Jan 14, 2014 at 11:12 AM, Peter Keegan peterlkee...@gmail.comwrote: I have a custom data import handler that creates an ExternalFileField from a source that is different from the main index. If the import fails (in my case, a connection refused in URLDataSource), I don't want to roll back any uncommitted changes to the main index. However, this seems to be the default behavior. Is there a way to override the IndexWriter rollback? Thanks, Peter
Re: How to override rollback behavior in DIH
Can you try using onError=skip on your entities which use this data source? It's been some time since I looked at the code so I don't know if this works with data source. Worth a try I guess. On Fri, Jan 17, 2014 at 7:20 PM, Peter Keegan peterlkee...@gmail.com wrote: Following up on this a bit - my main index is updated by a SolrJ client in another process. If the DIH fails, the SolrJ client is never informed of the index rollback, and any pending updates are lost. For now, I've made sure that the DIH processor never throws an exception, but this makes it a bit harder to detect the failure via the admin interface. Thanks, Peter On Tue, Jan 14, 2014 at 11:12 AM, Peter Keegan peterlkee...@gmail.comwrote: I have a custom data import handler that creates an ExternalFileField from a source that is different from the main index. If the import fails (in my case, a connection refused in URLDataSource), I don't want to roll back any uncommitted changes to the main index. However, this seems to be the default behavior. Is there a way to override the IndexWriter rollback? Thanks, Peter -- Regards, Shalin Shekhar Mangar.
Re: How to override rollback behavior in DIH
I'm actually doing the 'skip' on every successful call to 'nextRow' with this trick: row.put($externalfield,null); // DocBuilder.addFields will skip fields starting with '$' because I'm only creating ExternalFieldFields. However, an error could also occur in the 'init' call, so exceptions have to be caught there, too. Thanks, Peter On Fri, Jan 17, 2014 at 10:19 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: Can you try using onError=skip on your entities which use this data source? It's been some time since I looked at the code so I don't know if this works with data source. Worth a try I guess. On Fri, Jan 17, 2014 at 7:20 PM, Peter Keegan peterlkee...@gmail.com wrote: Following up on this a bit - my main index is updated by a SolrJ client in another process. If the DIH fails, the SolrJ client is never informed of the index rollback, and any pending updates are lost. For now, I've made sure that the DIH processor never throws an exception, but this makes it a bit harder to detect the failure via the admin interface. Thanks, Peter On Tue, Jan 14, 2014 at 11:12 AM, Peter Keegan peterlkee...@gmail.com wrote: I have a custom data import handler that creates an ExternalFileField from a source that is different from the main index. If the import fails (in my case, a connection refused in URLDataSource), I don't want to roll back any uncommitted changes to the main index. However, this seems to be the default behavior. Is there a way to override the IndexWriter rollback? Thanks, Peter -- Regards, Shalin Shekhar Mangar.
RE: How to override rollback behavior in DIH
Peter, I think you can override org.apache.solr.handler.dataimport.SolrWriter to have a custom (no-op) rollback method. Your new writer should implement org.apache.solr.handler.dataimport.DIHWriter. You can specify the writerImpl request parameter to specify the new class. Unfortunately, it isn't actually this easy because your new writer is going to have to know what to do for all the other methods. That is, there is no easy way to tell it how to write/commit/etc to Solr. The default SolrWriter has a lot of hardcoded parameters it gets sent on construction in DataImportHandler#handleRequestBody. You would have to somehow duplicate this construction on your own custom class. See SOLR-3671 for an explanation of this dilemma. James Dyer Ingram Content Group (615) 213-4311 -Original Message- From: pkeegan01...@gmail.com [mailto:pkeegan01...@gmail.com] On Behalf Of Peter Keegan Sent: Friday, January 17, 2014 7:51 AM To: solr-user@lucene.apache.org Subject: Re: How to override rollback behavior in DIH Following up on this a bit - my main index is updated by a SolrJ client in another process. If the DIH fails, the SolrJ client is never informed of the index rollback, and any pending updates are lost. For now, I've made sure that the DIH processor never throws an exception, but this makes it a bit harder to detect the failure via the admin interface. Thanks, Peter On Tue, Jan 14, 2014 at 11:12 AM, Peter Keegan peterlkee...@gmail.comwrote: I have a custom data import handler that creates an ExternalFileField from a source that is different from the main index. If the import fails (in my case, a connection refused in URLDataSource), I don't want to roll back any uncommitted changes to the main index. However, this seems to be the default behavior. Is there a way to override the IndexWriter rollback? Thanks, Peter
Re: How to override rollback behavior in DIH
Hmm, this does get a bit complicated, and I'm not even doing any writes with the DIH SolrWriter. In retrospect, using a DIH to create only EFFs doesn't buy much except for the integration into the Solr Admin UI. Thanks for the pointer to 3671, James. Peter On Fri, Jan 17, 2014 at 10:59 AM, Dyer, James james.d...@ingramcontent.comwrote: Peter, I think you can override org.apache.solr.handler.dataimport.SolrWriter to have a custom (no-op) rollback method. Your new writer should implement org.apache.solr.handler.dataimport.DIHWriter. You can specify the writerImpl request parameter to specify the new class. Unfortunately, it isn't actually this easy because your new writer is going to have to know what to do for all the other methods. That is, there is no easy way to tell it how to write/commit/etc to Solr. The default SolrWriter has a lot of hardcoded parameters it gets sent on construction in DataImportHandler#handleRequestBody. You would have to somehow duplicate this construction on your own custom class. See SOLR-3671 for an explanation of this dilemma. James Dyer Ingram Content Group (615) 213-4311 -Original Message- From: pkeegan01...@gmail.com [mailto:pkeegan01...@gmail.com] On Behalf Of Peter Keegan Sent: Friday, January 17, 2014 7:51 AM To: solr-user@lucene.apache.org Subject: Re: How to override rollback behavior in DIH Following up on this a bit - my main index is updated by a SolrJ client in another process. If the DIH fails, the SolrJ client is never informed of the index rollback, and any pending updates are lost. For now, I've made sure that the DIH processor never throws an exception, but this makes it a bit harder to detect the failure via the admin interface. Thanks, Peter On Tue, Jan 14, 2014 at 11:12 AM, Peter Keegan peterlkee...@gmail.com wrote: I have a custom data import handler that creates an ExternalFileField from a source that is different from the main index. If the import fails (in my case, a connection refused in URLDataSource), I don't want to roll back any uncommitted changes to the main index. However, this seems to be the default behavior. Is there a way to override the IndexWriter rollback? Thanks, Peter
