[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions
[ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15728020#comment-15728020 ] Mark Miller commented on SOLR-3284: --- You can use those hacks for specific use cases, but the only great solution for the general user client is really doing the work of efficiently returning error information for what could be tons of failed updates. It's not a bad idea to offer the option of trying to quit on the first error. I'd make it a required construction param. Most users that I've seen that want to do this though, want to count on updates stopping after the first fail, so you can reason about how to handle the situation reasonably, but you can actually end up with a few updates beyond that in, so it's not as great as it sounds even when you do want that kind of behavior. > StreamingUpdateSolrServer swallows exceptions > - > > Key: SOLR-3284 > URL: https://issues.apache.org/jira/browse/SOLR-3284 > Project: Solr > Issue Type: Improvement > Components: clients - java >Affects Versions: 3.5, 4.0-ALPHA >Reporter: Shawn Heisey >Assignee: Shawn Heisey > Attachments: SOLR-3284.patch > > > StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as > HttpClient, when doing adds. It may happen with other methods, though I know > that query and deleteByQuery will throw exceptions. I believe that this is a > result of the queue/Runner design. That's what makes SUSS perform better, > but it means you sacrifice the ability to programmatically determine that > there was a problem with your update. All errors are logged via slf4j, but > that's not terribly helpful except with determining what went wrong after the > fact. > When using CommonsHttpSolrServer, I've been able to rely on getting an > exception thrown by pretty much any error, letting me use try/catch to detect > problems. > There's probably enough dependent code out there that it would not be a good > idea to change the design of SUSS, unless there were alternate constructors > or additional methods available to configure new/old behavior. Fixing this > is probably not trivial, so it's probably a better idea to come up with a new > server object based on CHSS. This is outside my current skillset. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions
[ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705521#comment-15705521 ] David Smiley commented on SOLR-3284: It's telling that nowadays, internally in Solr there are actually 2 subclasses of ConcurrentUpdateSolrClient who both exist to handle errors: SafeConcurrentUpdateSolrClient (in morphlines) and ErrorReportingConcurrentUpdateSolrClient (defined within and returned by StreamingSolrClients, used by SolrCmdDistributor used by DistributedUpdateProcessor). And note also that all constructors are of CUSC are deprecated even though the only way to actually implement handleError by a user implies you need to use them since the Builder doesn't expose configuring error handling. IMO the default behavior of CUSC should be basically just like SafeConcurrentUpdateSolrClient so that users see problems readily. Perhaps also check & throw in request(). StreamingSolrClients can override to do it's special handling. > StreamingUpdateSolrServer swallows exceptions > - > > Key: SOLR-3284 > URL: https://issues.apache.org/jira/browse/SOLR-3284 > Project: Solr > Issue Type: Improvement > Components: clients - java >Affects Versions: 3.5, 4.0-ALPHA >Reporter: Shawn Heisey >Assignee: Shawn Heisey > Attachments: SOLR-3284.patch > > > StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as > HttpClient, when doing adds. It may happen with other methods, though I know > that query and deleteByQuery will throw exceptions. I believe that this is a > result of the queue/Runner design. That's what makes SUSS perform better, > but it means you sacrifice the ability to programmatically determine that > there was a problem with your update. All errors are logged via slf4j, but > that's not terribly helpful except with determining what went wrong after the > fact. > When using CommonsHttpSolrServer, I've been able to rely on getting an > exception thrown by pretty much any error, letting me use try/catch to detect > problems. > There's probably enough dependent code out there that it would not be a good > idea to change the design of SUSS, unless there were alternate constructors > or additional methods available to configure new/old behavior. Fixing this > is probably not trivial, so it's probably a better idea to come up with a new > server object based on CHSS. This is outside my current skillset. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions
[ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995131#comment-13995131 ] Stewart Sims commented on SOLR-3284: This has been a source of errors that were difficult for us to track down, when using the ConcurrentUpdateSolrServer to index a large volume of data (including some very large individual documents). We were confused for a long time as to why some documents were not being indexed, which turned out to be a combination of data errors and bad request errors due to too many concurrent requests. Once we switched to HttpSolrServer the problems with bad requests went away and we were able to get more informative exceptions which helped to find the data problems. Had my colleague not found the thread below, we would have struggled to figure out the exact causes of our problems: http://lucene.472066.n3.nabble.com/Missing-documents-with-ConcurrentUpdateSolrServer-vs-HttpSolrServer-td4033637.html One alternative to a code change might be to advise caution with using the ConcurrentUpdateSolrServer with large volumes of data. HttpSolrServer seems for us (using Solr 4.2.1) to be more stable and only takes about 30% more time to index. > StreamingUpdateSolrServer swallows exceptions > - > > Key: SOLR-3284 > URL: https://issues.apache.org/jira/browse/SOLR-3284 > Project: Solr > Issue Type: Improvement > Components: clients - java >Affects Versions: 3.5, 4.0-ALPHA >Reporter: Shawn Heisey >Assignee: Shawn Heisey > Attachments: SOLR-3284.patch > > > StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as > HttpClient, when doing adds. It may happen with other methods, though I know > that query and deleteByQuery will throw exceptions. I believe that this is a > result of the queue/Runner design. That's what makes SUSS perform better, > but it means you sacrifice the ability to programmatically determine that > there was a problem with your update. All errors are logged via slf4j, but > that's not terribly helpful except with determining what went wrong after the > fact. > When using CommonsHttpSolrServer, I've been able to rely on getting an > exception thrown by pretty much any error, letting me use try/catch to detect > problems. > There's probably enough dependent code out there that it would not be a good > idea to change the design of SUSS, unless there were alternate constructors > or additional methods available to configure new/old behavior. Fixing this > is probably not trivial, so it's probably a better idea to come up with a new > server object based on CHSS. This is outside my current skillset. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions
[ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723187#comment-13723187 ] Shawn Heisey commented on SOLR-3284: I have a proposed patch that is very likely to need updating because it is so old. There is an issue for CloudSolrServer, the one to route documents to the correct shard, that has a concurrent mode that apparently still will throw exceptions. Can that be adapted for use here? > StreamingUpdateSolrServer swallows exceptions > - > > Key: SOLR-3284 > URL: https://issues.apache.org/jira/browse/SOLR-3284 > Project: Solr > Issue Type: Improvement > Components: clients - java >Affects Versions: 3.5, 4.0-ALPHA >Reporter: Shawn Heisey >Assignee: Shawn Heisey > Attachments: SOLR-3284.patch > > > StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as > HttpClient, when doing adds. It may happen with other methods, though I know > that query and deleteByQuery will throw exceptions. I believe that this is a > result of the queue/Runner design. That's what makes SUSS perform better, > but it means you sacrifice the ability to programmatically determine that > there was a problem with your update. All errors are logged via slf4j, but > that's not terribly helpful except with determining what went wrong after the > fact. > When using CommonsHttpSolrServer, I've been able to rely on getting an > exception thrown by pretty much any error, letting me use try/catch to detect > problems. > There's probably enough dependent code out there that it would not be a good > idea to change the design of SUSS, unless there were alternate constructors > or additional methods available to configure new/old behavior. Fixing this > is probably not trivial, so it's probably a better idea to come up with a new > server object based on CHSS. This is outside my current skillset. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions
[ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13636619#comment-13636619 ] Shawn Heisey commented on SOLR-3284: I think that having optional error reporting built into in the concurrent class is important, even if it cannot tell you which of your recent updates failed. Does anyone think my patch is a good first step, should I try another approach, or is this whole idea doomed? I am not planning to close this issue at this time, but this comment is part of an effort to close old issues that I have reported. Search tag: elyograg2013springclean > StreamingUpdateSolrServer swallows exceptions > - > > Key: SOLR-3284 > URL: https://issues.apache.org/jira/browse/SOLR-3284 > Project: Solr > Issue Type: Improvement > Components: clients - java >Affects Versions: 3.5, 4.0-ALPHA >Reporter: Shawn Heisey > Attachments: SOLR-3284.patch > > > StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as > HttpClient, when doing adds. It may happen with other methods, though I know > that query and deleteByQuery will throw exceptions. I believe that this is a > result of the queue/Runner design. That's what makes SUSS perform better, > but it means you sacrifice the ability to programmatically determine that > there was a problem with your update. All errors are logged via slf4j, but > that's not terribly helpful except with determining what went wrong after the > fact. > When using CommonsHttpSolrServer, I've been able to rely on getting an > exception thrown by pretty much any error, letting me use try/catch to detect > problems. > There's probably enough dependent code out there that it would not be a good > idea to change the design of SUSS, unless there were alternate constructors > or additional methods available to configure new/old behavior. Fixing this > is probably not trivial, so it's probably a better idea to come up with a new > server object based on CHSS. This is outside my current skillset. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions
[ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556991#comment-13556991 ] Shawn Heisey commented on SOLR-3284: I'll try to separate the patches for each half of this idea. > StreamingUpdateSolrServer swallows exceptions > - > > Key: SOLR-3284 > URL: https://issues.apache.org/jira/browse/SOLR-3284 > Project: Solr > Issue Type: Improvement > Components: clients - java >Affects Versions: 3.5, 4.0-ALPHA >Reporter: Shawn Heisey > Attachments: SOLR-3284.patch > > > StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as > HttpClient, when doing adds. It may happen with other methods, though I know > that query and deleteByQuery will throw exceptions. I believe that this is a > result of the queue/Runner design. That's what makes SUSS perform better, > but it means you sacrifice the ability to programmatically determine that > there was a problem with your update. All errors are logged via slf4j, but > that's not terribly helpful except with determining what went wrong after the > fact. > When using CommonsHttpSolrServer, I've been able to rely on getting an > exception thrown by pretty much any error, letting me use try/catch to detect > problems. > There's probably enough dependent code out there that it would not be a good > idea to change the design of SUSS, unless there were alternate constructors > or additional methods available to configure new/old behavior. Fixing this > is probably not trivial, so it's probably a better idea to come up with a new > server object based on CHSS. This is outside my current skillset. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions
[ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556990#comment-13556990 ] Shawn Heisey commented on SOLR-3284: I had an idea for tracking the first update that caused an error. This is a little convoluted, so bear with me a little, and let me know what you think. This first part is something that I think would be a good idea to add even if the rest is useless: The server object will have a field called requestNumber, an AtomicInteger, that will be incremented every time an UpdateRequest is used. When an UpdateRequest is made and it does not already contain a parameter called concurrentUpdateId, the string representation of the current requestNumber will be inserted into the request as the value of that parameter. Originally I had planned to have it always use the incrementing integer values, but there will be a lot of people may already have an existing ID value, and using that value would be MUCH easier. The concurrentUpdateId parameter that gets used will be included in the dummy response, and it can also be used when Runner logs status messages from each update. Now for the rest: One of the methods on the server object will set the error handling method that will get used. One choice will be exactly the way it works now, including the ability to override the handleError method. Another choice will be to raise a flag that can be checked manually using various get methods. The remaining choice would be very similar to the patch already attached to this issue, except that the update ID of the update that caused the first internal exception will be in the message on the encapsulating exception. I've got some of this written already. It is pushing my java skill boundaries a little. Even if the latter part is not implemented, if the first part goes in, the user could override a new HandleError(Exception,String) method to track the ID of every update that fails, not just the first. > StreamingUpdateSolrServer swallows exceptions > - > > Key: SOLR-3284 > URL: https://issues.apache.org/jira/browse/SOLR-3284 > Project: Solr > Issue Type: Improvement > Components: clients - java >Affects Versions: 3.5, 4.0-ALPHA >Reporter: Shawn Heisey > Attachments: SOLR-3284.patch > > > StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as > HttpClient, when doing adds. It may happen with other methods, though I know > that query and deleteByQuery will throw exceptions. I believe that this is a > result of the queue/Runner design. That's what makes SUSS perform better, > but it means you sacrifice the ability to programmatically determine that > there was a problem with your update. All errors are logged via slf4j, but > that's not terribly helpful except with determining what went wrong after the > fact. > When using CommonsHttpSolrServer, I've been able to rely on getting an > exception thrown by pretty much any error, letting me use try/catch to detect > problems. > There's probably enough dependent code out there that it would not be a good > idea to change the design of SUSS, unless there were alternate constructors > or additional methods available to configure new/old behavior. Fixing this > is probably not trivial, so it's probably a better idea to come up with a new > server object based on CHSS. This is outside my current skillset. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions
[ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13530665#comment-13530665 ] Mark Miller commented on SOLR-3284: --- I'd love to see a way to get back specific error information about fails. That seems preferable to just stopping or throwing an exception in a lot of cases. It would be nice to treat different cases separatley though - eg bad doc vs down server. > StreamingUpdateSolrServer swallows exceptions > - > > Key: SOLR-3284 > URL: https://issues.apache.org/jira/browse/SOLR-3284 > Project: Solr > Issue Type: Improvement > Components: clients - java >Affects Versions: 3.5, 4.0-ALPHA >Reporter: Shawn Heisey > Attachments: SOLR-3284.patch > > > StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as > HttpClient, when doing adds. It may happen with other methods, though I know > that query and deleteByQuery will throw exceptions. I believe that this is a > result of the queue/Runner design. That's what makes SUSS perform better, > but it means you sacrifice the ability to programmatically determine that > there was a problem with your update. All errors are logged via slf4j, but > that's not terribly helpful except with determining what went wrong after the > fact. > When using CommonsHttpSolrServer, I've been able to rely on getting an > exception thrown by pretty much any error, letting me use try/catch to detect > problems. > There's probably enough dependent code out there that it would not be a good > idea to change the design of SUSS, unless there were alternate constructors > or additional methods available to configure new/old behavior. Fixing this > is probably not trivial, so it's probably a better idea to come up with a new > server object based on CHSS. This is outside my current skillset. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions
[ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438072#comment-13438072 ] Mikhail Khludnev commented on SOLR-3284: Hi, Excuse me for hijacking, but server-side analog SOLR-3585 is really strict in case of any errors, it gives up and throws them back to clients. > StreamingUpdateSolrServer swallows exceptions > - > > Key: SOLR-3284 > URL: https://issues.apache.org/jira/browse/SOLR-3284 > Project: Solr > Issue Type: Improvement > Components: clients - java >Affects Versions: 3.5, 4.0-ALPHA >Reporter: Shawn Heisey > Attachments: SOLR-3284.patch > > > StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as > HttpClient, when doing adds. It may happen with other methods, though I know > that query and deleteByQuery will throw exceptions. I believe that this is a > result of the queue/Runner design. That's what makes SUSS perform better, > but it means you sacrifice the ability to programmatically determine that > there was a problem with your update. All errors are logged via slf4j, but > that's not terribly helpful except with determining what went wrong after the > fact. > When using CommonsHttpSolrServer, I've been able to rely on getting an > exception thrown by pretty much any error, letting me use try/catch to detect > problems. > There's probably enough dependent code out there that it would not be a good > idea to change the design of SUSS, unless there were alternate constructors > or additional methods available to configure new/old behavior. Fixing this > is probably not trivial, so it's probably a better idea to come up with a new > server object based on CHSS. This is outside my current skillset. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions
[ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437752#comment-13437752 ] Alan Woodward commented on SOLR-3284: - This interacts really badly with bad documents (SOLR-445) - once one of the Runners passes a bad document to SOLR, all documents subsequently processed by that Runner will be silently discarded, and there doesn't seem to be any way to detect this in client code. > StreamingUpdateSolrServer swallows exceptions > - > > Key: SOLR-3284 > URL: https://issues.apache.org/jira/browse/SOLR-3284 > Project: Solr > Issue Type: Improvement > Components: clients - java >Affects Versions: 3.5, 4.0-ALPHA >Reporter: Shawn Heisey > Attachments: SOLR-3284.patch > > > StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as > HttpClient, when doing adds. It may happen with other methods, though I know > that query and deleteByQuery will throw exceptions. I believe that this is a > result of the queue/Runner design. That's what makes SUSS perform better, > but it means you sacrifice the ability to programmatically determine that > there was a problem with your update. All errors are logged via slf4j, but > that's not terribly helpful except with determining what went wrong after the > fact. > When using CommonsHttpSolrServer, I've been able to rely on getting an > exception thrown by pretty much any error, letting me use try/catch to detect > problems. > There's probably enough dependent code out there that it would not be a good > idea to change the design of SUSS, unless there were alternate constructors > or additional methods available to configure new/old behavior. Fixing this > is probably not trivial, so it's probably a better idea to come up with a new > server object based on CHSS. This is outside my current skillset. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions
[ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292573#comment-13292573 ] Shawn Heisey commented on SOLR-3284: Lance, if your idea (or your idea in combination with Mike's) can be implemented without a ton of work, it should definitely replace my patch. If it's a royal pain, my patch is there as an interim solution and yours can be handled in another issue. Is the following a good summary of your idea? If so, I think your idea is better. I think it should be the default behavior for 4x and trunk. A user of ConcurrentUpdateSolrServer would make update requests, then call a publicly exposed flush(), either explicitly or implicitly by calling commit(). If the flush() fails due to a background failure, all requests since the last successful flush() would be rolled back. If the user code is designed with this in mind, error handling is as good as HttpSolrServer. HSS (and probably ESS) would use flush() internally. The user would be free to call flush() themselves, but it would not be required. I'm curious about how things would be handled with autoCommit or commitWithin. Is rollback possible when those are used? > StreamingUpdateSolrServer swallows exceptions > - > > Key: SOLR-3284 > URL: https://issues.apache.org/jira/browse/SOLR-3284 > Project: Solr > Issue Type: Improvement > Components: clients - java >Affects Versions: 3.5, 4.0 >Reporter: Shawn Heisey > Attachments: SOLR-3284.patch > > > StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as > HttpClient, when doing adds. It may happen with other methods, though I know > that query and deleteByQuery will throw exceptions. I believe that this is a > result of the queue/Runner design. That's what makes SUSS perform better, > but it means you sacrifice the ability to programmatically determine that > there was a problem with your update. All errors are logged via slf4j, but > that's not terribly helpful except with determining what went wrong after the > fact. > When using CommonsHttpSolrServer, I've been able to rely on getting an > exception thrown by pretty much any error, letting me use try/catch to detect > problems. > There's probably enough dependent code out there that it would not be a good > idea to change the design of SUSS, unless there were alternate constructors > or additional methods available to configure new/old behavior. Fixing this > is probably not trivial, so it's probably a better idea to come up with a new > server object based on CHSS. This is outside my current skillset. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions
[ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292507#comment-13292507 ] Mike Sokolov commented on SOLR-3284: Another approach we have been using in a similar parallel writing situation is to have the thread pool maintain a fixed-size list of exceptions. Whenever a worker throws one, it gets put on the list. Then at certain barrier conditions (flush, exception list full), the exceptions are collected together and re-thrown using an umbrella exception that wraps them. You do need flush(), or can you rely on the user calling commit() (and flush() internally then)? This enables writing to continue even in the face of errors, but the errors do get reported eventually. This makes the system to be more robust in the face of transient failure conditions. > StreamingUpdateSolrServer swallows exceptions > - > > Key: SOLR-3284 > URL: https://issues.apache.org/jira/browse/SOLR-3284 > Project: Solr > Issue Type: Improvement > Components: clients - java >Affects Versions: 3.5, 4.0 >Reporter: Shawn Heisey > Attachments: SOLR-3284.patch > > > StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as > HttpClient, when doing adds. It may happen with other methods, though I know > that query and deleteByQuery will throw exceptions. I believe that this is a > result of the queue/Runner design. That's what makes SUSS perform better, > but it means you sacrifice the ability to programmatically determine that > there was a problem with your update. All errors are logged via slf4j, but > that's not terribly helpful except with determining what went wrong after the > fact. > When using CommonsHttpSolrServer, I've been able to rely on getting an > exception thrown by pretty much any error, letting me use try/catch to detect > problems. > There's probably enough dependent code out there that it would not be a good > idea to change the design of SUSS, unless there were alternate constructors > or additional methods available to configure new/old behavior. Fixing this > is probably not trivial, so it's probably a better idea to come up with a new > server object based on CHSS. This is outside my current skillset. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions
[ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292206#comment-13292206 ] Lance Norskog commented on SOLR-3284: - In the trunk it is now ConcurrentUpdateSolrServer, and it swallows errors all over the place. I am a total fascist about fail-fast/fail-loud. A more obnoxious (fail-fast) design is: * Any exception in a worker causes the main driver to kill all other workers. * The default handler after this is a rollback. * Add a flush() method to SolrServer. ** CUSS uses this to report back to the main thread that everything blew up. ** Other server classes would do a socket flush and wait and drain out any error text. * Use a finalize() method to complain that the user did not call flush(). ** This is in all SolrServer classes- the same application code should do the same thing with every SolrServer implementation. We tell our customers to not use SUSS/CUSS because it does not do errors right. Partly this causes problems because people tend to "design to success" (assume everything works) instead of "design to failure" (assume everything breaks). When CUSS breaks, the production scripting and staff do not notice. > StreamingUpdateSolrServer swallows exceptions > - > > Key: SOLR-3284 > URL: https://issues.apache.org/jira/browse/SOLR-3284 > Project: Solr > Issue Type: Improvement > Components: clients - java >Affects Versions: 3.5, 4.0 >Reporter: Shawn Heisey > Attachments: SOLR-3284.patch > > > StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as > HttpClient, when doing adds. It may happen with other methods, though I know > that query and deleteByQuery will throw exceptions. I believe that this is a > result of the queue/Runner design. That's what makes SUSS perform better, > but it means you sacrifice the ability to programmatically determine that > there was a problem with your update. All errors are logged via slf4j, but > that's not terribly helpful except with determining what went wrong after the > fact. > When using CommonsHttpSolrServer, I've been able to rely on getting an > exception thrown by pretty much any error, letting me use try/catch to detect > problems. > There's probably enough dependent code out there that it would not be a good > idea to change the design of SUSS, unless there were alternate constructors > or additional methods available to configure new/old behavior. Fixing this > is probably not trivial, so it's probably a better idea to come up with a new > server object based on CHSS. This is outside my current skillset. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions
[ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255705#comment-13255705 ] Shawn Heisey commented on SOLR-3284: After looking at existing tests to see how I might implement tests for this new functionality, I couldn't see how to do it. Also, I noticed that there are tests for SolrCloud and something else called ChaosMonkey. All tests in solr/ pass with this patch, but I don't know how SolrCloud might be affected. I would hope that it already handles exceptions properly and therefore wouldn't have any problems, but I have never looked at the code or used SolrCloud. > StreamingUpdateSolrServer swallows exceptions > - > > Key: SOLR-3284 > URL: https://issues.apache.org/jira/browse/SOLR-3284 > Project: Solr > Issue Type: Improvement > Components: clients - java >Affects Versions: 3.5, 4.0 >Reporter: Shawn Heisey > Attachments: SOLR-3284.patch > > > StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as > HttpClient, when doing adds. It may happen with other methods, though I know > that query and deleteByQuery will throw exceptions. I believe that this is a > result of the queue/Runner design. That's what makes SUSS perform better, > but it means you sacrifice the ability to programmatically determine that > there was a problem with your update. All errors are logged via slf4j, but > that's not terribly helpful except with determining what went wrong after the > fact. > When using CommonsHttpSolrServer, I've been able to rely on getting an > exception thrown by pretty much any error, letting me use try/catch to detect > problems. > There's probably enough dependent code out there that it would not be a good > idea to change the design of SUSS, unless there were alternate constructors > or additional methods available to configure new/old behavior. Fixing this > is probably not trivial, so it's probably a better idea to come up with a new > server object based on CHSS. This is outside my current skillset. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions
[ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255659#comment-13255659 ] Shawn Heisey commented on SOLR-3284: The patch should also apply successfully to 3.6. > StreamingUpdateSolrServer swallows exceptions > - > > Key: SOLR-3284 > URL: https://issues.apache.org/jira/browse/SOLR-3284 > Project: Solr > Issue Type: Improvement > Components: clients - java >Affects Versions: 3.5, 4.0 >Reporter: Shawn Heisey > Attachments: SOLR-3284.patch > > > StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as > HttpClient, when doing adds. It may happen with other methods, though I know > that query and deleteByQuery will throw exceptions. I believe that this is a > result of the queue/Runner design. That's what makes SUSS perform better, > but it means you sacrifice the ability to programmatically determine that > there was a problem with your update. All errors are logged via slf4j, but > that's not terribly helpful except with determining what went wrong after the > fact. > When using CommonsHttpSolrServer, I've been able to rely on getting an > exception thrown by pretty much any error, letting me use try/catch to detect > problems. > There's probably enough dependent code out there that it would not be a good > idea to change the design of SUSS, unless there were alternate constructors > or additional methods available to configure new/old behavior. Fixing this > is probably not trivial, so it's probably a better idea to come up with a new > server object based on CHSS. This is outside my current skillset. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions
[ https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255602#comment-13255602 ] Shawn Heisey commented on SOLR-3284: If the Solr server goes down in between updates done with the concurrent server, doing further updates will fail, but the calling code will not know that. With the Commons or Http server, an exception is thrown that my code catches. I don't think that just overriding handleError is enough. If Solr goes down but the machine is still up, you have immediate failure detection because the connection will be refused. If the server goes away entirely, it could take a couple of minutes to fail. You would have to provide methods to check that 1) all background operations are complete and 2) they were error free. I can no longer remember whether an exception is thrown when trying a commit against a down machine with the concurrent server. IIRC it does throw one in this instance. I definitely believe that it should. Perhaps the current handleError code could update class-level members (with names like "boolean updateErrored" and "SolrServerException updateException") that could be checked and used by the commit method. If they are set, it would reset them and throw an exception (fast-fail) without actually trying the commit. There should probably be a constructor option and a set method to either activate this new behavior or restore the original behavior. When I first designed my code, I was relying on the exceptions thrown by the commons server when doing the actual update, so it's too late by the time it reaches the commit - it has already updated the position values. I now realize that this is incorrect design, though I might never have figured it out without my attempt to use the concurrent server. It's going to be a bit painful to redesign my code to put off updating position values until after a successful commit operation. It's something I do intend to do. > StreamingUpdateSolrServer swallows exceptions > - > > Key: SOLR-3284 > URL: https://issues.apache.org/jira/browse/SOLR-3284 > Project: Solr > Issue Type: Improvement > Components: clients - java >Affects Versions: 3.5, 4.0 >Reporter: Shawn Heisey > > StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as > HttpClient, when doing adds. It may happen with other methods, though I know > that query and deleteByQuery will throw exceptions. I believe that this is a > result of the queue/Runner design. That's what makes SUSS perform better, > but it means you sacrifice the ability to programmatically determine that > there was a problem with your update. All errors are logged via slf4j, but > that's not terribly helpful except with determining what went wrong after the > fact. > When using CommonsHttpSolrServer, I've been able to rely on getting an > exception thrown by pretty much any error, letting me use try/catch to detect > problems. > There's probably enough dependent code out there that it would not be a good > idea to change the design of SUSS, unless there were alternate constructors > or additional methods available to configure new/old behavior. Fixing this > is probably not trivial, so it's probably a better idea to come up with a new > server object based on CHSS. This is outside my current skillset. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3284) StreamingUpdateSolrServer swallows exceptions
[
https://issues.apache.org/jira/browse/SOLR-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255433#comment-13255433
]
Sami Siren commented on SOLR-3284:
--
What are the operations/error situations where you are not seeing an Exception
when you expect one?
By default the ConcurrentUpdateSolrServer (StreamingUpdateSolrServer) just logs
the exceptions from updates but you can override this functionality:
{code}
SolrServer server = new
ConcurrentUpdateSolrServer("http://127.0.0.1:8983/solr";, 1, 1){
public void handleError(Throwable ex) {
//do something with the Throwable here
System.out.println("Something wrong!" + ex.getMessage());
}
};
server.add(new SolrInputDocument());
{code}
The current exception reporting is pretty limited and it is impossible to see
which operation triggered the exception but such improvements should be done in
separate issues.
> StreamingUpdateSolrServer swallows exceptions
> -
>
> Key: SOLR-3284
> URL: https://issues.apache.org/jira/browse/SOLR-3284
> Project: Solr
> Issue Type: Improvement
> Components: clients - java
>Affects Versions: 3.5, 4.0
>Reporter: Shawn Heisey
>
> StreamingUpdateSolrServer eats exceptions thrown by lower level code, such as
> HttpClient, when doing adds. It may happen with other methods, though I know
> that query and deleteByQuery will throw exceptions. I believe that this is a
> result of the queue/Runner design. That's what makes SUSS perform better,
> but it means you sacrifice the ability to programmatically determine that
> there was a problem with your update. All errors are logged via slf4j, but
> that's not terribly helpful except with determining what went wrong after the
> fact.
> When using CommonsHttpSolrServer, I've been able to rely on getting an
> exception thrown by pretty much any error, letting me use try/catch to detect
> problems.
> There's probably enough dependent code out there that it would not be a good
> idea to change the design of SUSS, unless there were alternate constructors
> or additional methods available to configure new/old behavior. Fixing this
> is probably not trivial, so it's probably a better idea to come up with a new
> server object based on CHSS. This is outside my current skillset.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
