Hi Costin - we updated our dependencies to use elasticsearch-hadoop 2.0.2.BUILD-SNAPSHOT, but that didn't seem to change anything. We're still seeing the same task failures while trying to write to Elasticsearch. The only difference in the logs is that now I don't see the SimpleHttpConnectionManager warnings.
Any ideas what we could try next? Thanks, Zach On Tuesday, September 30, 2014 10:54:27 AM UTC-5, Costin Leau wrote: > > Can you please try the 2.0.2.BUILD-SNAPSHOT? I think you might be running > into issue #256 which was fixed some time ago > and will be part of the upcoming > 2.0.2, 2.1 Beta2. > > Cheers, > > On 9/30/14 6:43 PM, Zach Cox wrote: > > Hi Costin: > > > > elasticsearch-hadoop 2.0.0 > > cascading 2.5.4 > > scalding 0.10.0 > > > > Thanks, > > Zach > > > > > > On Tuesday, September 30, 2014 10:25:10 AM UTC-5, Costin Leau wrote: > > > > What version of es-hadoop/es/cascading are you using? > > > > On 9/30/14 6:16 PM, Zach Cox wrote: > > > Hi - we're having problems with one of our map-reduce jobs that > writes to Elasticsearch. Lots of map tasks are failing > > > due to ES being "unavailable", with logs like this: > > > > > > > https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt > > > < > https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt> > > > > > > > > > > Seems like elasticsearch-hadoop tries talking to an ES node, it > times out, tries the next one, it times out, etc until > > > all nodes in the cluster are exhausted and then it gives up. > > > > > > As far as I can tell, the ES cluster is healthy while this is > occurring. May map tasks are succeeding - probably about > > > 10% of the attempts are killed due to this issue. The main problem > is that these killed tasks waste a lot of time, and > > > slow down the overall job execution. > > > > > > I'm not sure where to troubleshoot this next. Does anyone have any > idea what would cause all of these time outs & failures? > > > > > > I'm also curious about the lines like this: > > > > > > 2014-09-30 12:49:20,469 WARN > org.apache.commons.httpclient.SimpleHttpConnectionManager: > SimpleHttpConnectionManager being used incorrectly. Be sure that > > HttpMethod.releaseConnection() is always called and that only one > thread and/or method is using this connection > > manager at a time. > > > > > > > > > Would that be related to the timeout problem we're seeing? > > > > > > Thanks, > > > Zach > > > > > > -- > > > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, > send an email to > > >[email protected] <javascript:> <mailto: > [email protected] <javascript:> <javascript:>>. > > > To view this discussion on the web visit > > > > https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com > > > < > https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com> > > > > > < > https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer > > > < > https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer>>. > > > > > > > For more options, visithttps://groups.google.com/d/optout < > https://groups.google.com/d/optout>. > > -- > > Costin > > > > -- > > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to > > [email protected] <javascript:> <mailto: > [email protected] <javascript:>>. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com > > > < > https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > > For more options, visit https://groups.google.com/d/optout. > > -- > Costin > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
