Is there anything else we could try here to debug elasticsearch-hadoop being unable to write to Elasticsearch? We're still seeing the same number of these fails during the nightly batch runs even after switching to 2.0.2.BUILD-SNAPSHOT, and I don't see any additional lines from org.elasticsearch.hadoop.rest in the tasktracker logs after adding <logger name="org.elasticsearch.hadoop.rest" level="DEBUG"/> to our logback.xml.
Here are example logs for a task that failed this morning: https://gist.githubusercontent.com/zcox/c5c81f4f8ee26d7bedbf/raw/ac9bef7bfe3a8e1ef69a62aabe4c7c3983882f19/gistfile1.txt Is it reasonable to expect 4 Elasticsearch nodes to handle the batch write volume from 5 Hadoop nodes (25 map tasks)? Thanks, Zach On Wed, Oct 1, 2014 at 8:00 AM, Zach Cox <[email protected]> wrote: > Hi Costin - by "bulk size/entries number" are you referring to the > es.batch.size.bytes and es.batch.size.entries config values described here? > > > http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/master/configuration.html#configuration-serialization > > It looks like the only elasticsearch-related config values we're setting > is this: > > es.input.json = true > > So we must be using default values for those es.batch.size config values. > > Thanks, > Zach > > > > On Wed, Oct 1, 2014 at 7:41 AM, Zach Cox <[email protected]> wrote: > >> This particular job has 1353 map tasks, Hadoop cluster has 5 nodes with >> total map task capacity of 25. Elasticsearch cluster has 4 nodes. >> >> Where can I find the bulk size/entries numbers? >> >> >> Thanks, >> Zach >> >> >> >> On Wed, Oct 1, 2014 at 7:19 AM, Costin Leau <[email protected]> >> wrote: >> > The error indicates the ES nodes don't reply in a timely fashion and >> thus >> > the connection drops. Based on your logs it seems to be either a GC or a >> > network issue. >> > You could try turning on logging in package >> 'org.elasticsearch.hadoop.rest' >> > to DEBUG. >> > How many tasks do you have and what's your bulk size/entries number? >> > >> > >> > On 10/1/14 2:15 PM, Zach Cox wrote: >> >> >> >> Hi Costin - we updated our dependencies to use elasticsearch-hadoop >> >> 2.0.2.BUILD-SNAPSHOT, but that didn't seem to change >> >> anything. We're still seeing the same task failures while trying to >> write >> >> to Elasticsearch. The only difference in the >> >> logs is that now I don't see the SimpleHttpConnectionManager warnings. >> >> >> >> Any ideas what we could try next? >> >> >> >> Thanks, >> >> Zach >> >> >> >> >> >> On Tuesday, September 30, 2014 10:54:27 AM UTC-5, Costin Leau wrote: >> >> >> >> Can you please try the 2.0.2.BUILD-SNAPSHOT? I think you might be >> >> running into issue #256 which was fixed some time ago >> >> and will be part of the upcoming >> >> 2.0.2, 2.1 Beta2. >> >> >> >> Cheers, >> >> >> >> On 9/30/14 6:43 PM, Zach Cox wrote: >> >> > Hi Costin: >> >> > >> >> > elasticsearch-hadoop 2.0.0 >> >> > cascading 2.5.4 >> >> > scalding 0.10.0 >> >> > >> >> > Thanks, >> >> > Zach >> >> > >> >> > >> >> > On Tuesday, September 30, 2014 10:25:10 AM UTC-5, Costin Leau >> wrote: >> >> > >> >> > What version of es-hadoop/es/cascading are you using? >> >> > >> >> > On 9/30/14 6:16 PM, Zach Cox wrote: >> >> > > Hi - we're having problems with one of our map-reduce jobs >> >> that writes to Elasticsearch. Lots of map tasks are failing >> >> > > due to ES being "unavailable", with logs like this: >> >> > > >> >> > >> >> > >> https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt >> >> >> >> < >> https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt >> > >> >> >> >> > >> >> < >> https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt >> >> >> >> < >> https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt >> >> >> >> >> >> > >> >> > > >> >> > > Seems like elasticsearch-hadoop tries talking to an ES >> node, >> >> it times out, tries the next one, it times out, etc until >> >> > > all nodes in the cluster are exhausted and then it gives >> up. >> >> > > >> >> > > As far as I can tell, the ES cluster is healthy while this >> is >> >> occurring. May map tasks are succeeding - probably about >> >> > > 10% of the attempts are killed due to this issue. The main >> >> problem is that these killed tasks waste a lot of time, and >> >> > > slow down the overall job execution. >> >> > > >> >> > > I'm not sure where to troubleshoot this next. Does anyone >> have >> >> any idea what would cause all of these time outs & failures? >> >> > > >> >> > > I'm also curious about the lines like this: >> >> > > >> >> > > 2014-09-30 12:49:20,469 WARN >> >> org.apache.commons.httpclient.SimpleHttpConnectionManager: >> >> SimpleHttpConnectionManager being used incorrectly. Be sure that >> >> > HttpMethod.releaseConnection() is always called and that only >> >> one thread and/or method is using this connection >> >> > manager at a time. >> >> > > >> >> > > >> >> > > Would that be related to the timeout problem we're seeing? >> >> > > >> >> > > Thanks, >> >> > > Zach >> >> > > >> >> > > -- >> >> > > You received this message because you are subscribed to the >> >> Google Groups "elasticsearch" group. >> >> > > To unsubscribe from this group and stop receiving emails >> from >> >> it, send an email to >> >> > >[email protected] <javascript:> >> >> <mailto:[email protected] <javascript:> >> >> <javascript:>>. >> >> > > To view this discussion on the web visit >> >> > >> >> > >> https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com >> >> >> >> < >> https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com >> > >> >> > >> >> < >> https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com >> >> >> >> < >> https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com >> >> >> >> > > >> >> < >> https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer >> >> >> >> < >> https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer >> > >> >> >> >> > >> >> < >> https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer >> >> >> >> < >> https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer >> >>>. >> >> >> >> > >> >> > > For more options, visithttps://groups.google.com/d/optout >> >> <http://groups.google.com/d/optout> < >> https://groups.google.com/d/optout >> >> <https://groups.google.com/d/optout>>. >> >> > -- >> >> > Costin >> >> > >> >> > -- >> >> > You received this message because you are subscribed to the >> Google >> >> Groups "elasticsearch" group. >> >> > To unsubscribe from this group and stop receiving emails from it, >> >> send an email to >> >> >[email protected] <javascript:> >> >> <mailto:[email protected] <javascript:>>. >> >> > To view this discussion on the web visit >> >> >> >> > >> https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com >> >> >> >> < >> https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com >> > >> >> > >> >> < >> https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer >> >> >> >> < >> https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer >> >>. >> >> >> >> > For more options, visithttps://groups.google.com/d/optout >> >> <https://groups.google.com/d/optout>. >> >> >> >> -- >> >> Costin >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> Groups >> >> "elasticsearch" group. >> >> To unsubscribe from this group and stop receiving emails from it, send >> an >> >> email to >> >> [email protected] >> >> <mailto:[email protected]>. >> >> To view this discussion on the web visit >> >> >> >> >> https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com >> >> >> >> < >> https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com?utm_medium=email&utm_source=footer >> >. >> >> For more options, visit https://groups.google.com/d/optout. >> > >> > >> > -- >> > Costin >> > >> > -- >> > You received this message because you are subscribed to a topic in the >> > Google Groups "elasticsearch" group. >> > To unsubscribe from this topic, visit >> > https://groups.google.com/d/topic/elasticsearch/BKR18lczF1w/unsubscribe >> . >> > To unsubscribe from this group and all its topics, send an email to >> > [email protected]. >> > To view this discussion on the web visit >> > >> https://groups.google.com/d/msgid/elasticsearch/542BF16B.6010103%40gmail.com >> . >> > >> > For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANjo42zgBXS4Y%3DSoRG67Q%3DeqH4ctdJDbOnzFSR6PjVZvQr-A8w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
