Hi Costin - by "bulk size/entries number" are you referring to the es.batch.size.bytes and es.batch.size.entries config values described here?
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/master/configuration.html#configuration-serialization It looks like the only elasticsearch-related config values we're setting is this: es.input.json = true So we must be using default values for those es.batch.size config values. Thanks, Zach On Wed, Oct 1, 2014 at 7:41 AM, Zach Cox <[email protected]> wrote: > This particular job has 1353 map tasks, Hadoop cluster has 5 nodes with > total map task capacity of 25. Elasticsearch cluster has 4 nodes. > > Where can I find the bulk size/entries numbers? > > > Thanks, > Zach > > > > On Wed, Oct 1, 2014 at 7:19 AM, Costin Leau <[email protected]> wrote: > > The error indicates the ES nodes don't reply in a timely fashion and thus > > the connection drops. Based on your logs it seems to be either a GC or a > > network issue. > > You could try turning on logging in package > 'org.elasticsearch.hadoop.rest' > > to DEBUG. > > How many tasks do you have and what's your bulk size/entries number? > > > > > > On 10/1/14 2:15 PM, Zach Cox wrote: > >> > >> Hi Costin - we updated our dependencies to use elasticsearch-hadoop > >> 2.0.2.BUILD-SNAPSHOT, but that didn't seem to change > >> anything. We're still seeing the same task failures while trying to > write > >> to Elasticsearch. The only difference in the > >> logs is that now I don't see the SimpleHttpConnectionManager warnings. > >> > >> Any ideas what we could try next? > >> > >> Thanks, > >> Zach > >> > >> > >> On Tuesday, September 30, 2014 10:54:27 AM UTC-5, Costin Leau wrote: > >> > >> Can you please try the 2.0.2.BUILD-SNAPSHOT? I think you might be > >> running into issue #256 which was fixed some time ago > >> and will be part of the upcoming > >> 2.0.2, 2.1 Beta2. > >> > >> Cheers, > >> > >> On 9/30/14 6:43 PM, Zach Cox wrote: > >> > Hi Costin: > >> > > >> > elasticsearch-hadoop 2.0.0 > >> > cascading 2.5.4 > >> > scalding 0.10.0 > >> > > >> > Thanks, > >> > Zach > >> > > >> > > >> > On Tuesday, September 30, 2014 10:25:10 AM UTC-5, Costin Leau > wrote: > >> > > >> > What version of es-hadoop/es/cascading are you using? > >> > > >> > On 9/30/14 6:16 PM, Zach Cox wrote: > >> > > Hi - we're having problems with one of our map-reduce jobs > >> that writes to Elasticsearch. Lots of map tasks are failing > >> > > due to ES being "unavailable", with logs like this: > >> > > > >> > > >> > > https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt > >> > >> < > https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt > > > >> > >> > > >> < > https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt > >> > >> < > https://gist.githubusercontent.com/zcox/3d6cf4329d49ca03271b/raw/57c46a5e4c9ea04d5c4209414d6f847492d16c0d/gistfile1.txt > >> > >> > >> > > >> > > > >> > > Seems like elasticsearch-hadoop tries talking to an ES node, > >> it times out, tries the next one, it times out, etc until > >> > > all nodes in the cluster are exhausted and then it gives up. > >> > > > >> > > As far as I can tell, the ES cluster is healthy while this > is > >> occurring. May map tasks are succeeding - probably about > >> > > 10% of the attempts are killed due to this issue. The main > >> problem is that these killed tasks waste a lot of time, and > >> > > slow down the overall job execution. > >> > > > >> > > I'm not sure where to troubleshoot this next. Does anyone > have > >> any idea what would cause all of these time outs & failures? > >> > > > >> > > I'm also curious about the lines like this: > >> > > > >> > > 2014-09-30 12:49:20,469 WARN > >> org.apache.commons.httpclient.SimpleHttpConnectionManager: > >> SimpleHttpConnectionManager being used incorrectly. Be sure that > >> > HttpMethod.releaseConnection() is always called and that only > >> one thread and/or method is using this connection > >> > manager at a time. > >> > > > >> > > > >> > > Would that be related to the timeout problem we're seeing? > >> > > > >> > > Thanks, > >> > > Zach > >> > > > >> > > -- > >> > > You received this message because you are subscribed to the > >> Google Groups "elasticsearch" group. > >> > > To unsubscribe from this group and stop receiving emails > from > >> it, send an email to > >> > >[email protected] <javascript:> > >> <mailto:[email protected] <javascript:> > >> <javascript:>>. > >> > > To view this discussion on the web visit > >> > > >> > > https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com > >> > >> < > https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com > > > >> > > >> < > https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com > >> > >> < > https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com > >> > >> > > > >> < > https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer > >> > >> < > https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer > > > >> > >> > > >> < > https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer > >> > >> < > https://groups.google.com/d/msgid/elasticsearch/f304a286-399f-4dea-b7f0-032b19ad67e6%40googlegroups.com?utm_medium=email&utm_source=footer > >>>. > >> > >> > > >> > > For more options, visithttps://groups.google.com/d/optout > >> <http://groups.google.com/d/optout> <https://groups.google.com/d/optout > >> <https://groups.google.com/d/optout>>. > >> > -- > >> > Costin > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > >> Groups "elasticsearch" group. > >> > To unsubscribe from this group and stop receiving emails from it, > >> send an email to > >> >[email protected] <javascript:> > >> <mailto:[email protected] <javascript:>>. > >> > To view this discussion on the web visit > >> > >> > > https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com > >> > >> < > https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com > > > >> > > >> < > https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer > >> > >> < > https://groups.google.com/d/msgid/elasticsearch/034d651e-8562-4dde-bbb9-b3fef6d0d0b9%40googlegroups.com?utm_medium=email&utm_source=footer > >>. > >> > >> > For more options, visithttps://groups.google.com/d/optout > >> <https://groups.google.com/d/optout>. > >> > >> -- > >> Costin > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "elasticsearch" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to > >> [email protected] > >> <mailto:[email protected]>. > >> To view this discussion on the web visit > >> > >> > https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com > >> > >> < > https://groups.google.com/d/msgid/elasticsearch/90b059ef-680e-4b5c-a0c9-dc5e5038205a%40googlegroups.com?utm_medium=email&utm_source=footer > >. > >> For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > Costin > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > https://groups.google.com/d/topic/elasticsearch/BKR18lczF1w/unsubscribe. > > To unsubscribe from this group and all its topics, send an email to > > [email protected]. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/elasticsearch/542BF16B.6010103%40gmail.com > . > > > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANjo42wT_Govm3zL%2ByhGvO--wA1%2BJS2FPSgbxj5bfdcM5Aghgg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
