My theory is that i was overloading my ES VM's on initial loads or when doing large loads.
My cpu would jump to 99% and during the pulls it would fail silently and the river thought it got all the documents but didn't. I have since rewritten my rivers using Stored procedures instead and it pulls much faster and CPU doesn't go to high. The procedure keeps track of the created_at and updated_at for me instead of using the column strategy. I also chunked my initial calls into logical batches. Mine happen to be date based by month. I couldn't figure out a way that made sense using row sizes or anything and month seemed just as good as any and it works. This also solved my problem of when / ES is turned off. It now will just start back up where it last left off because SQL server is storing that info instead of ES. I also made it so I can rebuild at will. Thanks Garrett On Thursday, April 23, 2015 at 2:48:03 AM UTC-4, Jörg Prante wrote: > > There are log messages at ES cluster side, you should look there why bulk > indexing failed. > > Jörg > > On Thu, Apr 23, 2015 at 5:45 AM, GWired <garrett...@gmail.com > <javascript:>> wrote: > >> Found this in the logs: >> >> [2015-04-22 22:01:25,063][ERROR][river.jdbc.BulkNodeClient] bulk [15] >> failed with 945 failed items, failure message = failure in bulk execution: >> >> >> >> On Wednesday, April 22, 2015 at 7:53:25 PM UTC-5, GWired wrote: >>> >>> Hi All, >>> >>> I've just been informed that i'm off by up to 100k records or so in my >>> jdbc river fed index. >>> >>> I am using the column strategy using a createddate and lastmodified >>> date. >>> >>> Kibana is reporting an entirely different # than what i see reported in >>> the DB.. >>> >>> Table A has 978634 in SQL, 934646 shown in Kibana. >>> Table B has 957327 in SQL, 876725 shown in Kibana. >>> Table C has 312826 in SQL, 238534 shown in Kibana >>> >>> I see in the ES logs >>> >>> Table A metrics: 979044 rows, >>> Table B metrics: 957591 rows >>> Table C metrics: 312827 rows, >>> >>> These are the right numbers...well at least closer to right. >>> >>> But if i do this using Sense: >>> >>> GET jdbc/mytable/_count?q=* >>> >>> It returns the same # as Kibana is return. >>> >>> This erring version is running on ES 1.5.1 with Kibana version 3.0 >>> >>> On another server with ES 1.5.0 and Kibana 3.0 it is working just fine >>> #'s match up. >>> >>> Any ideas? >>> >>> >>> >>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearc...@googlegroups.com <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/5561753d-9553-4bc5-bea2-102b7e030396%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/5561753d-9553-4bc5-bea2-102b7e030396%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5c2016b5-ef5f-4f54-ac37-b2991eba0775%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
