Nice work, can you share the recipe with the community? I could post it on the JDBC plugin wiki
Jörg On Wed, Apr 29, 2015 at 1:56 PM, GWired <garrettcjohn...@gmail.com> wrote: > My theory is that i was overloading my ES VM's on initial loads or when > doing large loads. > > My cpu would jump to 99% and during the pulls it would fail silently and > the river thought it got all the documents but didn't. > > I have since rewritten my rivers using Stored procedures instead and it > pulls much faster and CPU doesn't go to high. > > The procedure keeps track of the created_at and updated_at for me instead > of using the column strategy. I also chunked my initial calls into logical > batches. Mine happen to be date based by month. I couldn't figure out a > way that made sense using row sizes or anything and month seemed just as > good as any and it works. > > This also solved my problem of when / ES is turned off. It now will just > start back up where it last left off because SQL server is storing that > info instead of ES. I also made it so I can rebuild at will. > > Thanks Garrett > > > > On Thursday, April 23, 2015 at 2:48:03 AM UTC-4, Jörg Prante wrote: >> >> There are log messages at ES cluster side, you should look there why bulk >> indexing failed. >> >> Jörg >> >> On Thu, Apr 23, 2015 at 5:45 AM, GWired <garrett...@gmail.com> wrote: >> >>> Found this in the logs: >>> >>> [2015-04-22 22:01:25,063][ERROR][river.jdbc.BulkNodeClient] bulk [15] >>> failed with 945 failed items, failure message = failure in bulk execution: >>> >>> >>> >>> On Wednesday, April 22, 2015 at 7:53:25 PM UTC-5, GWired wrote: >>>> >>>> Hi All, >>>> >>>> I've just been informed that i'm off by up to 100k records or so in my >>>> jdbc river fed index. >>>> >>>> I am using the column strategy using a createddate and lastmodified >>>> date. >>>> >>>> Kibana is reporting an entirely different # than what i see reported in >>>> the DB.. >>>> >>>> Table A has 978634 in SQL, 934646 shown in Kibana. >>>> Table B has 957327 in SQL, 876725 shown in Kibana. >>>> Table C has 312826 in SQL, 238534 shown in Kibana >>>> >>>> I see in the ES logs >>>> >>>> Table A metrics: 979044 rows, >>>> Table B metrics: 957591 rows >>>> Table C metrics: 312827 rows, >>>> >>>> These are the right numbers...well at least closer to right. >>>> >>>> But if i do this using Sense: >>>> >>>> GET jdbc/mytable/_count?q=* >>>> >>>> It returns the same # as Kibana is return. >>>> >>>> This erring version is running on ES 1.5.1 with Kibana version 3.0 >>>> >>>> On another server with ES 1.5.0 and Kibana 3.0 it is working just fine >>>> #'s match up. >>>> >>>> Any ideas? >>>> >>>> >>>> >>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearc...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/5561753d-9553-4bc5-bea2-102b7e030396%40googlegroups.com >>> <https://groups.google.com/d/msgid/elasticsearch/5561753d-9553-4bc5-bea2-102b7e030396%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/5c2016b5-ef5f-4f54-ac37-b2991eba0775%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/5c2016b5-ef5f-4f54-ac37-b2991eba0775%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGzT%2BCFJ9XOSt%2BYPBJ724A40yUF3HPB7iOqoMf%2BygAnKA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
