Since rivers are just a style of execution of ES plugins, I am sure there will be replacements at the plugin level, after rivers are officially deprecated.
Logstash or custom middleware, or the JDBC feed mode, are push-style applications, which can ingest data into ES, without the requirement of a plugin. I do not think that push-style is always feasible to replace river style execution. An example is the multi tenancy scenario, where a provider wants a single piece of SaaS code to maintain for all tenants. Or stateful indexing, where the last known state of the indexing should be available at the ES side. You could implement a pull-style framework within the ES JVM, with provisioning, job processing, drivers, dispatchers, multi tenancy etc. (prominent example would be a web harvester). In this scenario, no change at middleware would be required, or adding extra machinery to the indexing chain, like Logstash, or JDBC feed mode, because all is hosted within ES JVM. So in the end it is all a matter of style how to index docs. Both push or pull approaches do not interfere. Jörg On Sun, Aug 31, 2014 at 6:10 AM, Lukáš Vlček <[email protected]> wrote: > Nitin, > > If I am not mistaken rivers in ES will be deprecated. So in any case river > is not a good way to go for design of future application. We are using > rivers quite a lot in our project and we will be moving the functionality > to middleware layer. > > Regards, > Lukáš > Dne 31.8.2014 3:58 "Nitin Maheshwari" <[email protected]> napsal(a): > > Thanks for the reference, its helpful. >> >> >> On Thu, Aug 28, 2014 at 12:33 PM, [email protected] < >> [email protected]> wrote: >> >>> I work on such a component which can handle jobs and delegate them to >>> other nodes but no working implementation available yet. >>> >>> For a service component that is able to maintain state in the cluster >>> state, see RiverState class in JDBC plugin. >>> >>> Jörg >>> >>> >>> On Thu, Aug 28, 2014 at 8:08 AM, Nitin Maheshwari <[email protected]> >>> wrote: >>> >>>> Thanks Jörg for your quick and timely response. >>>> >>>> I am new to ES, can you point me to any reference implementation for >>>> the ES service component? >>>> >>>> Once again thanks for the help. >>>> >>>> Nitin >>>> >>>> >>>> On Tuesday, 26 August 2014 15:07:05 UTC+5:30, Jörg Prante wrote: >>>> >>>>> For multi tenant, the river concept is awkward. River is a singleton >>>>> and is bound to single user execution, and you are right, creating river >>>>> instances per DB and per index does not scale. >>>>> >>>>> There are several options: >>>>> >>>>> - write a more sophisticated plugin which acts as a service and not as >>>>> a singleton. The ES service component, which would maintain state in the >>>>> cluster state, could accept job requests where each job request is >>>>> equivalent to a JDBC pull. The job requests are delegated to a node which >>>>> is not very busy with jobs (load balancing). The code of the JDBC river >>>>> can >>>>> be reused for that. >>>>> >>>>> - write a separate middleware for your tenants where they can have >>>>> separate access to the DB and prepare ES JSON bulk files from (maybe be by >>>>> REST API calls similar in style to ES). This would be a domain specific >>>>> solution but offers most flexibility to the tenants, they are free to >>>>> decide how and when to create and index the data from DB. >>>>> >>>>> Jörg >>>>> >>>>> >>>>> >>>>> >>>>> On Tue, Aug 26, 2014 at 11:21 AM, Nitin Maheshwari <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Jörg, >>>>>> >>>>>> I am working on a multi tenant application where each tenant has its >>>>>> own database. I am planning to use ES for indexing the data, and JDBC >>>>>> river >>>>>> for doing periodic bulk indexing. I do not want to create one river per >>>>>> DB >>>>>> per object type. This will lead to too many rivers. >>>>>> >>>>>> I wanted to modify the JDBC river so that I can give parent DB >>>>>> location, where all tenant db connection information is available. And >>>>>> then >>>>>> inside the river, modify it such that a feader thread is created for each >>>>>> river. >>>>>> >>>>>> Do you see any issue with this or do you have any other >>>>>> recommendation? >>>>>> >>>>>> Thanks, >>>>>> Nitin >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elasticsearch" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> >>>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>>> msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473% >>>>>> 40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/elasticsearch/771fc3a8-2203-4db8-a07b-067430e7a473%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com >>>> <https://groups.google.com/d/msgid/elasticsearch/76780b26-5dfa-4de4-9d14-5e1a9cacb2d3%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/elasticsearch/w88yWcbx4lM/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGTXjgHFeV9ip4n0vO_TJNthcvD1RK-pHWruwL0gx_Rew%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGTXjgHFeV9ip4n0vO_TJNthcvD1RK-pHWruwL0gx_Rew%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Nitin (Nits) >> http://nitinmaheshwari.in >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAHDjFTEe5cxJMX6v%3DS1HzGATmnyA7jvHa-EKEQhE%3Dv5E8Nuvjg%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAHDjFTEe5cxJMX6v%3DS1HzGATmnyA7jvHa-EKEQhE%3Dv5E8Nuvjg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAO9cvUZzsJpeZ5ZDKsAWp8Ko6hpOPjRh4nowTh9nbjrPUn9tfw%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAO9cvUZzsJpeZ5ZDKsAWp8Ko6hpOPjRh4nowTh9nbjrPUn9tfw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHxPCeYiMj4wznCaLyJrzJ%3DcHwpzfN1C5_%2Bs_wnmy1RxA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
