Hi Ilya The data comes in huge batches of records (each burst can be up to 50-100 MB, which I plan to spread across multiple streamers) so, the streamer seems to be the way to go. Also, I don't want to establish a JDBC connection each time. So, if the streamer is the way to go, is it feasible to deploy it as a service? thanks.
On Mon, Feb 3, 2020 at 6:51 AM Ilya Kasnacheev <[email protected]> wrote: > Hello! > > Contrary to its name, data streamer is not actually suitable for > long-lived, low-intensity streaming. What it's good for is burst load of > large number of data in a short period of time. > > If your data arrives in large batches, you can use Data Streamer for each > batch. If not, better use Cache API. > > If you are worried that plain Cache API is slow, but also want failure > resilience, there's catch-22. The only way to make something resilient is > to put it into cache :) > > Regards, > -- > Ilya Kasnacheev > > > пн, 3 февр. 2020 г. в 14:34, narges saleh <[email protected]>: > >> Hi, >> But services are by definition long lived, right? Here is my layout: The >> data is continuously generated and sent to the streamer services (via JDBC >> connection with set streaming on option), deployed, say, as node singleton >> (actually deployed also as microservices) to load the data into the caches. >> The streamers do flush data based on some timers. >> If the streamer crashes before the buffer is flushed, the client catches >> the exception and resends the batch. Any issue with this layout? >> >> thanks. >> >> On Mon, Feb 3, 2020 at 5:02 AM Ilya Kasnacheev <[email protected]> >> wrote: >> >>> Hello! >>> >>> It is not recommended to have long-lived data streamers, it's best to >>> acquire it when it is needed. >>> >>> If you have to keep data streamer around, don't forget to flush() it. >>> This way you don't have to worry about its queue. >>> >>> Regards, >>> -- >>> Ilya Kasnacheev >>> >>> >>> пн, 3 февр. 2020 г. в 13:24, narges saleh <[email protected]>: >>> >>>> Hi, >>>> My specific question/concern is with regard to the state of the >>>> streamer when it run as a service, i.e. when it crashes and it gets >>>> redeployed. Specifically, what happens to the data? >>>> I have a similar question with regard to the state of a continuous >>>> query when it is deployed as a service, what happens to the data in the >>>> listener's queue? >>>> >>>> thanks. >>>> >>>> On Sun, Feb 2, 2020 at 4:18 PM Mikael <[email protected]> >>>> wrote: >>>> >>>>> Hi! >>>>> >>>>> Not as far as I know, I have a number of services using streamers >>>>> without any problems, do you have any specific problem with it ? >>>>> >>>>> Mikael >>>>> >>>>> >>>>> Den 2020-02-02 kl. 22:33, skrev narges saleh: >>>>> > Hi All, >>>>> > >>>>> > Is there a problem with running the datastreamer as a service, being >>>>> > instantiated in init method? Or loading the data via JDBC connection >>>>> > with streaming mode enabled? >>>>> > In either case, the deployment is affinity based. >>>>> > >>>>> > thanks. >>>>> >>>>
