Thanks Saikat for the feedback. But if I use the overwrite option set to true to avoid duplicates in case I have to resend the entire payload in case of a streamer node failure, then I won't get optimal performance, right? What's the best practice for dealing with data streamer node failures? Are there examples?
On Thu, Jan 9, 2020 at 9:12 PM Saikat Maitra <[email protected]> wrote: > Hi, > > AFAIK, the DataStreamer check for presence of key and if it is present in > the cache then it does not allow overwrite of value if allowOverwrite is > set to false. > > Regards, > Saikat > > On Thu, Jan 9, 2020 at 6:04 AM narges saleh <[email protected]> wrote: > >> Thanks Andrei. >> >> If the external data source client sending batches of 2-3 MB say via TCP >> socket connection to a bunch of socket streamers (deployed as ignite >> services deployed to each ignite node) and say of the streamer nodes die, >> the data source client catching the exception, has to check the cache to >> see how much of the 2-4MB batch has been flushed to cache and resend the >> rest? Would setting streamer with overwrite set to true work, if the data >> source client resend the entire batch? >> A question regarding streamer with overwrite option set to true. How does >> the streamer compare the content the data in hand with the data in cache, >> if each record is being assigned UUID when being inserted to cache? >> >> >> On Tue, Jan 7, 2020 at 4:40 AM Andrei Aleksandrov < >> [email protected]> wrote: >> >>> Hi, >>> >>> Not flushed data in a data streamer will be lost. Data streamer works >>> thought some Ignite node and in case if this the node failed it can't >>> somehow start working with another one. So your application should think >>> about how to track that all data was loaded (wait for completion of >>> loading, catch the exceptions, check the cache sizes, etc) and use >>> another client for data loading in case if previous one was failed. >>> >>> BR, >>> Andrei >>> >>> 1/6/2020 2:37 AM, narges saleh пишет: >>> > Hi All, >>> > >>> > Another question regarding ignite's streamer. >>> > What happens to the data if the streamer node crashes before the >>> > buffer's content is flushed to the cache? Is the client responsible >>> > for making sure the data is persisted or ignite redirects the data to >>> > another node's streamer? >>> > >>> > thanks. >>> >>
