Hello! You could write this message to developers list in a separate thread, see if there will be any discussion.
Regards, -- Ilya Kasnacheev ср, 5 дек. 2018 г. в 18:29, piotr.romanski <[email protected]>: > Hi all, I think that Krzysztof raised a valid concern. Actually, in my > opinion the manual deduplication in some use cases may lead to possible > memory problems on the client side. In order to remove duplicated > notifications which we are receiving in the local listener, we need to keep > all initial query results in memory (or at least their unique ids). > Unfortunately, there is no way (is there?) to find a point in time when we > can be sure that no dups will arrive anymore. That would mean that we need > to keep that data indefinitely and use it every time a new notification > arrives. In case of multiple continuous queries run from a single JVM, this > might eventually become a memory or performance problem. I can see the > following possible improvements to Ignite: > > 1. The deduplication between initial query and incoming notification could > be done fully in Ignite. As far as I know there is already the > updateCounter > and partition id for all the objects so it could be used internally. > > 2. Add a guarantee that notifications arriving in the local listener after > query() method returns are not duplicates. This kind of functionality would > require a specific synchronization inside Ignite. It would also mean that > the query() method cannot return before all potential duplicates are > processed by a local listener what looks wrong. > > 3. Notify users that starting from a given notification they can be sure > they will not receive any duplicates anymore. This could be an additional > boolean flag in the CacheQueryEntryEvent. > > 4. CacheQueryEntryEvent already exposes the partitionUpdateCounter. > Unfortunately we don't have this information for initial query results. If > we had, a client could manually deduplicate notifications and get rid of > initial query results for a given partition after newer notifications > arrive. Also it would be very convenient to expose partition id as well but > now we can figure it out using the affinity service. The assumption here is > that notifications are ordered by partitionUpdateCounter (is it true?). > > Please correct me if I'm missing anything. > > What do you think? > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
