Partition counter is internal implemenattion detail, which has no sensible meaning to end users. It should not be exposed through public API.
On Thu, Dec 13, 2018 at 10:14 PM Denis Magda <dma...@apache.org> wrote: > Hello Piotr, > > That's a known problem and I thought a JIRA ticket already exists. However, > failed to locate it. The ticket for the improvement should be created as a > result of this conversation. > > Speaking of an initial query type, I would differentiate from ScanQueries > and SqlQueries. For the former, it sounds reasonable to apply the > partitionCounter logic. As for the latter, Vladimir Ozerov will it be > addressed as part of MVCC/Transactional SQL activities? > > Btw, Piotr what's your initial query type? > > -- > Denis > > On Thu, Dec 13, 2018 at 3:28 AM Piotr Romański <piotr.roman...@gmail.com> > wrote: > > > Hi, as suggested by Ilya here: > > > > > http://apache-ignite-users.70518.x6.nabble.com/Continuous-queries-and-duplicates-td25314.html > > I'm resending it to the developers list. > > > > From that thread we know that there might be duplicates between initial > > query results and listener entries received as part of continuous query. > > That means that users need to manually dedupe data. > > > > In my opinion the manual deduplication in some use cases may lead to > > possible memory problems on the client side. In order to remove > duplicated > > notifications which we are receiving in the local listener, we need to > keep > > all initial query results in memory (or at least their unique ids). > > Unfortunately, there is no way (is there?) to find a point in time when > we > > can be sure that no dups will arrive anymore. That would mean that we > need > > to keep that data indefinitely and use it every time a new notification > > arrives. In case of multiple continuous queries run from a single JVM, > this > > might eventually become a memory or performance problem. I can see the > > following possible improvements to Ignite: > > > > 1. The deduplication between initial query and incoming notification > could > > be done fully in Ignite. As far as I know there is already the > > updateCounter and partition id for all the objects so it could be used > > internally. > > > > 2. Add a guarantee that notifications arriving in the local listener > after > > query() method returns are not duplicates. This kind of functionality > would > > require a specific synchronization inside Ignite. It would also mean that > > the query() method cannot return before all potential duplicates are > > processed by a local listener what looks wrong. > > > > 3. Notify users that starting from a given notification they can be sure > > they will not receive any duplicates anymore. This could be an additional > > boolean flag in the CacheQueryEntryEvent. > > > > 4. CacheQueryEntryEvent already exposes the partitionUpdateCounter. > > Unfortunately we don't have this information for initial query results. > If > > we had, a client could manually deduplicate notifications and get rid of > > initial query results for a given partition after newer notifications > > arrive. Also it would be very convenient to expose partition id as well > but > > now we can figure it out using the affinity service. The assumption here > is > > that notifications are ordered by partitionUpdateCounter (is it true?). > > > > Please correct me if I'm missing anything. > > > > What do you think? > > > > Piotr > > >
