RE: Using a cache as an affinity co-located processing buffer in Ignite.Net

Are cache interceptors an internal Ignite construct not available in the Java client? *From:* Pavel Tupitsyn [mailto:ptupit...@apache.org] *Sent:* Friday, May 18, 2018 1:53 AM *To:* user@ignite.apache.org *Subject:* Re: Using a cache as an affinity co-located processing buffer in Ignite.Net

RE: Using a cache as an affinity co-located processing buffer in Ignite.Net

Hi Dmitriy, My question regarding cache interceptor availability in .Net is still outstanding. I have working code using a continuous query, but am interested in cache interceptor if this is a better solution. Thanks. Raymond. *From:* Dmitry Pavlov [mailto:dpavlov@gmail.com]

Re: Using a cache as an affinity co-located processing buffer in Ignite.Net

Hi Raymond, Was this question answered? Sincerely, Dmitriy Pavlov вт, 1 мая 2018 г. в 0:20, Raymond Wilson : > Cross posting to dev list for comment on cache interceptor availability on > Ignite .Net client. > > -Original Message- > From: Raymond Wilson

RE: Using a cache as an affinity co-located processing buffer in Ignite.Net

Cross posting to dev list for comment on cache interceptor availability on Ignite .Net client. -Original Message- From: Raymond Wilson [mailto:raymond_wil...@trimble.com] Sent: Saturday, April 28, 2018 10:35 PM To: 'user@ignite.apache.org' Subject: RE: Using a

RE: Using a cache as an affinity co-located processing buffer in Ignite.Net

Further investigation shows CacheInterceptor is not a part of the Ignite.NET API. Is there a plan/ticket for this to be done? -Original Message- From: Raymond Wilson [mailto:raymond_wil...@trimble.com] Sent: Saturday, April 28, 2018 1:08 PM To: user@ignite.apache.org Subject: Re: Using a

Re: Using a cache as an affinity co-located processing buffer in Ignite.Net

Val, Are the interceptors invoked in the affinity co-located context of the item? The help is a little unclear on that. Thanks, Raymond. Sent from my iPhone > On 28/04/2018, at 12:12 PM, vkulichenko wrote: > > Raymond, > > If you go with approach I described

Re: Using a cache as an affinity co-located processing buffer in Ignite.Net

Raymond, If you go with approach I described above, I would actually recommend to use interceptors: https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/CacheInterceptor.html Continuous query seems to be a bit cumbersome for this. -Val -- Sent from:

Re: Using a cache as an affinity co-located processing buffer in Ignite.Net

If I use local continuous queries can't I omit the remote filter and just use the local delivery handler for the continuous query? Sent from my iPhone > On 28/04/2018, at 9:43 AM, vkulichenko wrote: > > Raymond, > > It sounds like you want to run certain

RE: Using a cache as an affinity co-located processing buffer in Ignite.Net

Raymond, It sounds like you want to run certain computation on every data update in the cache, is that right? To achieve that you can use local continuous queries, but: - Remote filter would be executed on both primary and backup, so computation will be executed more than once. - You can filter

RE: Using a cache as an affinity co-located processing buffer in Ignite.Net

Val, Thanks for the feedback. Essentially what I really want in Ignite.Net version of the Ignite distributed Queue with local consumers on each server reading and processing just the elements that server is responsible for (which may be a slight mangling of queue semantics...) I have written a

Re: Using a cache as an affinity co-located processing buffer in Ignite.Net

Val, What I really want is a .Net implementation of the Ignite distributed queue, assuming co-located server processing can pull members off the queue that are co-located with that node (yeah, that's probably mangling queue semantics a bit, but not that badly :) ) I have previously made a POC

RE: Using a cache as an affinity co-located processing buffer in Ignite.Net

Raymond, If you want to have a single continuous query on one of the server nodes, then I think singleton service is the best option to achieve that. It will not only guarantee that there is one query at a time, but will also make sure to redeploy it in case of failure. Also I would be accurate

RE: Using a cache as an affinity co-located processing buffer in Ignite.Net

Getting back to best practices… I now have some code that can create and establish the continuous query that will inject the filters into the remote nodes to perform the processing work on the new items arriving in the cache. What is the usual metaphor for managing the lifecycle of such a

Re: Using a cache as an affinity co-located processing buffer in Ignite.Net

There will be two instances, and these instances will have different lifetimes. > it seems weird that there are two different interfaces for essentially the same purpose They are a bit different, one operates on ICacheEntry, another on ICacheEntryEvent (which carries additional data) On Wed,

RE: Using a cache as an affinity co-located processing buffer in Ignite.Net

Let’s say I do this, where I define a single class FileFilter that implements both remote filter interfaces (BTW, it seems weird that there are two different interfaces for essentially the same purpose): public class RemoteFileFilter : ICacheEntryFilter

Re: Using a cache as an affinity co-located processing buffer in Ignite.Net

What do you mean by "instance"? In terms of CLR that would be a different instance on every node. On Wed, Apr 25, 2018 at 2:50 AM, Raymond Wilson wrote: > I’m using a Continuous Query in both options (grid deployed service using > a CQ versus an independent context

RE: Using a cache as an affinity co-located processing buffer in Ignite.Net

I’m using a Continuous Query in both options (grid deployed service using a CQ versus an independent context using a CQ). I was curious which context using a CQ would be seen as desirable. In the case where a filter is provided to a CQ for both the initial query and for newly items arriving in

Re: Using a cache as an affinity co-located processing buffer in Ignite.Net

ContinuousQuery is the best practice for most kinds of streaming use cases. I think it fits your use case as well. On Tue, Apr 24, 2018 at 10:08 AM, Raymond Wilson wrote: > Thanks, that makes sense. > > From a best practices perspective, is better to have a grid

Re: Using a cache as an affinity co-located processing buffer in Ignite.Net

Thanks, that makes sense. From a best practices perspective, is better to have a grid deployed service on each node executing local continuous queries against the cache and orchestrating the processing from within the service, versus having some singular context in the grid that uses the

Re: Using a cache as an affinity co-located processing buffer in Ignite.Net

Sorry, looks like I have misunderstood you. If you need initial scan, of course you can have it by using ScanQuery as initialQuery. Place all the processing logic into the ScanQuery filter, and return false from there. This way you can process all existing entries in a co-located fashion without

RE: Using a cache as an affinity co-located processing buffer in Ignite.Net

Not being able to do an initial scan of elements on the remote nodes is a bit of a problem (possibly a bug?) Something that’s occurred to me is to wrap this behaviour into an Ignite service deployed onto all of the server nodes, and use a local mode continuous query from within each service to

Re: Using a cache as an affinity co-located processing buffer in Ignite.Net

> Is the initial query also run in the context of the remote node and the remote filter? No, it is just a query (can be SQL or Scan) which allows you to get a "full picture" on the calling node: all existing data and all future data. So in your scenario it is not very useful. > return false

Re: Using a cache as an affinity co-located processing buffer in Ignite.Net

Remote Listener is deployed on every cache node and is invoked only on a primary node for that key. In other words, for each key there is only one invocation of the remote filter, and that invocation is local to that key. So you can place your processing logic into the Remote Filter. On Mon, Apr

RE: Using a cache as an affinity co-located processing buffer in Ignite.Net

Hi Pavel, Yes, I looked at continuous queries. They appear to be oriented toward a single context being sent the newly arrived elements in the cache from all primary nodes hosting the cache involved in the query. In the use case I outlined below, I would like to have the items processed in

Re: Using a cache as an affinity co-located processing buffer in Ignite.Net

Hi Raymond, To process incoming data in a co-located fashion there is a Continuous Query feature [1]. Looks like it fits your use case quite well. [1] https://apacheignite-net.readme.io/docs/continuous-queries On Mon, Apr 23, 2018 at 7:32 AM, Raymond Wilson wrote:

RE: Using a cache as an affinity co-located processing buffer in Ignite.Net

I did find ICache.GetLocalEntries() method and have written the following as a proof of concept (yet to exercise it though): IEnumerable> localItems = QueueCache.GetLocalEntries(new [] {CachePeekMode.Primary});