Re: Continuous query with Thin client

There was a mistake in the release notes, I've just fixed it. Continuous queries were added to .NET thin client for 2.10, and Java thin client gets them in 2.11. On Mon, Apr 19, 2021 at 12:00 PM Stephen Darlington < stephen.darling...@gridgain.com> wrote: > I suspect it’s the backend changes

Re: Continuous query with Thin client

I suspect it’s the backend changes required to support continuous queries in thin-clients. > On 19 Apr 2021, at 07:59, sourav dihidar wrote: > > Thanks for your reply. I see in the release note of 2.10.0 under Java thin > client , it is mentioned > > Added thin Client Continuous Query > Is

Re: Continuous query with Thin client

Thanks for your reply. I see in the release note of 2.10.0 under Java thin client , it is mentioned - Added thin Client Continuous Query Is that something else? Thanks On Fri, Apr 16, 2021, 10:11 PM Stephen Darlington < stephen.darling...@gridgain.com> wrote: > Looks like that’s coming in

Re: Continuous query with Thin client

Looks like that’s coming in 2.11: IGNITE-14402 It’s present in the .net thin client since 2.10. Regards, Stephen > On 16 Apr 2021, at 17:21, sourav dihidar wrote: > > Hi Team, > > I understood that continuous query for java thin client

Re: Continuous query not transactional ?

Hi Ilya, That is what I assume too, could someone from the developers community help confirm/comment on this ? regards, Veena. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Continuous query not transactional ?

Hello! I'm not sure, but I would assume that changes are visible after commit(), but you can see these changes in any order, and you can see cache a update without cache b update, for example. This is for committed transactions. For rolled back transactions, I don't know. I expect you won't be

Re: Continuous Query

Thanks Raymond for the pointer. On Wed, Oct 7, 2020 at 4:39 PM Raymond Wilson wrote: > It's possible to configure the continuous query to return only keys in the > cache that are stored on the local node. > > On the C# client we do it like this: > > var _queryHandle = queueCache.QueryContinuous

Re: Continuous Query

It's possible to configure the continuous query to return only keys in the cache that are stored on the local node. On the C# client we do it like this: var _queryHandle = queueCache.QueryContinuous (qry: new ContinuousQuery<[..KeyType..], [...ValueType...]>(listener) {Local = true},

Re: Continuous Query

Thanks for the detailed explanation. I don't know how efficient it would be if you have to filter each record one by one and then update each record, three times, to keep track of the status, if you're dealing with millions of records each hour, even if the cache is partitioned. I guess I will

Re: Continuous Query

I recalled a complete solution. That's what you would need to do if decide to process records in real-time with continuous queries in the *fault-tolerant fashion* (using pseudo-code rather than actual Ignite APIs). First. You need to add a flag field to the record's class that keeps the current

Re: Continuous Query

> > So, a lesson for the future, would the continuous query approach still be > preferable if the calculation involves the cache with continuous query and > say a look up table? For example, if I want to see the country in the cache > employee exists in the list of the countries that I am

Re: Continuous Query

Also, what do I need to do if I want the filter for the continuous query to execute on the cache on the local node only? Say, I have the continuous query deployed as singleton service on each node, to capture certain changes to a cache on the local node. On Wed, Oct 7, 2020 at 5:54 AM narges

Re: Continuous Query

Thank you, Denis. So, a lesson for the future, would the continuous query approach still be preferable if the calculation involves the cache with continuous query and say a look up table? For example, if I want to see the country in the cache employee exists in the list of the countries that I am

Re: Continuous Query

Thanks Then, I would consider the continuous queries based solution as long as the records can be updated in real-time: - You can process the records on the fly and don't need to come up with any batch task. - The continuous query filter will be executed once on a node that stores

Re: Continuous Query

Denis The calculation itself doesn't involve an update or read of another record, but based on the outcome of the calculation, the process might make changes in some other tables. thanks. On Mon, Oct 5, 2020 at 7:04 PM Denis Magda wrote: > Good. Another clarification: > >- Does that

Re: Continuous Query

Good. Another clarification: - Does that calculation change the state of the record (updates any fields)? - Does the calculation read or update any other records? - Denis On Sat, Oct 3, 2020 at 1:34 PM narges saleh wrote: > The latter; the server needs to perform some calculations

Re: Continuous Query

2020 2:03 AM > *To:* user@ignite.apache.org > *Subject:* Re: Continuous Query > > The latter; the server needs to perform some calculations on the data > without sending any notification to the app. > > On Fri, Oct 2, 2020 at 4:25 PM Denis Magda wrote: > > And after you detec

Re: Continuous Query

unsubscribe From: narges saleh Sent: Sunday, 4 October 2020 2:03 AM To: user@ignite.apache.org Subject: Re: Continuous Query The latter; the server needs to perform some calculations on the data without sending any notification to the app. On Fri, Oct 2, 2020

Re: Continuous Query

The latter; the server needs to perform some calculations on the data without sending any notification to the app. On Fri, Oct 2, 2020 at 4:25 PM Denis Magda wrote: > And after you detect a record that satisfies the condition, do you need to > send any notification to the application? Or is it

Re: Continuous Query

And after you detect a record that satisfies the condition, do you need to send any notification to the application? Or is it more like a server detects and does some calculation logically without updating the app. - Denis On Fri, Oct 2, 2020 at 11:22 AM narges saleh wrote: > The detection

Re: Continuous Query

The detection should happen at most a couple of minutes after a record is inserted in the cache but all the detections are local to the node. But some records with the current timestamp might show up in the system with big delays. On Fri, Oct 2, 2020 at 12:23 PM Denis Magda wrote: > What are

Re: Continuous Query

What are your requirements? Do you need to process the records as soon as they are put into the cluster? On Friday, October 2, 2020, narges saleh wrote: > Thank you Dennis for the reply. > From the perspective of performance/resource overhead and reliability, > which approach is preferable?

Re: Continuous Query

Thank you Dennis for the reply. >From the perspective of performance/resource overhead and reliability, which approach is preferable? Does a continuous query based approach impose a lot more overhead? On Fri, Oct 2, 2020 at 9:52 AM Denis Magda wrote: > Hi Narges, > > Use continuous queries if

Re: Continuous Query

Hi Narges, Use continuous queries if you need to be notified in real-time, i.e. 1) a record is inserted, 2) the continuous filter confirms the record's time satisfies your condition, 3) the continuous queries notifies your application that does require processing. The jobs are better for a

Re: Continuous Query on a varying set of keys

Hello! Yes, using other continuous query to watch changes of this set looks OK. Of course there will be some yak shaving to do. Regards, -- Ilya Kasnacheev вт, 26 мая 2020 г. в 12:28, zork : > Thanks. > So I can think of two ways using which such a set could be maintained by > the > remote

Re: Continuous Query on a varying set of keys

Thanks. So I can think of two ways using which such a set could be maintained by the remote node: 1. The remote node listens to a new topic through which the local node sends it a message whenever the set changes. 2. Or, the local node puts the set values in a new table in the cache itself and

Re: Continuous Query on a varying set of keys

Hello! No, updating local set will not be sent over to the remote node, but on the remote node you can probably update it from your filter: qry.setRemoteFilterFactory(new Factory>() { @Override public CacheEntryEventFilter create() { return new

Re: Continuous Query on a varying set of keys

Hi, Sorry but I could not get it to work. The standard way of defining a remote filter as shown in sample repo is something like: qry.setRemoteFilterFactory(new Factory>() { @Override public CacheEntryEventFilter create() { return new

Re: Continuous Query on a varying set of keys

Hello! Remote filter is code. It can execute arbitrary logic. It can adjust to what it needs to filter, change its own behavior with time. Regards, -- Ilya Kasnacheev пн, 18 мая 2020 г. в 15:40, zork : > Hi Ilya, > Thanks for your response. > I'm aware of remote filters but can these filters

Re: Continuous Query on a varying set of keys

Hi Ilya, Thanks for your response. I'm aware of remote filters but can these filters be modified once the query is already attached? Because if not, then this would not solve my use case as the filter would always give me updates on a fixed subset of keys, however in my case this subset is varying

Re: Continuous Query on a varying set of keys

Hello! Continuous query has a notion of 'remote filter'. This is a piece of code which is executed near data (on server nodes) to determine if the update needs to be sent over network. https://apacheignite.readme.io/docs/continuous-queries#remote-filter If you define a proper remote filter,

Re: Continuous query setPageSize and setTimeInterval question

It is likely entries are being quickly produced and filling up the buffer giving the effect of an immediate update. You can test it via specific delays and logging. Make the server print out the counter of the object. say: if the key is an integer and the value is too then print out the key as

Re: Continuous query setPageSize and setTimeInterval question

This was what I thought, however, no matter what number I set to the pageSize, e.g., 5, 20, 100, my local listener got the update immediately, 1 by 1, but not batching by the page size... -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Continuous query setPageSize and setTimeInterval question

You are correct. Your local reciever will get an update every 100 entries. setPageSize sets the number of entries to batch together before sending. When the server has accumulated a number of entries larger than getPageSize it sends a message to the receiver. Like I mentioned before,

Re: Continuous query setPageSize and setTimeInterval question

Thanks Alex, But I still not quite understand the expected behaviour. If I set the page size to 100 and interval to 0, should the local query be triggered for every 100 updates? Regards, Rick -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Continuous query setPageSize and setTimeInterval question

Hi, setTimeInterval limits the time Ignite will wait for an internal buffer to fill up before sending. It is not a time delay setting. From the doc: https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/query/ContinuousQuery.html#setTimeInterval-long- When a

Re: Continuous Query Questions

So, if I define the CQ as a service, and the node crashes, wouldn't ignite start a new service with CQ already registered, say if the CQ registration is in service init? I can do initial query at the as well. On Wed, Apr 1, 2020 at 7:33 PM Evgenii Zhuravlev wrote: > Well, with this use case,

Re: Continuous Query Questions

Well, with this use case, if one of the nodes goes down, there is always a chance to lost notifications. I don't think that it's possible to recover lost notifications with out of the box solution, but if you will be able to track the last processed notification and store update time in entries,

Re: Continuous Query Questions

Thanks Evgenii for the recommendation and the heads up. Is there a way to recover the lost notifications or even know if a notification is lost? On Wed, Apr 1, 2020 at 12:15 PM Evgenii Zhuravlev wrote: > Hi, > > 1) I would recommend checking ContinouousQuery.setLocal: >

Re: Continuous Query Questions

Hi, 1) I would recommend checking ContinouousQuery.setLocal: https://www.gridgain.com/sdk/ce/latest/javadoc/org/apache/ignite/cache/query/Query.html#setLocal-boolean-. Please check if it fits your requirements. 2) You will need to do this in a separate thread, because cache operations shouldn't

Re: Continuous Query Questions

I'm not sure, because the final overhead depends on the object sizes. There is a buffer for CQ, which stores 1000 entries by default, but you can decrease it using property IGNITE_CONTINUOUS_QUERY_SERVER_BUFFER_SIZE Evgenii вт, 18 февр. 2020 г. в 18:09, narges saleh : > Hi Evgeni, > > There

Re: Continuous Query Questions

Hi Evgeni, There will be several thousands notifications/day if I have it send notification only when certain patterns are visited, in about 100+ caches, which brings up another question: wouldn't having 100+ CQs be creating too much overhead? thanks. On Tue, Feb 18, 2020 at 2:17 PM Evgenii

Re: Continuous Query Questions

Hi, How many notifications do you want to get? If it's just a several notifications, then you can even register separate CQ for each of the entry with its own remote filters. At the same time, if you have a requirement to send these notifications for a lot of entries, then this approach will

Re: Continuous query order on transactional cache

Hello! Why? I don't think that transactions, semantically have any guarantees about the order of updates inside a transaction. I'd go with A). Regards, -- Ilya Kasnacheev чт, 16 янв. 2020 г. в 17:39, Barney Pippin : > Hi, > > If I have a continuous query running and it's listening to a

Re: Continuous Query from client app

Fixed it! I needed to put my thread to sleep while the query was open, like this: try (QueryCursor> cur = cityCache.query(qry)) { // Iterating over existing data stored in cache. for (Cache.Entry e : cur) System.out.println("key=" + e.getKey() + ", val=" + e.getValue()); boolean done =

RE: Continuous query - Exactly once based event across multiple nodes..

.com] On Behalf Of Nikolay Izhikov Sent: Monday, May 7, 2018 6:40 PM To: user@ignite.apache.org Cc: JP Subject: Re: Continuous query - Exactly once based event across multiple nodes.. Hello, JP. You should use target node in remote filter. You should check "Is primary node for some record

Re: Continuous query - Exactly once based event across multiple nodes..

JP, Do you have a solution for this? Do you need any more help? -Val -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Continuous query - Exactly once based event across multiple nodes..

Vkulichenko, I want to update in multiple databases based on event triggered in the ignite cache. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Continuous query - Exactly once based event across multiple nodes..

JP, Can you please describe the business case behind this? What are you trying to achieve on application level? What guarantees are needed and why? -Val -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Continuous query - Exactly once based event across multiple nodes..

Thanks... This solution worked but problem is, it creating multiple remote filter instance.. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: continuous query - changes from local server only

Som, You could create the continuous query on each client node with the filter described above. 2018-02-08 19:55 GMT+03:00 Som Som <2av10...@gmail.com>: > i've got both client and server nodes on each of 3 physical servers, that > is my cluster. there is a partitioned cache, each server node

Re: continuous query - changes from local server only

i've got both client and server nodes on each of 3 physical servers, that is my cluster. there is a partitioned cache, each server node stores only a part of keys. i start the application on my dev machine that app is also client of the cluster further i put new key into the cluster. i would like

Re: continuous query - changes from local server only

Hi, You may fuse filter for that, for example: ContinuousQuery qry = new ContinuousQuery<>(); final Set nodes = new HashSet<>(client.cluster().forDataNodes("cache") .forHost(client.cluster().localNode()).nodes()); qry.setRemoteFilterFactory(new

Re: Continuous Query remote listener misses some events or respond really late

Hi.. I know its quite late to reply, But I am seeing this issue intermittently almost everyday. But can't reproduce it locally on dev machine. As suggested I have moved logs before null check to see if null event is logged. However, I didn't see it printed in logs. Also, it was suggested to check

Re: Continuous Query on Multiple caches

Hi Rishikesh, Is it possible to create another kafka stream based on Curr_stream1 & Curr_stream2? In this case, you will be able to stream (Curr_stream1.f0 - Curr_stream2.f0) into a new Ignite cache and use continuous query. In any way, it would be great if you can share your solution with the

Re: Continuous Query event buffering OOME

Hi Michal, I've looked at code and your points look reasonable. In now moment, how you correct noted, you can decrease size of the buffer via IGNITE_CONTINUOUS_QUERY_SERVER_BUFFER_SIZE property to 50 or 100. On Tue, Sep 5, 2017 at 9:14 PM, mcherkasov wrote: > Hi

Re: Continuous Query event buffering OOME

Hi Michal, Those buffers are required to make sure that all messages are delivered to all subscribers and delivered in right order. However I agree, 1M is a relatively large number for this. I will check this question with Continuous Query experts and will update you tomorrow. Thanks, Mikhail.

Re: Continuous Query on Multiple caches

Hi, In our case data is coming from 2 kafka streams. We want to compare current data from 2 streams and take some action(e.g. raise alert). We want to make this processing event based i.e. as soon as data comes from 2 streams, we should take action associated with this event. For example,

Re: Continuous Query on Multiple caches

Hi Rishikesh, ContinuosQuery is designed to work with single cache only. So, there is no way to use it with multiple caches. Could you please share your use case in more details? Thanks. -- View this message in context:

Re: Continuous Query remote listener misses some events or respond really late

Thank for your reply. From code I see that you log only entries with non null values. If your absolutely shure that you never put null in cache - I will create loadtest to reproduce it and create issue for you. But it will be great, if you move logging before event.getValue! = null. среда, 7 июня

Re: Continuous Query remote listener misses some events or respond really late

As far as I understant you create CQ in Service.init, so node with running service is CQ node. All other nodes from grid will send CQ events to this node to process in your service and if you don't configure nodeFilter for service - any node can run it, so any node can be CQ node. But it shouldn't

Re: Continuous Query remote listener misses some events or respond really late

Thanks, In my application, all nodes are server nodes And how do we be sure that nodes removed/ reconnect to grid is CQ node, it can be any. Also, Is this issue possible in all below scenarios? 1. if node happens to be CQ node or any node? 2. node is removed from grid forcefully(manual shutdown)

Re: Continuous Query remote listener misses some events or respond really late

If node with CQ leave grid (or just reconnect to grid, if it client node) - you should recreate CQ, because some cache updates can happen when node with CQ listener can't receive it. What happen it this case: 1) Node with changed cache entry process CQ, entry pass remote filter and node try to

Re: Continuous Query remote listener misses some events or respond really late

Umm. actually nothing get logged in such scenario. However, as you indicated earlier, I could see trades get stuck if a node leaves the grid(not always). Do you know why that happens? Is that a bug? -- View this message in context:

Re: Continuous Query remote listener misses some events or respond really late

Can you share you log files? 2017-05-03 19:05 GMT+07:00 begineer : > 1) How you use ContinuousQuery: with initialQuery or without? : with > initial > query having same predicate > 2) Did some nodes disconnect when you loose updates? no > 3) Did you log entries in

Re: Continuous Query remote listener misses some events or respond really late

1) How you use ContinuousQuery: with initialQuery or without? : with initial query having same predicate 2) Did some nodes disconnect when you loose updates? no 3) Did you log entries in CQ.localListener? Just to be sure that error in CQ logic, not in your service logic. : No log entries in

Re: Continuous Query remote listener misses some events or respond really late

1) How you use ContinuousQuery: with initialQuery or without? 2) Did some nodes disconnect when you loose updates? 3) Did you log entries in CQ.localListener? Just to be sure that error in CQ logic, not in your service logic. 4) Can someone update old entries? Maybe they just get into CQ again

Re: Continuous Query remote listener misses some events or respond really late

Hi Thanks for looking into this. Its not easily reproduce-able. I only see it some times. Here is my cache and service configuration Cache configuration: readThrough="true" writeThrough="true" writeBehindEnabled="true" writeBehindFlushThreadCount="5" backups="1" readFromBackup="true" service

Re: Continuous Query remote listener misses some events or respond really late

Hi, I'm trying to reproduce it in one host (with 6 ignite server node) but all work fine for me. Can you share ignite configuration, cache configuration, logs or some reproducer? 2017-05-02 15:48 GMT+07:00 begineer : > Hi, > I am currently facing intermittent issue with

Re: Continuous Query

In my case the remote filters are being called and return true appropriately, however it seems that the local listeners are not always being called. I am trying to understand what could be going wrong, starting with ruling out a possible misunderstanding of the framework behavior regarding events

Re: Continuous Query

Hi Alex, All the local listeners that satisfy remote filters’ conditions will be notified. However there is one to one relationship between remote filter and local listener meaning that if you have 4 CQs then 4 remote filters will be executed and 4 local listeners notified if needed. What