Also, what do I need to do if I want the filter for the continuous query to execute on the cache on the local node only? Say, I have the continuous query deployed as singleton service on each node, to capture certain changes to a cache on the local node.
On Wed, Oct 7, 2020 at 5:54 AM narges saleh <snarges...@gmail.com> wrote: > Thank you, Denis. > So, a lesson for the future, would the continuous query approach still be > preferable if the calculation involves the cache with continuous query and > say a look up table? For example, if I want to see the country in the cache > employee exists in the list of the countries that I am interested in. > > On Tue, Oct 6, 2020 at 4:11 PM Denis Magda <dma...@apache.org> wrote: > >> Thanks >> >> Then, I would consider the continuous queries based solution as long as >> the records can be updated in real-time: >> >> - You can process the records on the fly and don't need to come up >> with any batch task. >> - The continuous query filter will be executed once on a node that >> stores the record's primary copy. If the primary node fails in the middle >> of the filter's calculation execution, then the filter will be executed on >> a backup node. So, you will not lose any updates but might need to >> introduce some logic/flag that confirms the calculation is not executed >> twice for a single record (this can happen if the primary node failed in >> the middle of the calculation execution and then the backup node picked up >> and started executing the calculation from scratch). >> - Updates of other tables or records from within the continuous query >> filter must go through an async thread pool. You need to use >> IgniteAsyncCallback annotation for that: >> >> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/lang/IgniteAsyncCallback.html >> >> Alternatively, you can always run the calculation in the batch-fashion: >> >> - Run a compute task once in a while >> - Read all the latest records that satisfy the requests with SQL or >> any other APIs >> - Complete the calculation, mark already processed records just in >> case if everything is failed in the middle and you need to run the >> calculation from scratch >> >> >> - >> Denis >> >> >> On Mon, Oct 5, 2020 at 8:33 PM narges saleh <snarges...@gmail.com> wrote: >> >>> Denis >>> The calculation itself doesn't involve an update or read of another >>> record, but based on the outcome of the calculation, the process might make >>> changes in some other tables. >>> >>> thanks. >>> >>> On Mon, Oct 5, 2020 at 7:04 PM Denis Magda <dma...@apache.org> wrote: >>> >>>> Good. Another clarification: >>>> >>>> - Does that calculation change the state of the record (updates any >>>> fields)? >>>> - Does the calculation read or update any other records? >>>> >>>> - >>>> Denis >>>> >>>> >>>> On Sat, Oct 3, 2020 at 1:34 PM narges saleh <snarges...@gmail.com> >>>> wrote: >>>> >>>>> The latter; the server needs to perform some calculations on the data >>>>> without sending any notification to the app. >>>>> >>>>> On Fri, Oct 2, 2020 at 4:25 PM Denis Magda <dma...@apache.org> wrote: >>>>> >>>>>> And after you detect a record that satisfies the condition, do you >>>>>> need to send any notification to the application? Or is it more like a >>>>>> server detects and does some calculation logically without updating the >>>>>> app. >>>>>> >>>>>> - >>>>>> Denis >>>>>> >>>>>> >>>>>> On Fri, Oct 2, 2020 at 11:22 AM narges saleh <snarges...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> The detection should happen at most a couple of minutes after a >>>>>>> record is inserted in the cache but all the detections are local to the >>>>>>> node. But some records with the current timestamp might show up in the >>>>>>> system with big delays. >>>>>>> >>>>>>> On Fri, Oct 2, 2020 at 12:23 PM Denis Magda <dma...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> What are your requirements? Do you need to process the records as >>>>>>>> soon as they are put into the cluster? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Friday, October 2, 2020, narges saleh <snarges...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thank you Dennis for the reply. >>>>>>>>> From the perspective of performance/resource overhead and >>>>>>>>> reliability, which approach is preferable? Does a continuous query >>>>>>>>> based >>>>>>>>> approach impose a lot more overhead? >>>>>>>>> >>>>>>>>> On Fri, Oct 2, 2020 at 9:52 AM Denis Magda <dma...@apache.org> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Narges, >>>>>>>>>> >>>>>>>>>> Use continuous queries if you need to be notified in real-time, >>>>>>>>>> i.e. 1) a record is inserted, 2) the continuous filter confirms the >>>>>>>>>> record's time satisfies your condition, 3) the continuous queries >>>>>>>>>> notifies >>>>>>>>>> your application that does require processing. >>>>>>>>>> >>>>>>>>>> The jobs are better for a batching use case when it's ok to >>>>>>>>>> process records together with some delay. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> - >>>>>>>>>> Denis >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Oct 2, 2020 at 3:50 AM narges saleh <snarges...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi All, >>>>>>>>>>> If I want to watch for a rolling timestamp pattern in all the >>>>>>>>>>> records that get inserted to all my caches, is it more efficient to >>>>>>>>>>> use >>>>>>>>>>> timer based jobs (that checks all the records in some interval) or >>>>>>>>>>> continuous queries that locally filter on the pattern? These >>>>>>>>>>> records can >>>>>>>>>>> get inserted in any order and some can arrive with delays. >>>>>>>>>>> An example is to watch for all the records whose timestamp ends >>>>>>>>>>> in 50, if the timestamp is in the format yyyy-mm-dd hh:mi. >>>>>>>>>>> >>>>>>>>>>> thanks >>>>>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> - >>>>>>>> Denis >>>>>>>> >>>>>>>>
