Thank you, Denis. So, a lesson for the future, would the continuous query approach still be preferable if the calculation involves the cache with continuous query and say a look up table? For example, if I want to see the country in the cache employee exists in the list of the countries that I am interested in.
On Tue, Oct 6, 2020 at 4:11 PM Denis Magda <dma...@apache.org> wrote: > Thanks > > Then, I would consider the continuous queries based solution as long as > the records can be updated in real-time: > > - You can process the records on the fly and don't need to come up > with any batch task. > - The continuous query filter will be executed once on a node that > stores the record's primary copy. If the primary node fails in the middle > of the filter's calculation execution, then the filter will be executed on > a backup node. So, you will not lose any updates but might need to > introduce some logic/flag that confirms the calculation is not executed > twice for a single record (this can happen if the primary node failed in > the middle of the calculation execution and then the backup node picked up > and started executing the calculation from scratch). > - Updates of other tables or records from within the continuous query > filter must go through an async thread pool. You need to use > IgniteAsyncCallback annotation for that: > > https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/lang/IgniteAsyncCallback.html > > Alternatively, you can always run the calculation in the batch-fashion: > > - Run a compute task once in a while > - Read all the latest records that satisfy the requests with SQL or > any other APIs > - Complete the calculation, mark already processed records just in > case if everything is failed in the middle and you need to run the > calculation from scratch > > > - > Denis > > > On Mon, Oct 5, 2020 at 8:33 PM narges saleh <snarges...@gmail.com> wrote: > >> Denis >> The calculation itself doesn't involve an update or read of another >> record, but based on the outcome of the calculation, the process might make >> changes in some other tables. >> >> thanks. >> >> On Mon, Oct 5, 2020 at 7:04 PM Denis Magda <dma...@apache.org> wrote: >> >>> Good. Another clarification: >>> >>> - Does that calculation change the state of the record (updates any >>> fields)? >>> - Does the calculation read or update any other records? >>> >>> - >>> Denis >>> >>> >>> On Sat, Oct 3, 2020 at 1:34 PM narges saleh <snarges...@gmail.com> >>> wrote: >>> >>>> The latter; the server needs to perform some calculations on the data >>>> without sending any notification to the app. >>>> >>>> On Fri, Oct 2, 2020 at 4:25 PM Denis Magda <dma...@apache.org> wrote: >>>> >>>>> And after you detect a record that satisfies the condition, do you >>>>> need to send any notification to the application? Or is it more like a >>>>> server detects and does some calculation logically without updating the >>>>> app. >>>>> >>>>> - >>>>> Denis >>>>> >>>>> >>>>> On Fri, Oct 2, 2020 at 11:22 AM narges saleh <snarges...@gmail.com> >>>>> wrote: >>>>> >>>>>> The detection should happen at most a couple of minutes after a >>>>>> record is inserted in the cache but all the detections are local to the >>>>>> node. But some records with the current timestamp might show up in the >>>>>> system with big delays. >>>>>> >>>>>> On Fri, Oct 2, 2020 at 12:23 PM Denis Magda <dma...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> What are your requirements? Do you need to process the records as >>>>>>> soon as they are put into the cluster? >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Friday, October 2, 2020, narges saleh <snarges...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Thank you Dennis for the reply. >>>>>>>> From the perspective of performance/resource overhead and >>>>>>>> reliability, which approach is preferable? Does a continuous query >>>>>>>> based >>>>>>>> approach impose a lot more overhead? >>>>>>>> >>>>>>>> On Fri, Oct 2, 2020 at 9:52 AM Denis Magda <dma...@apache.org> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Narges, >>>>>>>>> >>>>>>>>> Use continuous queries if you need to be notified in real-time, >>>>>>>>> i.e. 1) a record is inserted, 2) the continuous filter confirms the >>>>>>>>> record's time satisfies your condition, 3) the continuous queries >>>>>>>>> notifies >>>>>>>>> your application that does require processing. >>>>>>>>> >>>>>>>>> The jobs are better for a batching use case when it's ok to >>>>>>>>> process records together with some delay. >>>>>>>>> >>>>>>>>> >>>>>>>>> - >>>>>>>>> Denis >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Oct 2, 2020 at 3:50 AM narges saleh <snarges...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> If I want to watch for a rolling timestamp pattern in all the >>>>>>>>>> records that get inserted to all my caches, is it more efficient to >>>>>>>>>> use >>>>>>>>>> timer based jobs (that checks all the records in some interval) or >>>>>>>>>> continuous queries that locally filter on the pattern? These records >>>>>>>>>> can >>>>>>>>>> get inserted in any order and some can arrive with delays. >>>>>>>>>> An example is to watch for all the records whose timestamp ends >>>>>>>>>> in 50, if the timestamp is in the format yyyy-mm-dd hh:mi. >>>>>>>>>> >>>>>>>>>> thanks >>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> - >>>>>>> Denis >>>>>>> >>>>>>>
