RE: Question about persisting stream processing results
I believe that’s correct, although listening for javax.cache.event.EventType.EXPIRED in ContinuousQuery may be more straightforward. Also, I don’t think you need to disable eager TTL for that. Eager TTL is about when to pay for deletion of the old data – other words, if you specify a TTL with an ExpiryPolicy then your data will be removed sooner or later. If Eager TTL is enabled, you’ll be paying for the deletion periodically (when the cleaning thread wakes up), and if Eager TTL is disabled you’ll pay for it when a new query is executed. I’d suggest to try running both with and without Eager TTL and compare performance. Stan From: svonn Sent: 23 января 2018 г. 21:05 To: user@ignite.apache.org Subject: Re: Question about persisting stream processing results Hi, It's less about a specific code snipped, more about a conceptual question: How to keep a time window of 5 minutes in Offheap/Inmemory while storing anything older on the harddrive. @Stan: That sounds like possible solution! I'll just have to figure out how to catch, process and delete those properly. The docu for expiry policies says: "Eager TTL Entries that are expired can be removed from cache either eagerly or when they are touched by different cache operations. If there is at least one cache configured with eager TTL enabled, Ignite will create a single thread to clean up expired entries in the background." That "or when they are touched by different cache operations" sounds like I could disable Eager TTL and just set up: ignite.events().localListen(MyListener, EventType.EVT_CACHE_OBJECT_EXPIRED); And it would delete the cache entries after executing the listener, is that correct? - Svonn -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Question about persisting stream processing results
Hi, It's less about a specific code snipped, more about a conceptual question: How to keep a time window of 5 minutes in Offheap/Inmemory while storing anything older on the harddrive. @Stan: That sounds like possible solution! I'll just have to figure out how to catch, process and delete those properly. The docu for expiry policies says: "Eager TTL Entries that are expired can be removed from cache either eagerly or when they are touched by different cache operations. If there is at least one cache configured with eager TTL enabled, Ignite will create a single thread to clean up expired entries in the background." That "or when they are touched by different cache operations" sounds like I could disable Eager TTL and just set up: ignite.events().localListen(MyListener, EventType.EVT_CACHE_OBJECT_EXPIRED); And it would delete the cache entries after executing the listener, is that correct? - Svonn -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Question about persisting stream processing results
> This keeps the performance stable, but I'm not sure how to persist the > expired data properly Perhaps you could listen to the EXPIRED events of the cache that stores the data you want to persist, create an additional cache with persistence enabled and put all your expired data there. Thus you’ll have your current caches with persistence disabled (there are two of them, right?) storing relevant data that you execute queries for, and one more cache with persistence enabled that you’ll only write old data to. Haven’t tried that myself though. Stan From: svonn Sent: 22 января 2018 г. 20:34 To: user@ignite.apache.org Subject: Question about persisting stream processing results Hi! I'm receiving two streams of events, stream one (1) is basically only used as basis for interpolating and putting data in stream two (2). Whenever an element in stream (1) arrives, the local listener of my ContinuousQuery starting searching the previous element belonging to the same group. More specifically, its a ScanQuery that compares some IDs and searches for the one that has a timestamp bigger than the current one minus 1500ms while being smaller than the current timestamp. Currently, I want to persist stream (2) while keeping a stable performance. Whats the best way to do that? Simply activating Ignite persistence sounds like it's simply starting to move data from off-heap RAM to the harddrive when the RAM space is shrinking. However, if I understood it correctly, it will still query those elements for all my stream processing queries. So trying to find the previous element of stream (1) or trying to find all elements that are between those two elements in stream (2) would become slower and slower the longer the task runs. The incoming data is relevant for about 5 minutes, thus I tried using an expiration policy. This keeps the performance stable, but I'm not sure how to persist the expired data properly. Also, for calibration purposes, I'm generating a Map to store and apply calibration on elements - when I'm activating the expiry policy, I'm starting to run in Nullpointer Exceptions after about 5 minutes - is the policy also deleting the Map? Best regards Svonn -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Question about persisting stream processing results
Hi Svonn, I'm not sure that I properly understand your issue. Could you please provide a problematic code snipped? > is the policy also deleting the Map Yes, if it was stored as a value. Thanks! -Dmitry -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Question about persisting stream processing results
Hi! I'm receiving two streams of events, stream one (1) is basically only used as basis for interpolating and putting data in stream two (2). Whenever an element in stream (1) arrives, the local listener of my ContinuousQuery starting searching the previous element belonging to the same group. More specifically, its a ScanQuery that compares some IDs and searches for the one that has a timestamp bigger than the current one minus 1500ms while being smaller than the current timestamp. Currently, I want to persist stream (2) while keeping a stable performance. Whats the best way to do that? Simply activating Ignite persistence sounds like it's simply starting to move data from off-heap RAM to the harddrive when the RAM space is shrinking. However, if I understood it correctly, it will still query those elements for all my stream processing queries. So trying to find the previous element of stream (1) or trying to find all elements that are between those two elements in stream (2) would become slower and slower the longer the task runs. The incoming data is relevant for about 5 minutes, thus I tried using an expiration policy. This keeps the performance stable, but I'm not sure how to persist the expired data properly. Also, for calibration purposes, I'm generating a Map to store and apply calibration on elements - when I'm activating the expiry policy, I'm starting to run in Nullpointer Exceptions after about 5 minutes - is the policy also deleting the Map? Best regards Svonn -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
