Cache expiry policy not deleting records from disk(native persistence)
Hi all, I have set cache expiry policy like this And batch inserting records to one of the table which is created with above cache template. Around 10 minutes, I ingested ~1.5GB of data and after 10 minutes records started reducing(expiring) when I monitored from sqlline. 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS; COUNT(ID) 248896 1 row selected (0.86 seconds) 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS; COUNT(ID) 222174 1 row selected (0.313 seconds) 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS; COUNT(ID) 118154 1 row selected (0.15 seconds) 0: jdbc:ignite:thin://192.168.*.*:10800> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS; COUNT(ID) 76061 1 row selected (0.106 seconds) 0: jdbc:ignite:thin://192.168.*.*:10800> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS; COUNT(ID) 41671 1 row selected (0.063 seconds) 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS; COUNT(ID) 18455 1 row selected (0.037 seconds) 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from DIMENSIONS; COUNT(ID) 0 1 row selected (0.014 seconds) But in the meantime, the disk space used by the persistence store was in the same usage level instead of decreasing. [ignite@ignite-cluster-ign-shiv-0 ignite]$ while true ; do df -h /opt/ignite/persistence/; sleep 1s; done Filesystem Size Used Avail Use% Mounted on /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence Filesystem Size Used Avail Use% Mounted on /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence Filesystem Size Used Avail Use% Mounted on /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence Filesystem Size Used Avail Use% Mounted on /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence Filesystem Size Used Avail Use% Mounted on /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence Filesystem Size Used Avail Use% Mounted on /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence Filesystem Size Used Avail Use% Mounted on /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence This means that expiry policy not deleting records from the disk, but ignite document says when expiry policy is set and native persistence is enabled then it deletes records from disk as well. Am I missing some configuration? Any help is appreciated. Shiva
Re: Using Ignite Native Persistence as a "temporary durable" cache
origonal question: ferhadcebi...@gmail.com 5:00 AM (5 hours ago) to me Thanks for answer. But, how Ignite will store that datas to store? Will append to the end of WAL? If, then it is sure faster than storing cache operations in some 3rd part database. Ignite does store data to WAL first, but per my benchmarks, storing 100,000 records took a few seconds, and deleting took a few seconds more. You can experiment using : https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/persistentstore/PersistentStoreExample.java Here is sample code: (modified section in above example) [image: image.png] On Thu, Sep 5, 2019 at 12:17 PM Alexander Korenshteyn < alexanderko...@gmail.com> wrote: > Hello, >Ignite native persistence has a good track record, is fast and > reliable, you can use it in your application. > >Take a look at the following example of how to use a streamer to > quickly insert data: > https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/persistentstore/PersistentStoreExample.java > >Use cache.removeAll() -- to quickly remove all entries once you are > done. > Thanks, Alex > > On Thu, Sep 5, 2019 at 10:23 AM Farhad Jabiyev > wrote: > >> Hi all, >> >> We have MS SQL database server which contains all data. Our application >> will fetch some datas from database server and put them to the cache. And >> then, during 5-10 seconds we will do some updates to that objects and push >> that changes to ignite in-memory cache. And then, after 5-10 seconds we >> will take that changes and sync them will database as a bulk operation and >> then will clear that permanent storage. So, we need some permanent storage >> to store those updates for 5-10 seconds. >> And actually, we will work at most with 100.000 entity. >> >> The idea behind that flow is that we can't now scale DB and users already >> putting load to the database. >> >> We can't decide whether we have to use Native Persistent or some another >> 3rd party database like Maria or PostgreSQL for storing that cache >> operations for 5-10 seconds. >> >> Will Ignite works fast if we will use native persistent and clear the >> cache periodically? >> >
Re: ML stable and performance
Python is 25 times slower than Java for ML at runtimes, which I found out online. But I don't know that statement is true or not. I need insiders' opinion. Which ml other packages are best options for Ignite? On Fri, Sep 6, 2019 at 7:28 PM Mikael wrote: > Hi! > > I have never used it myself but it's been there for long time and I > would expect it to be stable, and yes it will run distributed, I can't > say anything about performance as I have never used it. > > You will find a lot of more information at: > > https://apacheignite.readme.io/docs/machine-learning > > Mikael > > > Den 2019-09-06 kl. 11:50, skrev David Williams: > > > > > > I am evaluating ML framework for Java platform. I knew Ignite has ML > > package. > > But I like to know its stability and performance for production. Can > > Ignite > > ML code run in distribute way? > > > > Except its own ML package, which ml packages are best options for Ignite? >
Re: ML stable and performance
Hi! I have never used it myself but it's been there for long time and I would expect it to be stable, and yes it will run distributed, I can't say anything about performance as I have never used it. You will find a lot of more information at: https://apacheignite.readme.io/docs/machine-learning Mikael Den 2019-09-06 kl. 11:50, skrev David Williams: I am evaluating ML framework for Java platform. I knew Ignite has ML package. But I like to know its stability and performance for production. Can Ignite ML code run in distribute way? Except its own ML package, which ml packages are best options for Ignite?
ML stable and performance
I am evaluating ML framework for Java platform. I knew Ignite has ML package. But I like to know its stability and performance for production. Can Ignite ML code run in distribute way? Except its own ML package, which ml packages are best options for Ignite?
Ignite ignores cache config when putting entries through near cache
Hi, I have a problem with putting entries to partitioned or replicated cache through near cache on client node. Even when I configured cache on server to put values to off-heap space they are stored on heap. I already descibed it on jira https://issues.apache.org/jira/projects/IGNITE/issues/IGNITE-12142. I'm writing also here - mailing list seems to be more active. Is it a bug in Ignite or a problem with configuration? -- Bartłomiej Stefański
