Hello!
"Use as little memory as possible" is something Ignite is not optimized
for. As you limit the amount of memory in the data region and increase
amount of data, even all sorts of metadata pages (cache info, free lists,
indexes, etc) will stop fitting in memory and then performance will grind
to a halt or you can even catch IgniteOOM.
I guess that you will have to live with the limit that you have determined.
Regards,
--
Ilya Kasnacheev
вс, 14 мар. 2021 г. в 23:15, Sebastien Blind <sebastien.bl...@gmail.com>:
> Hello Apache Ignite friends,
> I am trying to load 1bn records in an ignite cache with persistence
> enabled for a poc. Records only consist of an id, a uuid and a state, and
> I'd like to lookup by id (this is the key in the cache) and by token (sql
> queries for that are fine or another cache keyed by uuid is another option).
>
> The experiment is running locally on a MacOs (16Gb memory) laptop and I'd
> like to use as little memory as possible, and use disk storage as much as
> possible. Everything works ok until ~150M with query index enabled, and
> ~600M without index or side cache, so I'd be curious about what type of
> settings (if any) could help w/ the setup, and what amount of data can a
> single node be expected to handle. Restarting my ignite node and the
> loading process seems to help and more data can be stuffed in the cache.
>
> I've tried to play w/ settings like size, number of partitions etc and to
> find some info on how to control what's onheap/offheap etc, any
> overhead/sizing that could be taken into consideration without much
> success. Maybe the experiment is doomed to fail just based on the specs,
> but it would be good to understand the constraints.. so any help would be
> much appreciated!
>
> Thanks in advance!
> Sebastien
>
> PS:
> JVM is running w/ -Xms4g -Xmx4g -server -XX:MaxMetaspaceSize=256m settings.
>
> Some snippet of the config:
>
> <property name=*"persistenceEnabled"* value=*"true"* />
>
> <property name=*"initialSize"* value=*"#{4L * 1024 * 1024 * 1024}"* />
>
> <property name=*"maxSize" *value=*"#{4L * 1024 * 1024 * 1024}"* />
>
> <property name=*"pageEvictionMode"* value=*"RANDOM_2_LRU"* />
>
> Cache configuration
>
> <property name=*"cacheConfiguration"*>
>
> <bean class=*"org.apache.ignite.configuration.CacheConfiguration"*>
>
> <!-- Set the cache name. -->
>
> <property name=*"name"* value=*"qa_sqr_txn"* />
>
> <!-- Set the cache mode. -->
>
> <property name=*"cacheMode"* value=*"PARTITIONED"* />
>
> <property name=*"backups"* value=*"0"* />
>
> <property name=*"storeKeepBinary"* value=*"true"* />
>
>
> <property name=*"affinity"*>
>
> <bean class=
> *"org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction"*
> >
>
> <property name=*"partitions"* value=*"8192"* />
>
> </bean>
>
> </property>
>
>
> <!-- Configure query entities -->
>
> <property name=*"queryEntities"*>
>
> <list>
>
> <bean class=*"org.apache.ignite.cache.QueryEntity"*>
>
> <!-- Setting the type of the key -->
>
> <property name=*"keyType"* value=*"java.lang.String"* />
>
> <property name=*"keyFieldName"* value=*"id"* />
>
>
> <!-- Setting type of the value -->
>
> <property name=*"valueType"* value=*"com.xxx.Txn"* />
>
>
> <property name=*"fields"*>
>
> <map>
>
> <entry key=*"id"* value=*"java.lang.String"* />
>
> <entry key=*"token"* value=*"java.lang.String"* />
>
> <entry key=*"state"* value=*"java.lang.String "* />
>
> </map>
>
> </property>
>
> <!--
>
> <property name="indexes">
>
> <list>
>
> <bean class="org.apache.ignite.cache.QueryIndex">
>
> <constructor-arg value="token" />
>
> </bean>
>
> </list>
>
> </property>
>
> -->
>
> </bean>
>
> </list>
>
> </property>
>
> </bean>
>
> </property>
>
