mfronczyk wrote > I need a data structure that holds 1 billion of objects. Each object will > consist of 4 integers. This will be used to model a large 2-dimensional > grid.
To keep a data structure this big in memory, you should consider using Ignite Off-Heap Memory support [1]. Also make sure that you have enough memory on your server to fit the whole data set. [1] http://apacheignite.gridgain.org/docs/off-heap-memory mfronczyk wrote > Another important feature is that the memory layout of the structure > matches the array indexes - element with index n is before the element n + > 1. There will be a need to read 16384 consecutive elements quickly and > that memory layout will be very cache friendly for such operation. If you remove this consecutive memory layout requirement, then you can actually store 1 billion of objects separately, using array index as a key. In this case, you can use full IgniteCache API, access each element separately, and update different elements concurrently. Depending on your requirements, this approach may be easier to use. However, this approach will also consume more memory than the approach suggested by Valentin. mfronczyk wrote > Elements written to the structure need to be durable. The plan is to > replicate the writes on 2 nodes. Writes to about 1% of the indexes are > more important than others and it should be guaranteed that acknowledged > writes for that indexes survive a complete crash of one node. There should > be no time window during which such writes are lost. But it's OK for the > rest of 99% other writes to have a chance of data loss for performance > reasons. So we need async replication with a possibility to enable > synchronous replication for certain puts. Synchronous vs. asynchronous backups can be configured on per-cache level via FULL_SYNC and PRIMARY_SYNC modes [2]. So if you really need to separate synchronous updates from asynchronous updates, you will have to split your data structure into 2 parts and store it in 2 different caches. My preference would be that you start off with one cache in FULL_SYNC mode, and introduce another one in PRIMARY_SYNC mode only if you run into performance issue. [2] http://apacheignite.gridgain.org/v1.3/docs/primary-and-backup-copies#synchronous-and-asynchronous-backups -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Can-Ignite-be-configured-to-provide-an-array-like-replicated-and-durable-map-tp1304p1329.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
