Hi Pascal, Looks like you do not actually run a query. Ignite doesn't fetch full resultset by default and use paging for fetching.
Try to make *cursor.getAll() *called on every iteration or (which is better) get iterator and iterate over rows to reduce heap memory pressure. On Wed, Aug 2, 2017 at 2:07 PM, Pascal <[email protected]> wrote: > Hi, > > I try to use Ignite 2.1 to simulate the situation where a table doesn't fit > in memory and data is read from disk every time a full table scan occurs. > I've adapted the PersistentStoreExample to create a cache of a million > items > which is stored in 200MB of data files and 120MB of index files. I call > PersistentStoreConfiguration.setFileIOFactory() to register a custom > FileIOFactory that tracks what file I/O is occurring. This tracker shows > the > files are read during startup of the example, but not while the SQL query > is > running. The cache shouldn't fit in 10MB and I would expect that during the > query the data would be read from disk. Why is this not happening and what > do I need to change to test this? > > Thanks for your help. > Pascal > > <bean class="org.apache.ignite.configuration.MemoryConfiguration"> > <property name="pageSize" value="2048"/> > <property name="systemCacheInitialSize" value="#{10 * 1024 * > 1024}"/> > <property name="systemCacheMaxSize" value="#{10 * 1024 * 1024}"/> > <property name="defaultMemoryPolicyName" value="default_mem_plc"/> > <property name="memoryPolicies"> > <list> > <bean class="org.apache.ignite.configuration. > MemoryPolicyConfiguration"> > <property name="name" > value="default_mem_plc"/> > <property name="initialSize" value="#{10 * > 1024 * 1024}"/> > <property name="maxSize" value="#{10 * > 1024 * 1024}"/> > </bean> > </list> > </property> > </bean> > > public class PersistentStoreExample { > private static final boolean UPDATE = false; > private static final long COUNT = 1000000; > public static void doMain() throws Exception { > Ignition.setClientMode(false); > Ignite ig = > Ignition.start("D:/Ignite/2.1/examples/config/persistentstore/example- > persistent-store.xml"); > IgniteConfiguration ic = ig.configuration(); > PersistentStoreConfiguration pc = > ic.getPersistentStoreConfiguration(); > pc.setPersistentStorePath("C:\\tmp\\"); > MyFileIOFactory myFact = new MyFileIOFactory(); > pc.setFileIOFactory(myFact); > ig.active(true); > IgniteCache<Long, MyPerson> cache = ig.cache("personCache"); > > if (UPDATE) { > System.out.println("Populating the cache..."); > try (IgniteDataStreamer<Long, MyPerson> streamer = > ig.dataStreamer("personCache")) { > streamer.allowOverwrite(true); > for (long i = 0; i < COUNT; i++) { > MyPerson o = new MyPerson(i, "per-" + i); > streamer.addData(i, o); > if (i > 0 && i % 10_000 == 0) System.out.println("Done: > " + i); > } > } > } > > while(true) > { > Thread.sleep(3000); > System.out.println("\nFull scan " + COUNT + " times..."); > for(int getId = 0; getId< COUNT; getId++) { > QueryCursor cur = cache.query(new SqlFieldsQuery("select > id, > name from MyPerson where name like ?").setArgs("per-" + getId)); > if(getId % 100000 == 0) { > System.out.println(" SQL Result: " + cur.getAll()); > } > } > } > } > } > > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/MemoryPolicy-size-vs-cache-size-tp15892.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. > -- Best regards, Andrey V. Mashenkov
