Hi Vadim, You wrote to Apache Ignite Community dev list. Ignite does not have versions you mentioned.
On Thu, Feb 16, 2017 at 7:58 PM, Vadim Dedkov <[email protected]> wrote: > Hello to all. > I use GG version 8.0.1.ea7 & ignite version 1.10.1.ea7. > On my cluster I have 4 server nodes. > Each node have 8 cpu and 16gb memory. > GG visor info on each node: Heap memory initialized: 10gb, Heap memory > maximum: 10gb. > I tried to process file (~4.7gb) with Hadoop MR: parse and write file data > to GG cache. > > And after a while since start job I got errors (end of letter). > I tried to analise error log & source code and one of my versions is: GG > tries to swap memory on hard disk --> I don't have enough free space in GG > directory --> error. > But I didn't setup cache config "swapEnabled" and I cannot find such > config option in Ignite 1.10.ea7 source code. > And in this hypothesis I cannot realise, why GG tries to use swap: 10gb * > 4node is not enough for 4.7 gb data? > > > On one of my GG node: > [16:35:08,442][SEVERE][sys-#10%null%][GridDhtAtomicCache] > <calcScorePointsCacheIn_2001> Unexpected exception during cache update > class org.apache.ignite.internal.mem.OutOfMemoryException: null > at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl. > allocatePage(PageMemoryNoStoreImpl.java:242) > at org.apache.ignite.internal.processors.cache.database. > freelist.FreeListImpl.allocateDataPage(FreeListImpl.java:295) > at org.apache.ignite.internal.processors.cache.database. > freelist.FreeListImpl.insertDataRow(FreeListImpl.java:325) > at org.apache.ignite.internal.processors.cache.database. > RowStore.addRow(RowStore.java:80) > at org.apache.ignite.internal.processors.cache. > IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update( > IgniteCacheOffheapManagerImpl.java:842) > at org.apache.ignite.internal.processors.cache. > IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl. > java:302) > at org.apache.ignite.internal.processors.cache. > GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3576) > at org.apache.ignite.internal.processors.cache. > GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2093) > at org.apache.ignite.internal.processors.cache.distributed. > dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2353) > at org.apache.ignite.internal.processors.cache.distributed. > dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0( > GridDhtAtomicCache.java:1755) > at org.apache.ignite.internal.processors.cache.distributed. > dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal( > GridDhtAtomicCache.java:1593) > at org.apache.ignite.internal.processors.cache.distributed. > dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest > (GridDhtAtomicCache.java:3059) > at org.apache.ignite.internal.processors.cache.distributed. > dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:131) > at org.apache.ignite.internal.processors.cache.distributed. > dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:288) > at org.apache.ignite.internal.processors.cache.distributed. > dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:283) > at org.apache.ignite.internal.processors.cache.GridCacheIoManager. > processMessage(GridCacheIoManager.java:758) > at org.apache.ignite.internal.processors.cache. > GridCacheIoManager.onMessage0(GridCacheIoManager.java:363) > at org.apache.ignite.internal.processors.cache.GridCacheIoManager. > handleMessage(GridCacheIoManager.java:287) > at org.apache.ignite.internal.processors.cache. > GridCacheIoManager.access$000(GridCacheIoManager.java:89) > at org.apache.ignite.internal.processors.cache. > GridCacheIoManager$1.onMessage(GridCacheIoManager.java:232) > at org.apache.ignite.internal.managers.communication. > GridIoManager.invokeListener(GridIoManager.java:1369) > at org.apache.ignite.internal.managers.communication. > GridIoManager.processRegularMessage0(GridIoManager.java:997) > at org.apache.ignite.internal.managers.communication. > GridIoManager.access$2100(GridIoManager.java:113) > at org.apache.ignite.internal.managers.communication. > GridIoManager$6.run(GridIoManager.java:950) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > And in MR job log: > Error: org.apache.ignite.cache.CachePartialUpdateException: Failed to > update keys (retry update if possible).: [00-1-10060940] > at org.apache.ignite.internal.processors.cache.GridCacheUtils. > convertToCacheException(GridCacheUtils.java:1478) > at org.apache.ignite.internal.processors.cache.IgniteCacheProxy. > cacheException(IgniteCacheProxy.java:2118) > at org.apache.ignite.internal.processors.cache. > IgniteCacheProxy.put(IgniteCacheProxy.java:1315) > at ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load. > GridGainLineCacheFiller.fillCache(GridGainLineCacheFiller.scala:10) > at ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load. > ValueGroupingReducer.reduce(ValueGroupingReducer.scala:36) > at ru.sberbank.bigdata.mlfw.hadoop.hadoop_to_gridgain_table_load. > ValueGroupingReducer.reduce(ValueGroupingReducer.scala:16) > at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) > at org.apache.hadoop.mapred.ReduceTask.runNewReducer( > ReduceTask.java:627) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1709) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: class org.apache.ignite.internal.processors.cache. > CachePartialUpdateCheckedException: Failed to update keys (retry update > if possible).: [00-1-10060940] > at org.apache.ignite.internal.processors.cache.distributed. > dht.atomic.GridNearAtomicSingleUpdateFuture.onResult( > GridNearAtomicSingleUpdateFuture.java:227) > at org.apache.ignite.internal.processors.cache.distributed. > dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRespons > e(GridDhtAtomicCache.java:3077) > at org.apache.ignite.internal.processors.cache.distributed. > dht.atomic.GridDhtAtomicCache.access$700(GridDhtAtomicCache.java:131) > at org.apache.ignite.internal.processors.cache.distributed. > dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:306) > at org.apache.ignite.internal.processors.cache.distributed. > dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:301) > at org.apache.ignite.internal.processors.cache. > GridCacheIoManager.processMessage(GridCacheIoManager.java:758) > at org.apache.ignite.internal.processors.cache. > GridCacheIoManager.onMessage0(GridCacheIoManager.java:363) > at org.apache.ignite.internal.processors.cache. > GridCacheIoManager.handleMessage(GridCacheIoManager.java:287) > at org.apache.ignite.internal.processors.cache. > GridCacheIoManager.access$000(GridCacheIoManager.java:89) > at org.apache.ignite.internal.processors.cache. > GridCacheIoManager$1.onMessage(GridCacheIoManager.java:232) > at org.apache.ignite.internal.managers.communication. > GridIoManager.invokeListener(GridIoManager.java:1369) > at org.apache.ignite.internal.managers.communication. > GridIoManager.processRegularMessage0(GridIoManager.java:997) > at org.apache.ignite.internal.managers.communication. > GridIoManager.access$2100(GridIoManager.java:113) > at org.apache.ignite.internal.managers.communication. > GridIoManager$6.run(GridIoManager.java:950) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Suppressed: class org.apache.ignite.IgniteCheckedException: > Failed to update keys on primary node. > at org.apache.ignite.internal. > processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse. > addFailedKeys(GridNearAtomicUpdateResponse.java:370) > at org.apache.ignite.internal. > processors.cache.distributed.dht.atomic.GridDhtAtomicCache. > updateAllAsyncInternal0(GridDhtAtomicCache.java:1828) > at org.apache.ignite.internal. > processors.cache.distributed.dht.atomic.GridDhtAtomicCache. > updateAllAsyncInternal(GridDhtAtomicCache.java:1593) > at org.apache.ignite.internal. > processors.cache.distributed.dht.atomic.GridDhtAtomicCache. > processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3059) > at org.apache.ignite.internal. > processors.cache.distributed.dht.atomic.GridDhtAtomicCache. > access$600(GridDhtAtomicCache.java:131) > at org.apache.ignite.internal. > processors.cache.distributed.dht.atomic.GridDhtAtomicCache$ > 5.apply(GridDhtAtomicCache.java:288) > at org.apache.ignite.internal. > processors.cache.distributed.dht.atomic.GridDhtAtomicCache$ > 5.apply(GridDhtAtomicCache.java:283) > ... 12 more > Suppressed: class > org.apache.ignite.internal.mem.OutOfMemoryException: > null > at org.apache.ignite.internal.pagemem.impl. > PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:242) > at org.apache.ignite.internal. > processors.cache.database.freelist.FreeListImpl. > allocateDataPage(FreeListImpl.java:295) > at org.apache.ignite.internal. > processors.cache.database.freelist.FreeListImpl. > insertDataRow(FreeListImpl.java:325) > at org.apache.ignite.internal. > processors.cache.database.RowStore.addRow(RowStore.java:80) > at org.apache.ignite.internal.processors.cache. > IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update( > IgniteCacheOffheapManagerImpl.java:842) > at org.apache.ignite.internal.processors.cache. > IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl. > java:302) > at org.apache.ignite.internal.processors.cache. > GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3576) > at org.apache.ignite.internal.processors.cache. > GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2093) > at org.apache.ignite.internal. > processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle( > GridDhtAtomicCache.java:2353) > at org.apache.ignite.internal. > processors.cache.distributed.dht.atomic.GridDhtAtomicCache. > updateAllAsyncInternal0(GridDhtAtomicCache.java:1755) > ... 17 more > > > > -- > _______________ _______________ > Best regards, С уважением > Vadim Dedkov. Вадим Дедков. >
