Im am prototyping Apache Ignite to store some simple data grid with simple objects:
Key: private final int securityId; private final long time; OHLC: private final int securityId; private final long time; private final double open; private final double high; private final double low; private final double close; private final double marketVWAP; After inserting 200000 to 300000 such key-value entries getting: [16:33:32] __________ ________________ [16:33:32] / _/ ___/ |/ / _/_ __/ __/ [16:33:32] _/ // (7 7 // / / / / _/ [16:33:32] /___/\___/_/|_/___/ /_/ /___/ [16:33:32] [16:33:32] ver. 2.1.0#20170721-sha1:a6ca5c8a [16:33:32] 2017 Copyright(C) Apache Software Foundation [16:33:32] [16:33:32] Ignite documentation: http://ignite.apache.org [16:33:32] [16:33:32] Quiet mode. [16:33:32] ^-- Logging to file '/Users/pranas/Apps/apache-ignite-fabric-2.1.0-bin/work/log/ignite-2ac0c6b2.log' [16:33:32] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat} [16:33:32] [16:33:32] OS: Mac OS X 10.12.6 x86_64 [16:33:32] VM information: Java(TM) SE Runtime Environment 1.8.0_40-b27 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.40-b25 [16:33:32] Configured plugins: [16:33:32] ^-- None [16:33:32] [16:33:32] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [16:33:32] Security status [authentication=off, tls/ssl=off] [16:33:32] REST protocols do not start on client node. To start the protocols on client node set '-DIGNITE_REST_START_ON_CLIENT=true' system property. [16:33:33] Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning [16:33:33] [16:33:33] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [16:33:33] [16:33:33] Ignite node started OK (id=2ac0c6b2) [16:33:33] Topology snapshot [ver=6, servers=1, clients=1, CPUs=8, heap=11.0GB] >>> Apache Ignite node is up and running. >>> Simulator - Real code would process journal events ... TODO. Processed 0 events so far Processed 100000 events so far Processed 200000 events so far [2017-08-28 16:33:46,592][ERROR][data-streamer-#36%null%][DataStreamerImpl] DataStreamer operation failed. class org.apache.ignite.IgniteCheckedException: Failed to finish operation (too many remaps): 32 at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$5.apply(DataStreamerImpl.java:869) at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$5.apply(DataStreamerImpl.java:834) at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382) at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346) at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334) at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:494) at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:473) at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.onResponse(DataStreamerImpl.java:1803) at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$3.onMessage(DataStreamerImpl.java:333) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184) at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126) at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1097) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: class org.apache.ignite.IgniteCheckedException: DataStreamer request failed [node=74d746ea-6b57-49be-85eb-3064262ab039] at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.onResponse(DataStreamerImpl.java:1792) ... 8 more Caused by: class org.apache.ignite.IgniteCheckedException: Failed to marshal response error, see node log for details. at org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.start(DataStreamProcessor.java:102) at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1788) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:938) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1896) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1648) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1076) at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:994) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:880) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:779) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:649) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:618) at org.apache.ignite.Ignition.start(Ignition.java:347) at com.algoteq.ignite.IgniteNode.main(IgniteNode.java:37) [16:33:47] (err) Failed to execute compound future reducer: GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=0, done=false, cancelled=false, err=null, futs=[true]]class org.apache.ignite.IgniteCheckedException: DataStreamer request failed [node=74d746ea-6b57-49be-85eb-3064262ab039] at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.onResponse(DataStreamerImpl.java:1792) at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$3.onMessage(DataStreamerImpl.java:333) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184) at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126) at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1097) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: class org.apache.ignite.IgniteCheckedException: Failed to marshal response error, see node log for details. at org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.start(DataStreamProcessor.java:102) at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1788) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:938) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1896) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1648) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1076) at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:994) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:880) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:779) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:649) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:618) at org.apache.ignite.Ignition.start(Ignition.java:347) at com.algoteq.ignite.IgniteNode.main(IgniteNode.java:37) Exception in thread "main" java.lang.IllegalStateException: Data streamer has been closed. at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.enterBusy(DataStreamerImpl.java:404) at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.addDataInternal(DataStreamerImpl.java:611) at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.addData(DataStreamerImpl.java:673) at com.algoteq.ignite.IgniteNode.main(IgniteNode.java:57) Suppressed: javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException: Data streamer has been cancelled: DataStreamerImpl [rcvr=org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual@361c294e, ioPlcRslvr=null, cacheName=OHLCV, bufSize=1024, parallelOps=16, timeout=-1, autoFlushFreq=0, bufMappings={74d746ea-6b57-49be-85eb-3064262ab039=Buffer [node=TcpDiscoveryNode [id=74d746ea-6b57-49be-85eb-3064262ab039, addrs=[0:0:0:0:0:0:0:1, 10.0.1.6, 127.0.0.1], sockAddrs=[/0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500, /10.0.1.6:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1503902013125, loc=false, ver=2.1.0#20170721-sha1:a6ca5c8a, isClient=false], isLocNode=false, idGen=4035, sem=java.util.concurrent.Semaphore@240d0282[Permits = 16], batchTopVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], entriesCnt=0, locFutsSize=0, reqsSize=0]}, cacheObjProc=GridProcessorAdapter [], cacheObjCtx=org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryContext@7aee5e6c, cancelled=true, failCntr=235443, activeFuts=GridConcurrentHashSet [elements=[GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=2117103751], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=94452364], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1767674765], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1717547480], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=337912484], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1267082248], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1013734212], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=426212061], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=274059873], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=225137455], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=270925300], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1421957909], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1654104932], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1851532609], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1853633821], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=621449265], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1609269939], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1379939065], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1173681213], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1161615764], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=170123144], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1110359161], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1546180757], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1716909005], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1969238242], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=366878600], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=777205720], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1835529707], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=65172208], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=212927822], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1428874286], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1106675375], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=98818836], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=164733543], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1904273153], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=2096842550], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=277780535], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=618173717], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1279013285], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1349323953], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1224918404], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1218640657], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=294203776], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=203926563], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1579030959], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=879372185], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=78506224], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1663202868], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=2131033138], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1609486039], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=725960838], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=150964565], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=357653585], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1778993228], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1192766628], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=671948173], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=748975217], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=364480205], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1690481990], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1100636920], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1590983023], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=738695357], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1329164354], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=16145357], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1812681178], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1465263219], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1668305954], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1001100746], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1357371578], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1890858939], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=954382324], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=2136419383], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1263102442], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=567612461], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1906635841], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1988962280], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1688242339], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1099667416]]], jobPda=org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$DataStreamerPda@4a23c7c8, depCls=null, fut=DataStreamerFuture [super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1853484976]], publicFut=IgniteFuture [orig=DataStreamerFuture [super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1853484976]]], disconnectErr=null, closed=true, lastFlushTime=1503902027656, skipStore=false, keepBinary=false, maxRemapCnt=32, remapSem=java.util.concurrent.Semaphore@2ab9cef8[Permits = 2147483647], remapOwning=false] at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1312) at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.close(DataStreamerImpl.java:1166) at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.close(DataStreamerImpl.java:1250) at com.algoteq.ignite.IgniteNode.main(IgniteNode.java:63) Caused by: class org.apache.ignite.IgniteCheckedException: Data streamer has been cancelled: DataStreamerImpl [rcvr=org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$Individual@361c294e, ioPlcRslvr=null, cacheName=OHLCV, bufSize=1024, parallelOps=16, timeout=-1, autoFlushFreq=0, bufMappings={74d746ea-6b57-49be-85eb-3064262ab039=Buffer [node=TcpDiscoveryNode [id=74d746ea-6b57-49be-85eb-3064262ab039, addrs=[0:0:0:0:0:0:0:1, 10.0.1.6, 127.0.0.1], sockAddrs=[/0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500, /10.0.1.6:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1503902013125, loc=false, ver=2.1.0#20170721-sha1:a6ca5c8a, isClient=false], isLocNode=false, idGen=4035, sem=java.util.concurrent.Semaphore@240d0282[Permits = 16], batchTopVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], entriesCnt=0, locFutsSize=0, reqsSize=0]}, cacheObjProc=GridProcessorAdapter [], cacheObjCtx=org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryContext@7aee5e6c, cancelled=true, failCntr=235443, activeFuts=GridConcurrentHashSet [elements=[GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=2117103751], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=94452364], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1767674765], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1717547480], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=337912484], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1267082248], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1013734212], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=426212061], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=274059873], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=225137455], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=270925300], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1421957909], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1654104932], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1851532609], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1853633821], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=621449265], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1609269939], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1379939065], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1173681213], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1161615764], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=170123144], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1110359161], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1546180757], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1716909005], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1969238242], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=366878600], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=777205720], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1835529707], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=65172208], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=212927822], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1428874286], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1106675375], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=98818836], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=164733543], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1904273153], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=2096842550], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=277780535], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=618173717], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1279013285], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1349323953], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1224918404], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1218640657], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=294203776], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=203926563], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1579030959], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=879372185], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=78506224], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1663202868], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=2131033138], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1609486039], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=725960838], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=150964565], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=357653585], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1778993228], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1192766628], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=671948173], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=748975217], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=364480205], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1690481990], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1100636920], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1590983023], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=738695357], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1329164354], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=16145357], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1812681178], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1465263219], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1668305954], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1001100746], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1357371578], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1890858939], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=954382324], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=2136419383], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1263102442], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=567612461], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1906635841], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1988962280], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1688242339], GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1099667416]]], jobPda=org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$DataStreamerPda@4a23c7c8, depCls=null, fut=DataStreamerFuture [super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1853484976]], publicFut=IgniteFuture [orig=DataStreamerFuture [super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1853484976]]], disconnectErr=null, closed=true, lastFlushTime=1503902027656, skipStore=false, keepBinary=false, maxRemapCnt=32, remapSem=java.util.concurrent.Semaphore@2ab9cef8[Permits = 2147483647], remapOwning=false] at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$5.apply(DataStreamerImpl.java:865) at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$5.apply(DataStreamerImpl.java:834) at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382) at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346) at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334) at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:494) at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:473) at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.onResponse(DataStreamerImpl.java:1803) at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$3.onMessage(DataStreamerImpl.java:333) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184) at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126) at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1097) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: class org.apache.ignite.IgniteCheckedException: DataStreamer request failed [node=74d746ea-6b57-49be-85eb-3064262ab039] at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.onResponse(DataStreamerImpl.java:1792) ... 8 more Caused by: class org.apache.ignite.IgniteCheckedException: Failed to marshal response error, see node log for details. at org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.start(DataStreamProcessor.java:102) at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1788) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:938) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1896) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1648) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1076) at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:994) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:880) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:779) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:649) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:618) at org.apache.ignite.Ignition.start(Ignition.java:347) at com.algoteq.ignite.IgniteNode.main(IgniteNode.java:37) My JVM params on 16g RAM: -XX:+UseG1GC -Xms2g -Xmx9g -XX:MaxDirectMemorySize=512m -XX:+DisableExplicitGC -XX:+AlwaysPreTouch config attached ignite-config.xml <http://apache-ignite-users.70518.x6.nabble.com/file/n16439/ignite-config.xml> Ignite 2.1.0: compile group: 'org.apache.ignite', name: 'ignite-spark', version: '2.1.0' I saw similar issues with very old versions mentioned, but this one is quite new .... Some new ideas? -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/DataStreamer-operation-failed-tp16439.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
