Hello. >From the logs, I could suggest that some nodes are leaving or joining topology while the data is being loaded. That could be due to many reasons, be it GC or connectivity issues. Please share your Ignite config. As well you could monitor the cluster for GC via Web Console or check GC metrics in a way of your preference.
Kind regards, Alex. On Tue, Jun 6, 2017 at 9:27 PM, baozipu [via Apache Ignite Users] < [email protected]> wrote: > Hello, I am currently writing data Spark RDD to IgniteRDD using > IgniteRDD.savePairs() function. > I have no problem when writing small amount of data, just like: > > sharedRDD.savePairs(ss.sparkContext.parallelize(1 to 10000, 64).map(i => > (i, i))) > > However, error occurred when I write a 5Gb data into the IgniteRDD, by > calling savePairs() > > data type of SparkRDD is sparkRDD[Long, Row], type, which Long is just > some unique index value, and Row is the org.apache.spark.sql.Row type, > containing many elements. > > Can Ignite able to handle such kind of data type? > > When I call the savePairs() function, I got all WARNs: > > WARN GridCachePartitionExchangeManager: Failed to wait for partition map > exchange [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=1], > node=35ecd3af-484f-4709-978a-58c0d799a636]. Dumping pending objects that > might be the cause: > > 17/06/06 11:07:07 WARN GridCachePartitionExchangeManager: Ready affinity > version: AffinityTopologyVersion [topVer=10, minorTopVer=0] > > 17/06/06 11:07:07 WARN GridCachePartitionExchangeManager: Last exchange > future: GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, > reassign=false, discoEvt=DiscoveryCustomEvent > [customMsg=CacheAffinityChangeMessage > [id=24a978e7c51-ae96b9e6-eb8d-4546-b28e-684d29d5bc61, > topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], exchId=null, > partsMsg=null, exchangeNeeded=true], affTopVer=AffinityTopologyVersion > [topVer=10, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode > [id=74163cf4-432f-4ed5-8ba3-52c0449d7c52, addrs=[0:0:0:0:0:0:0:1%1, > 127.0.0.1, 172.30.33.165], sockAddrs=[sv1-hp0213-16.sv. > aaa.com/172.30.33.165:47500, /127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500], > discPort=47500, order=1, intOrder=1, lastExchangeTime=1496771432366, > loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], topVer=10, > nodeId8=35ecd3af, msg=null, type=DISCOVERY_CUSTOM_EVT, > tstamp=1496772417443]], crd=TcpDiscoveryNode > [id=74163cf4-432f-4ed5-8ba3-52c0449d7c52, > addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.165], sockAddrs=[ > sv1-hp0213-16.sv.aaa.com/172.30.33.165:47500, /127.0.0.1:47500, > /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=1, intOrder=1, > lastExchangeTime=1496771432366, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, > isClient=false], exchId=GridDhtPartitionExchangeId > [topVer=AffinityTopologyVersion > [topVer=10, minorTopVer=1], nodeId=74163cf4, evt=DISCOVERY_CUSTOM_EVT], > added=true, initFut=GridFutureAdapter [resFlag=2, res=true, > startTime=1496772417443, endTime=1496772417453, ignoreInterrupts=false, > state=DONE], init=true, topSnapshot=null, lastVer=null, > partReleaseFut=null, affChangeMsg=CacheAffinityChangeMessage > [id=24a978e7c51-ae96b9e6-eb8d-4546-b28e-684d29d5bc61, > topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], exchId=null, > partsMsg=null, exchangeNeeded=true], skipPreload=true, > clientOnlyExchange=true, initTs=1496772417443, centralizedAff=false, > forcedRebFut=null, evtLatch=0, > remaining=[74163cf4-432f-4ed5-8ba3-52c0449d7c52, > a8878928-67bd-4b90-82cf-8cd7c10c3abc, e6506f86-18df-4057-8eb4-3a9d5fbcdc41, > 490782d6-53b5-4d2b-aca5-8b750a996f87, 0b022f35-5749-41a0-9830-69ae8c17f5a9, > 2bf278bb-bafb-4730-8d40-5ab4bdcd164b, dd58f6b0-5f15-41a7-b2d9-2dadf1ace3a7, > 279e2d61-05f7-4cdb-ac7f-6f4e1c8518de, 0ee4105c-907a-4d43-9186-126e08bd783a], > srvNodes=[TcpDiscoveryNode [id=74163cf4-432f-4ed5-8ba3-52c0449d7c52, > addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.165], sockAddrs=[ > sv1-hp0213-16.sv.aaa.com/172.30.33.165:47500, /127.0.0.1:47500, > /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=1, intOrder=1, > lastExchangeTime=1496771432366, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, > isClient=false], TcpDiscoveryNode [id=0ee4105c-907a-4d43-9186-126e08bd783a, > addrs=[0:0:0:0:0:0:0:1%1, 10.93.128.99, 10.93.136.99, 127.0.0.1], > sockAddrs=[mapr417.sv.aaa.com/10.93.128.99:47500, /127.0.0.1:47500, > mapr417-eth1.sv.aaa.com/10.93.136.99:47500, /0:0:0:0:0:0:0:1%1:47500], > discPort=47500, order=2, intOrder=2, lastExchangeTime=1496771432376, > loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], > TcpDiscoveryNode [id=490782d6-53b5-4d2b-aca5-8b750a996f87, > addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.34.21], sockAddrs=[ > sv1-hp0201-04.sv.aaa.com/172.30.34.21:47500, /127.0.0.1:47500, > /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=4, intOrder=4, > lastExchangeTime=1496772399342, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, > isClient=false], TcpDiscoveryNode [id=dd58f6b0-5f15-41a7-b2d9-2dadf1ace3a7, > addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.169], sockAddrs=[ > sv1-hp0213-20.sv.aaa.com/172.30.33.169:47500, /127.0.0.1:47500, > /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=5, intOrder=5, > lastExchangeTime=1496772404498, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, > isClient=false], TcpDiscoveryNode [id=a8878928-67bd-4b90-82cf-8cd7c10c3abc, > addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.32.59], sockAddrs=[ > sv1-hp0203-10.sv.aaa.com/172.30.32.59:47500, /127.0.0.1:47500, > /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=6, intOrder=6, > lastExchangeTime=1496772404641, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, > isClient=false], TcpDiscoveryNode [id=2bf278bb-bafb-4730-8d40-5ab4bdcd164b, > addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.140], sockAddrs=[/ > 127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500, sv1-hp0212-11.sv.aaa.com/172. > 30.33.140:47500], discPort=47500, order=7, intOrder=7, > lastExchangeTime=1496772404824, > loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], > TcpDiscoveryNode [id=e6506f86-18df-4057-8eb4-3a9d5fbcdc41, > addrs=[0:0:0:0:0:0:0:1%1, 10.93.131.11, 10.93.136.244, 127.0.0.1], > sockAddrs=[mapr354-eth1.sv.aaa.com/10.93.136.244:47500, /127.0.0.1:47500, > /0:0:0:0:0:0:0:1%1:47500, mapr354.sv.aaa.com/10.93.131.11:47500], > discPort=47500, order=8, intOrder=8, lastExchangeTime=1496772406203, > loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], > TcpDiscoveryNode [id=0b022f35-5749-41a0-9830-69ae8c17f5a9, > addrs=[0:0:0:0:0:0:0:1%1, 10.93.131.135, 10.93.137.135, 127.0.0.1], > sockAddrs=[mapr260-eth1.sv.aaa.com/10.93.137.135:47501, /127.0.0.1:47501, > /0:0:0:0:0:0:0:1%1:47501, mapr260.sv.aaa.com/10.93.131.135:47501], > discPort=47501, order=9, intOrder=9, lastExchangeTime=1496772406254, > loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], > TcpDiscoveryNode [id=279e2d61-05f7-4cdb-ac7f-6f4e1c8518de, > addrs=[0:0:0:0:0:0:0:1%1, 10.93.131.188, 10.93.137.165, 127.0.0.1], > sockAddrs=[bfd-proda059-eth1.sv.aaa.com/10.93.137.165:47500, / > 127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500, mapr379.sv.aaa.com/10.93.131. > 188:47500], discPort=47500, order=10, intOrder=10, > lastExchangeTime=1496772407085, > loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false]], > super=GridFutureAdapter [resFlag=0, res=null, startTime=1496772417443, > endTime=0, ignoreInterrupts=false, state=INIT]] > > > 17/06/06 11:07:07 WARN GridCachePartitionExchangeManager: Pending > exchange futures: > 17/06/06 11:07:07 WARN GridCachePartitionExchangeManager: Last 10 > exchange futures (total: 11): > 17/06/06 11:07:07 WARN GridCachePartitionExchangeManager: >>> > GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, > reassign=false, discoEvt=DiscoveryCustomEvent > [customMsg=CacheAffinityChangeMessage > [id=24a978e7c51-ae96b9e6-eb8d-4546-b28e-684d29d5bc61, > topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], exchId=null, > partsMsg=null, exchangeNeeded=true], affTopVer=AffinityTopologyVersion > [topVer=10, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode > [id=74163cf4-432f-4ed5-8ba3-52c0449d7c52, addrs=[0:0:0:0:0:0:0:1%1, > 127.0.0.1, 172.30.33.165], sockAddrs=[sv1-hp0213-16.sv. > aaa.com/172.30.33.165:47500, /127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500], > discPort=47500, order=1, intOrder=1, lastExchangeTime=1496771432366, > loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], topVer=10, > nodeId8=35ecd3af, msg=null, type=DISCOVERY_CUSTOM_EVT, > tstamp=1496772417443]], crd=TcpDiscoveryNode > [id=74163cf4-432f-4ed5-8ba3-52c0449d7c52, > addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.165], sockAddrs=[ > sv1-hp0213-16.sv.aaa.com/172.30.33.165:47500, /127.0.0.1:47500, > /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=1, intOrder=1, > lastExchangeTime=1496771432366, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, > isClient=false], exchId=GridDhtPartitionExchangeId > [topVer=AffinityTopologyVersion > [topVer=10, minorTopVer=1], nodeId=74163cf4, evt=DISCOVERY_CUSTOM_EVT], > added=true, initFut=GridFutureAdapter [resFlag=2, res=true, > startTime=1496772417443, endTime=1496772417453, ignoreInterrupts=false, > state=DONE], init=true, topSnapshot=null, lastVer=null, > partReleaseFut=null, affChangeMsg=CacheAffinityChangeMessage > [id=24a978e7c51-ae96b9e6-eb8d-4546-b28e-684d29d5bc61, > topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], exchId=null, > partsMsg=null, exchangeNeeded=true], skipPreload=true, > clientOnlyExchange=true, initTs=1496772417443, centralizedAff=false, > forcedRebFut=null, evtLatch=0, > remaining=[74163cf4-432f-4ed5-8ba3-52c0449d7c52, > a8878928-67bd-4b90-82cf-8cd7c10c3abc, e6506f86-18df-4057-8eb4-3a9d5fbcdc41, > 490782d6-53b5-4d2b-aca5-8b750a996f87, 0b022f35-5749-41a0-9830-69ae8c17f5a9, > 2bf278bb-bafb-4730-8d40-5ab4bdcd164b, dd58f6b0-5f15-41a7-b2d9-2dadf1ace3a7, > 279e2d61-05f7-4cdb-ac7f-6f4e1c8518de, 0ee4105c-907a-4d43-9186-126e08bd783a], > srvNodes=[TcpDiscoveryNode [id=74163cf4-432f-4ed5-8ba3-52c0449d7c52, > addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.165], sockAddrs=[ > sv1-hp0213-16.sv.aaa.com/172.30.33.165:47500, /127.0.0.1:47500, > /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=1, intOrder=1, > lastExchangeTime=1496771432366, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, > isClient=false], TcpDiscoveryNode [id=0ee4105c-907a-4d43-9186-126e08bd783a, > addrs=[0:0:0:0:0:0:0:1%1, 10.93.128.99, 10.93.136.99, 127.0.0.1], > sockAddrs=[mapr417.sv.aaa.com/10.93.128.99:47500, /127.0.0.1:47500, > mapr417-eth1.sv.aaa.com/10.93.136.99:47500, /0:0:0:0:0:0:0:1%1:47500], > discPort=47500, order=2, intOrder=2, lastExchangeTime=1496771432376, > loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], > TcpDiscoveryNode [id=490782d6-53b5-4d2b-aca5-8b750a996f87, > addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.34.21], sockAddrs=[ > sv1-hp0201-04.sv.aaa.com/172.30.34.21:47500, /127.0.0.1:47500, > /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=4, intOrder=4, > lastExchangeTime=1496772399342, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, > isClient=false], TcpDiscoveryNode [id=dd58f6b0-5f15-41a7-b2d9-2dadf1ace3a7, > addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.169], sockAddrs=[ > sv1-hp0213-20.sv.aaa.com/172.30.33.169:47500, /127.0.0.1:47500, > /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=5, intOrder=5, > lastExchangeTime=1496772404498, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, > isClient=false], TcpDiscoveryNode [id=a8878928-67bd-4b90-82cf-8cd7c10c3abc, > addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.32.59], sockAddrs=[ > sv1-hp0203-10.sv.aaa.com/172.30.32.59:47500, /127.0.0.1:47500, > /0:0:0:0:0:0:0:1%1:47500], discPort=47500, order=6, intOrder=6, > lastExchangeTime=1496772404641, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, > isClient=false], TcpDiscoveryNode [id=2bf278bb-bafb-4730-8d40-5ab4bdcd164b, > addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.33.140], sockAddrs=[/ > 127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500, sv1-hp0212-11.sv.aaa.com/172. > 30.33.140:47500], discPort=47500, order=7, intOrder=7, > lastExchangeTime=1496772404824, > loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], > TcpDiscoveryNode [id=e6506f86-18df-4057-8eb4-3a9d5fbcdc41, > addrs=[0:0:0:0:0:0:0:1%1, 10.93.131.11, 10.93.136.244, 127.0.0.1], > sockAddrs=[mapr354-eth1.sv.aaa.com/10.93.136.244:47500, /127.0.0.1:47500, > /0:0:0:0:0:0:0:1%1:47500, mapr354.sv.aaa.com/10.93.131.11:47500], > discPort=47500, order=8, intOrder=8, lastExchangeTime=1496772406203, > loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], > TcpDiscoveryNode [id=0b022f35-5749-41a0-9830-69ae8c17f5a9, > addrs=[0:0:0:0:0:0:0:1%1, 10.93.131.135, 10.93.137.135, 127.0.0.1], > sockAddrs=[mapr260-eth1.sv.aaa.com/10.93.137.135:47501, /127.0.0.1:47501, > /0:0:0:0:0:0:0:1%1:47501, mapr260.sv.aaa.com/10.93.131.135:47501], > discPort=47501, order=9, intOrder=9, lastExchangeTime=1496772406254, > loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false], > TcpDiscoveryNode [id=279e2d61-05f7-4cdb-ac7f-6f4e1c8518de, > addrs=[0:0:0:0:0:0:0:1%1, 10.93.131.188, 10.93.137.165, 127.0.0.1], > sockAddrs=[bfd-proda059-eth1.sv.aaa.com/10.93.137.165:47500, / > 127.0.0.1:47500, /0:0:0:0:0:0:0:1%1:47500, mapr379.sv.aaa.com/10.93.131. > 188:47500], discPort=47500, order=10, intOrder=10, > lastExchangeTime=1496772407085, > loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false]], > super=GridFutureAdapter [resFlag=0, res=null, startTime=1496772417443, > endTime=0, ignoreInterrupts=false, state=INIT]] > > What is about? I use Ignite 1.9 > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://apache-ignite-users.70518.x6.nabble.com/Error- > occurs-when-calling-IgniteRDD-savePairs-tp13432.html > To start a new topic under Apache Ignite Users, email > [email protected] > To unsubscribe from Apache Ignite Users, click here > <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=YWxleGFuZGVyLmZlZG90b2ZmQGdtYWlsLmNvbXwxfC0xMzYxNTU0NTg=> > . > NAML > <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Error-occurs-when-calling-IgniteRDD-savePairs-tp13432p13530.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
