Have spark version 1.6.1 with apache-ignite-1.6.0-src. Have spark installed on all 10 nodes, but, have ignite installed on only one node (master). Is that an issue ?
Tried the below lines and its been running for the past 20 mins without any errors... scala> val rdd = df2.map(row => (row.getAs[String]("DRIVER_TYPE"), row)) rdd: org.apache.spark.rdd.RDD[(String, org.apache.spark.sql.Row)] = MapPartitionsRDD[9] at map at <console>:42 scala> igniteRDD.savePairs(rdd) Is it expected to take this long. See below in ignite log console.... [23-11-2016 17:32:30][INFO ][grid-timeout-worker-#65%null%][IgniteKernal] Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=7a4a9bc9, name=null, uptime=02:12:00:684] ^-- H/N/C [hosts=1, nodes=2, CPUs=16] ^-- CPU [cur=0.07%, avg=0.04%, GC=0%] ^-- Heap [used=222MB, free=77.3%, comm=982MB] ^-- Non heap [used=33MB, free=89.04%, comm=33MB] ^-- Public thread pool [active=0, idle=32, qSize=0] ^-- System thread pool [active=0, idle=32, qSize=0] ^-- Outbound messages queue [size=0] [23-11-2016 17:33:30][INFO ][grid-timeout-worker-#65%null%][IgniteKernal] Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=7a4a9bc9, name=null, uptime=02:13:00:684] ^-- H/N/C [hosts=1, nodes=2, CPUs=16] ^-- CPU [cur=0.03%, avg=0.04%, GC=0%] ^-- Heap [used=229MB, free=76.62%, comm=982MB] ^-- Non heap [used=33MB, free=89.03%, comm=33MB] ^-- Public thread pool [active=0, idle=32, qSize=0] ^-- System thread pool [active=0, idle=32, qSize=0] ^-- Outbound messages queue [size=0] [23-11-2016 17:34:30][INFO ][grid-timeout-worker-#65%null%][IgniteKernal] Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=7a4a9bc9, name=null, uptime=02:14:00:685] ^-- H/N/C [hosts=1, nodes=2, CPUs=16] ^-- CPU [cur=0.03%, avg=0.04%, GC=0%] ^-- Heap [used=235MB, free=75.98%, comm=982MB] ^-- Non heap [used=33MB, free=89.03%, comm=33MB] ^-- Public thread pool [active=0, idle=32, qSize=0] ^-- System thread pool [active=0, idle=32, qSize=0] ^-- Outbound messages queue [size=0] [23-11-2016 17:35:30][INFO ][grid-timeout-worker-#65%null%][IgniteKernal] Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=7a4a9bc9, name=null, uptime=02:15:00:687] ^-- H/N/C [hosts=1, nodes=2, CPUs=16] ^-- CPU [cur=0.07%, avg=0.04%, GC=0%] ^-- Heap [used=242MB, free=75.33%, comm=982MB] ^-- Non heap [used=33MB, free=89.03%, comm=33MB] ^-- Public thread pool [active=0, idle=32, qSize=0] ^-- System thread pool [active=0, idle=32, qSize=0] ^-- Outbound messages queue [size=0] [23-11-2016 17:36:30][INFO ][grid-timeout-worker-#65%null%][IgniteKernal] Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=7a4a9bc9, name=null, uptime=02:16:00:691] ^-- H/N/C [hosts=1, nodes=2, CPUs=16] ^-- CPU [cur=0.07%, avg=0.04%, GC=0%] ^-- Heap [used=246MB, free=74.9%, comm=982MB] ^-- Non heap [used=33MB, free=89.03%, comm=33MB] ^-- Public thread pool [active=0, idle=32, qSize=0] ^-- System thread pool [active=0, idle=32, qSize=0] ^-- Outbound messages queue [size=0] Thanks, Vidhya ________________________________________ From: vkulichenko <valentin.kuliche...@gmail.com> Sent: Wednesday, November 23, 2016 3:07 PM To: user@ignite.apache.org Subject: Re: SparkRDD with Ignite What version of Scala do you have? Also why are you on Ignite 1.6? Can you switch to 1.7? -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/SparkRDD-with-Ignite-tp9160p9165.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.