Have spark version 1.6.1 with apache-ignite-1.6.0-src. Have spark installed
on all 10 nodes, but, have ignite installed on only one node (master). Is that
an issue ?
Tried the below lines and its been running for the past 20 mins without any
errors...
scala> val rdd = df2.map(row => (row.getAs[String]("DRIVER_TYPE"), row))
rdd: org.apache.spark.rdd.RDD[(String, org.apache.spark.sql.Row)] =
MapPartitionsRDD[9] at map at <console>:42
scala> igniteRDD.savePairs(rdd)
Is it expected to take this long.
See below in ignite log console....
[23-11-2016 17:32:30][INFO ][grid-timeout-worker-#65%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=7a4a9bc9, name=null, uptime=02:12:00:684]
^-- H/N/C [hosts=1, nodes=2, CPUs=16]
^-- CPU [cur=0.07%, avg=0.04%, GC=0%]
^-- Heap [used=222MB, free=77.3%, comm=982MB]
^-- Non heap [used=33MB, free=89.04%, comm=33MB]
^-- Public thread pool [active=0, idle=32, qSize=0]
^-- System thread pool [active=0, idle=32, qSize=0]
^-- Outbound messages queue [size=0]
[23-11-2016 17:33:30][INFO ][grid-timeout-worker-#65%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=7a4a9bc9, name=null, uptime=02:13:00:684]
^-- H/N/C [hosts=1, nodes=2, CPUs=16]
^-- CPU [cur=0.03%, avg=0.04%, GC=0%]
^-- Heap [used=229MB, free=76.62%, comm=982MB]
^-- Non heap [used=33MB, free=89.03%, comm=33MB]
^-- Public thread pool [active=0, idle=32, qSize=0]
^-- System thread pool [active=0, idle=32, qSize=0]
^-- Outbound messages queue [size=0]
[23-11-2016 17:34:30][INFO ][grid-timeout-worker-#65%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=7a4a9bc9, name=null, uptime=02:14:00:685]
^-- H/N/C [hosts=1, nodes=2, CPUs=16]
^-- CPU [cur=0.03%, avg=0.04%, GC=0%]
^-- Heap [used=235MB, free=75.98%, comm=982MB]
^-- Non heap [used=33MB, free=89.03%, comm=33MB]
^-- Public thread pool [active=0, idle=32, qSize=0]
^-- System thread pool [active=0, idle=32, qSize=0]
^-- Outbound messages queue [size=0]
[23-11-2016 17:35:30][INFO ][grid-timeout-worker-#65%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=7a4a9bc9, name=null, uptime=02:15:00:687]
^-- H/N/C [hosts=1, nodes=2, CPUs=16]
^-- CPU [cur=0.07%, avg=0.04%, GC=0%]
^-- Heap [used=242MB, free=75.33%, comm=982MB]
^-- Non heap [used=33MB, free=89.03%, comm=33MB]
^-- Public thread pool [active=0, idle=32, qSize=0]
^-- System thread pool [active=0, idle=32, qSize=0]
^-- Outbound messages queue [size=0]
[23-11-2016 17:36:30][INFO ][grid-timeout-worker-#65%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=7a4a9bc9, name=null, uptime=02:16:00:691]
^-- H/N/C [hosts=1, nodes=2, CPUs=16]
^-- CPU [cur=0.07%, avg=0.04%, GC=0%]
^-- Heap [used=246MB, free=74.9%, comm=982MB]
^-- Non heap [used=33MB, free=89.03%, comm=33MB]
^-- Public thread pool [active=0, idle=32, qSize=0]
^-- System thread pool [active=0, idle=32, qSize=0]
^-- Outbound messages queue [size=0]
Thanks,
Vidhya
________________________________________
From: vkulichenko <[email protected]>
Sent: Wednesday, November 23, 2016 3:07 PM
To: [email protected]
Subject: Re: SparkRDD with Ignite
What version of Scala do you have? Also why are you on Ignite 1.6? Can you
switch to 1.7?
-Val
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/SparkRDD-with-Ignite-tp9160p9165.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
