Hi team,
I try to save, index and query spark DataFrames with Ignite cache
this this my code:
import org.apache.spark.sql.Row
import org.apache.ignite.configuration._
import org.apache.ignite.spark.IgniteContext
import org.apache.ignite.configuration.CacheConfiguration
import sandbox._
// create rdd
val data = List[TRow1](TRow1("1", "test-1"), TRow1("2", "test-2"))
val df = sqlContext.createDataFrame[TRow1](data)
>> df: org.apache.spark.sql.DataFrame = [key: string, name: string]
// create ignite context (embeded mode)
val igniteContext = new IgniteContext[String, Row](sc, () => new
IgniteConfiguration(), false)
// cache config
val cacheCfg = new CacheConfiguration[String, Row]()
cacheCfg.setName("cache01")
cacheCfg.setIndexedTypes(classOf[String], classOf[Row])
// ignite cache
val cache = igniteContext.fromCache(cacheCfg)
>> cache: org.apache.ignite.spark.IgniteRDD[String,org.apache.spark.sql.Row]
>> = IgniteRDD[1] at RDD at IgniteAbstractRDD.scala:27
// df rdd save to cache
val df_rdd = df.rdd.map(r => (r.getAs[String](0), r))
>> df_rdd: org.apache.spark.rdd.RDD[(String, org.apache.spark.sql.Row)] =
>> MapPartitionsRDD[4] at map at <console>:38
cache.savePairs(df_rdd)
// query
val c = cache.count
>> res3: Long = 2
val result = cache.sql("select * from Row").collect
>> result: Array[org.apache.spark.sql.Row] = Array()
Unfortunately, running SQL query i don't get any result;
Could you plz recommend the correct way for storing, indexing and querying
sql.Rows with ignite cache
Best regards,
Dmitry
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/index-and-query-org-apache-ignite-spark-IgniteRDD-String-org-apache-spark-sql-Row-tp3343.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
