Re: Running Spark SQL on Spark Thrift Server with Ignite
Ravi, You can take a look at these pages: https://apacheignite-fs.readme.io/docs/ignitecontext-igniterdd https://apacheignite-fs.readme.io/docs/testing-integration-with-spark-shell -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Running-Spark-SQL-on-Spark-Thrift-Server-with-Ignite-tp16087p16184.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Running Spark SQL on Spark Thrift Server with Ignite
Hi Val, Thanks for the reply. Can you share some example or sample code snippet for the steps you have explained?. The link you have shared doesn't explain mapping Hive/Spar RDD to Ignite RDD context? Regards Ravi.P -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Running-Spark-SQL-on-Spark-Thrift-Server-with-Ignite-tp16087p16153.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Running Spark SQL on Spark Thrift Server with Ignite
Ravi, If you need to speed up SQL, you should make sure Ignite uses indexes to execute queries. I think you can do the following: - Create Hive RDD and map it to RDD of key value pairs. - Create new IgniteRDD on top of a cache and use IgniteRDD#savePairs method to load data from Hive to Ignite. - IgniteRDD#sql method to execute queries. Note that SQL needs to be configured in Ignite (i.e. you need to specify queryable fields, indexes, etc.). More information here: https://apacheignite.readme.io/docs/sql-queries -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Running-Spark-SQL-on-Spark-Thrift-Server-with-Ignite-tp16087p16115.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Running Spark SQL on Spark Thrift Server with Ignite
Hi Ravi, I don't think it currently will, because this will require integration with data frames. We have it plans, but it is not implemented yet. I think you should use IgniteRDD or Ignite APIs directly. Can you describe business use case you're trying to implement? -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Running-Spark-SQL-on-Spark-Thrift-Server-with-Ignite-tp16087p16091.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
