Currently, OPTION_DISABLE_SPARK_SQL_OPTIMIZATION option can only be set on spark session level. It means I can only have Ignite optimization or Spark optimization for one Spark job.
Let's say I want to load data into spark memory with pushdown filters using Ignite optimization. For example, I want to load one day's data using this sql "select * from tableA where date = '2018-09-01'". With Ignite optimization, this sql is executed on Ignite and the where clause filter is applied on Ignite. But with Spark optimization, all the data in this table will be loaded into Spark memory and do filter later. Then I want to join filtered tableA with filtered tableB which is also loaded from Ignite. But I want use Spark's join feature to do the join because both filtered tableA with filtered tableB contains millions or rows and Ignite is not optimized for join. How can I do that? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
