Re: Is there a way to use Ignite optimization and Spark optimization together when using Spark Dataframe API?

[aealexsandrov]

Hi,

I am not sure that it will work but you can try next:

        SparkSession spark = SparkSession
            .builder()
            .appName("SomeAppName")
            .master("spark://10.0.75.1:7077")
            .config(OPTION_DISABLE_SPARK_SQL_OPTIMIZATION, "false") //or
true
            .getOrCreate();

        JavaSparkContext sparkContext = new
JavaSparkContext(spark.sparkContext());

        JavaIgniteRDD<K, T> igniteRdd1 = igniteContext.<K,
T>fromCache("CACHE1");

        //here Ignite sql processor will be used because inside
SqlFieldsQuery
        Dataset<Row> ds1 = igniteRdd1.sql("select * from CACHE1");
        Dataset<Row> ds2 = igniteRdd1.sql("select * from CACHE2");

        //here spark sql processor will be used
        ds1.join(ds2).where(<SOME_COND>);

BR,
Andrei



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/