a. Data is stored in Ignite and Spark will fetch data for a particular partition when you execute something. It will be done lazily page by page while iteration is happening. Specifically, IgniteRDD overrides compute() method where it gets a scan query iterator over a partition. b. When you execute the query using IgniteRDD.sql() method, you go directly to Ignite through its SQL engine, completely bypassing Spark. So everything that is supported in Ignite is available here. User defined functions are also available [1].
[1] https://ignite.apache.org/releases/mobile/org/apache/ignite/cache/query/annotations/QuerySqlFunction.html -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p9027.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
