We have been using ignite on spark for one of our use cases. We are using Ignite’s SharedRDD feature. Following links should get you started in that direction. We have been using for the basic use case and works fine so far. There is not a whole lot of documentation on spark-ignite integration though. Some pain points that we observed are that it gives serialization errors when used on non-basic data types(UDTs etc.)
https://apacheignite.readme.io/docs/shared-rdd https://apacheignite.readme.io/docs/testing-integration-with-spark-shell -Ravi From: Umesh Kacha <[email protected]<mailto:[email protected]>> Date: Tuesday, January 5, 2016 at 11:47 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Cc: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: RE: Spark on Apache Ingnite? Hi Nate thanks much. I have exact same use cases mentioned by you. My spark job does heavy writing involving group by and huge data shuffling. Can you please provide any pointer how can I run my existing spark job which is running on yarn to make it run on ignite? Please guide. Thanks again. On Jan 6, 2016 02:28, <[email protected]<mailto:[email protected]>> wrote: We started playing with Ignite back Hadoop, hive and spark services, and looking to move to it as our default for deployment going forward, still early but so far its been pretty nice and excited for the flexibility it will provide for our particular use cases. Would say in general its worth looking into if your data workloads are: a) mix of read/write, or heavy write at times b) want write/read access to data from services/apps outside of your spark workloads (old Hadoop jobs, custom apps, etc) c) have strings of spark jobs that could benefit from caching your data across them (think similar usage to tachyon) d) you have sparksql queries that could benefit from indexing and mutability (see pt (a) about mix read/write) If your data is read exclusive and very batch oriented, and your workloads are strictly spark based, benefits will be less and ignite would probably act as more of a tachyon replacement as many of the other features outside of RDD caching wont be leveraged. -----Original Message----- From: unk1102 [mailto:[email protected]<mailto:[email protected]>] Sent: Tuesday, January 5, 2016 10:15 AM To: [email protected]<mailto:[email protected]> Subject: Spark on Apache Ingnite? Hi has anybody tried and had success with Spark on Apache Ignite seems promising? https://ignite.apache.org/ -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Apache-Ingnite- tp25884.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected]<mailto:[email protected]> For additional commands, e-mail: [email protected]<mailto:[email protected]>
