Hi, let me try to answer some of your questions. >>> is spark a good solution to write realtime to ignite or should i use the datastreamer option of ignite.
Yes, Spark Streaming should work fine for you as long as you still will be using JavaSharedRdd.savePairs [1] method that uses IgniteDataStreamer internally [2]. >>> also when i am using spark my master is yarn 12 clients node of ignite get started with the spark workers. I don't see any issues with this. Anyway, each Spark worker needs to establish a connection to the cluster. Also how do i safely shutdown a ignite cluster. currently i am using kill > -SIGTERM pid to shutdown the cluster. > but many this does not work and cluster remains running mode . You can use Ignite.close programmatically, use JMX Beans or GridGain WebConsole to stop the cluster gradually. [1] https://github.com/dmagda/IgniteSparkIoT/blob/master/src/main/java/org/apache/ignite/iot/SparkStreamerStartup.java#L162 [2] https://github.com/apache/ignite/blob/2e7553aa469a679c8a297e49f50b5464d9d76488/modules/spark/src/main/scala/org/apache/ignite/spark/IgniteRDD.scala#L232 - Denis On Fri, Feb 28, 2020 at 8:22 PM datta <[email protected]> wrote: > Hi, > I am using ignite 2.5.0 with spark 2.2.0 > > i have setup single node of ignite with data region size of max 60gb and > initsize of 10gb with jvm heap size of 32g > > I am streaming data from kafka and writing to ignite table. > > is spark a good solution to write realtime to ignite or should i use the > datastreamer option of ignite. > > also when i am using spark my master is yarn 12 clients node of ignite get > started with the spark workers. > > since my job is running in streaming mode is it safe if so many clients > remain connected to ignite. > > Also how do i safely shutdown a ignite cluster. currently i am using kill > -SIGTERM pid to shutdown the cluster. > but many this does not work and cluster remains running mode . > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
