I assume you mean starting an ignite node on the Spark master sets up a thick client cluster
Then does the spark master store any data? Also it is creating a client cluster and client nodes do not store data... Then where is all the distributed in-memory data stored? On Tue, 4 Jul, 2023, 2:41 pm Stephen Darlington, < stephen.darling...@gridgain.com> wrote: > You probably don’t want an Ignite server node on each of your Spark > workers. When you connect to Ignite in your Spark code, it will > automatically start up an Ignite thick-client node. > > On 3 Jul 2023, at 19:24, Arunima Barik <arunimabari...@gmail.com> wrote: > > Great > Thanks for the suggestion > > On Mon, 3 Jul, 2023, 11:23 pm Thomas Kramer, <don.tequ...@gmx.de> wrote: > >> Why can you then not do the same for the Ignite nodes? You create a >> start-ignite-nodes.sh script that has all the host addresses and then run a >> command like "ssh <host-address> start-ignite.sh" for all host ip >> addresses. This of course is most efficient if you have exchanged ssh keys >> between the servers so you don't need to login. >> >> >> On 03.07.23 18:43, Arunima Barik wrote: >> >> We are not using Docker currently. >> >> The java implementation starts only 1 node right. I require many nodes >> and that too on different hosts. >> >> Regarding how I start the Spark workers.. So I run the start-workers.sh >> script and I have already defined all host addresses in workers config >> file. >> >> On Mon, 3 Jul, 2023, 8:51 pm Gianluca Bonetti, < >> gianluca.bone...@gmail.com> wrote: >> >>> Hello Arunima >>> >>> I suppose you run Spark in containers so you can create a custom Docker >>> image to start Apache Ignite. >>> >>> You can also start the server nodes programmatically by Java code. >>> >>> Cheers >>> Gianluca >>> >>> On Mon, 3 Jul 2023 at 16:03, Arunima Barik <arunimabari...@gmail.com> >>> wrote: >>> >>>> I have around 20 spark worker nodes running on different hosts >>>> >>>> I need to start an Ignite node on every such spark worker >>>> >>>> I know that we can ssh into that host and run ignite.sh script to start >>>> the node >>>> >>>> Is there a simpler way to do so .. >>>> Without having to ssh into 20 hosts manually?? >>>> >>>> Regards >>>> Arunima >>>> >>> >
