By standalone cluster I just mean a regular Ignite cluster running
independently from Spark. The easiest way is to start a node is using
ignite.sh script providing proper configuration file.
Once you switch IgniteContext to standalone mode, all nodes started within
Spark processes will run in
Ranjit,
Generally, removing and adding nodes in unpredictable way (which happens in
embedded mode because we basically rely on Spark here) is a very bad anti
pattern when working with distributed data. It can have serous performance
implications as well as data loss.
Data nodes are supposed to
Hi Val,
What are the issue you guys have discovered with embededmode deploy ? If we
lost nodes in spark, the back up\replication should take care of those is
what i am thinking.
There is a overhead of rebalancing but how much is that compared to having
a stand alone cluster which we may not
Ranjit,
Then it sounds like you're recreating embedded mode on your own :) In this
case deprecation will not affect you of course, but this still a NOT
recommended way to use Ignite with Spark.
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Thanks Val. We are not using the RDD of ignite instead we are building the
ignite cluster with in spark with custom code and use the key value store.
We are using the static ip discovery while doing so.
What we do is we read the data in avro from hadoop. Start the ignite node
first in driver, and
Ranjit,
Embedded mode in Spark RDD will be deprecated in 2.4 which is about to be
released. My recommendation would be to use standalone mode instead.
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Hello Ranjit!
Running 20 nodes one one server is wasteful. I expect you would need to have
ports range of 20 to run 20 nodes.
This of course changes if each node runs under its own IP address (as it
happens with containers).
Also, client nodes don't need to bind to a fixed port. Any chance you
Hi,
We are using ignite key value store in spark embeded mode. As the resource
is managed by yarn, sometimes when we request for 30 executor nodes, 20
nodes may get into one server. Which means we are starting 20 ignite server
nodes in one server. Currently we have set the discovery port range as
