Sorry I have a typo. Which means spark does not use yarn or mesos in standalone mode...
> On Jun 11, 2016, at 14:35, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > > Hi Gavin, > > I believe in standalone mode a simple cluster manager is included with Spark > that makes it easy to set up a cluster. It does not rely on YARN or Mesos. > > In summary this is from my notes: > > Spark Local - Spark runs on the local host. This is the simplest set up and > best suited for learners who want to understand different concepts of Spark > and those performing unit testing. > Spark Standalone – a simple cluster manager included with Spark that makes it > easy to set up a cluster. > YARN Cluster Mode, the Spark driver runs inside an application master process > which is managed by YARN on the cluster, and the client can go away after > initiating the application. > Mesos. I have not used it so cannot comment > YARN Client Mode, the driver runs in the client process, and the application > master is only used for requesting resources from YARN. Unlike Local or Spark > standalone modes, in which the master’s address is specified in the --master > parameter, in YARN mode the ResourceManager’s address is picked up from the > Hadoop configuration. Thus, the --master parameter is yarn > > HTH > > > > > Dr Mich Talebzadeh > > LinkedIn > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > http://talebzadehmich.wordpress.com > > >> On 11 June 2016 at 22:26, Gavin Yue <yue.yuany...@gmail.com> wrote: >> The standalone mode is against Yarn mode or Mesos mode, which means spark >> uses Yarn or Mesos as cluster managements. >> >> Local mode is actually a standalone mode which everything runs on the single >> local machine instead of remote clusters. >> >> That is my understanding. >> >> >>> On Sat, Jun 11, 2016 at 12:40 PM, Ashok Kumar >>> <ashok34...@yahoo.com.invalid> wrote: >>> Thank you for grateful >>> >>> I know I can start spark-shell by launching the shell itself >>> >>> spark-shell >>> >>> Now I know that in standalone mode I can also connect to master >>> >>> spark-shell --master spark://<HOST>:7077 >>> >>> My point is what are the differences between these two start-up modes for >>> spark-shell? If I start spark-shell and connect to master what performance >>> gain will I get if any or it does not matter. Is it the same as for >>> spark-submit >>> >>> regards >>> >>> >>> On Saturday, 11 June 2016, 19:39, Mohammad Tariq <donta...@gmail.com> wrote: >>> >>> >>> Hi Ashok, >>> >>> In local mode all the processes run inside a single jvm, whereas in >>> standalone mode we have separate master and worker processes running in >>> their own jvms. >>> >>> To quickly test your code from within your IDE you could probable use the >>> local mode. However, to get a real feel of how Spark operates I would >>> suggest you to have a standalone setup as well. It's just the matter of >>> launching a standalone cluster either manually(by starting a master and >>> workers by hand), or by using the launch scripts provided with Spark >>> package. >>> >>> You can find more on this here. >>> >>> HTH >>> >>> >>> >>> Tariq, Mohammad >>> about.me/mti >>> >>> >>> >>> >>> >>> On Sat, Jun 11, 2016 at 11:38 PM, Ashok Kumar >>> <ashok34...@yahoo.com.invalid> wrote: >>> Hi, >>> >>> What is the difference between running Spark in Local mode or standalone >>> mode? >>> >>> Are they the same. If they are not which is best suited for non prod work. >>> >>> I am also aware that one can run Spark in Yarn mode as well. >>> >>> Thanks >
