Great Thanks a lot Costin.
Are people supposed to deploy the Spark workers on the same ES cluster? I
guess it would make sense for data to remain local and avoid network
transfers altogether?
Thanks a lot,
Mohamed.
On Monday, December 8, 2014 10:19:12 AM UTC-5, Costin Leau wrote:
>
> Hi,
>
> First off I recommend using the native integration (aka the Java/Scala
> APIs) instead of MapReduce. The latter works but
> the former is better performing and more flexible.
>
> ES works in a similar fashion to the HDFS store - the data doesn't go
> through the master rather, each task has its own
> partition on works on its own set of data. Behind the scenes we map each
> worker to an index shard (if there aren't
> enough workers, then some will work across multiple shards).
>
>
> On 12/8/14 4:59 PM, Mohamed Lrhazi wrote:
> > am trying to understand how spark and ES work... could someone please
> help me answer this question..
> >
> > val conf = new Configuration()
> > conf.set("es.resource", "radio/artists")
> > conf.set("es.query", "?q=me*")
> > val esRDD = sc.newHadoopRDD(conf, classOf[EsInputFormat[Text,
> MapWritable]],
> > classOf[Text], classOf[MapWritable]))
> > val docCount = esRDD.count();
> >
> >
> > When and where is data being transferred from ES? is it all collected on
> the Spark master node, then partitioned and
> > sent to the worker nodes? or is each worker node talking to ES to
> somehow get a partition of the data?
> >
> > How does this effectively work?
> >
> > Thanks a lot,
> > Mohamed.
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to
> > [email protected] <javascript:> <mailto:
> [email protected] <javascript:>>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/elasticsearch/CAEU_gmf9Nt0xn_0NbzDn_moRWUT96uWYf4cicJdZik3r0Zz8XA%40mail.gmail.com
>
> > <
> https://groups.google.com/d/msgid/elasticsearch/CAEU_gmf9Nt0xn_0NbzDn_moRWUT96uWYf4cicJdZik3r0Zz8XA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>
>
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> Costin
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/26361977-b5e1-45fa-b305-e59310e2ce3f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
