On 13 March 2014 22:19, Mircea Markus <[email protected]> wrote: > > On Mar 13, 2014, at 22:17, Sanne Grinovero <[email protected]> wrote: > >> On 13 March 2014 22:05, Mircea Markus <[email protected]> wrote: >>> >>> On Mar 13, 2014, at 20:59, Ales Justin <[email protected]> wrote: >>> >>>>> - also important to notice that we will have both an Hadoop and an >>>>> Infinispan cluster running in parallel: the user will interact with the >>>>> former in order to run M/R tasks. Hadoop will use Infinispan (integration >>>>> achieved through InputFormat and OutputFormat ) in order to get the data >>>>> to be processed. >>>> >>>> Would this be 2 JVMs, or you can trick Hadoop to start Infinispan as well >>>> -- hence 1JVM? >>> >>> good point, ideally it should be a single VM: reduced serialization cost >>> (in vm access) and simpler architecture. That's if you're not using C/S >>> mode, of course. >> >> ? >> Don't try confusing us again on that :-) >> I think we agreed that the job would *always* run in strict locality >> with the datacontainer (i.e. in the same JVM). Sure, an Hadoop client >> would be connecting from somewhere else but that's unrelated. > > we did discuss the possibility of running it over hotrod though, do you see a > problem with that?
No of course not, we discussed that. I just mean I think that needs to be clarified on the list that the Hadoop engine will always run in the same JVM. Clients (be it Hot Rod via new custom commands or Hadoop native clients, or Hadoop clients over Hot Rod) can indeed connect remotely, but it's important to clarify that the processing itself will take advantage of locality in all configurations. In other words, to clarify that the serialization cost you mention for clients is just to transfer the job definition and optionally the final processing result. Sanne > > Cheers, > -- > Mircea Markus > Infinispan lead (www.infinispan.org) > > > > > > _______________________________________________ > infinispan-dev mailing list > [email protected] > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list [email protected] https://lists.jboss.org/mailman/listinfo/infinispan-dev
