Thank you for your answers… I’m new to both… I’m not sure to understand your answers… I probably should try to reformulate my question…
If hdfs is in Mesos, and Spark too.. 1. - Is there a way to ensure, that there is always a spark and an hdfs instance on the same mesos worker ? - If they are on the same mesos worker, the two services would probably not know they could talk to each other locally ? 2. What I also not sure about, is how to handle the storage, if hdfs is running in mesos ? What happend when hdfs_1 is moved from mesos_worker_1 to mesos_worker_2, do all Data have to be copied ? How are handling this ? From: haosdent [mailto:[email protected]] Sent: mardi 23 juin 2015 15:03 To: [email protected] Subject: Re: mesos be aware of hdfs location using spark By the way, I think you problems are related to HDFS, not related to mesos. You could send it to hdfs user email list. On Tue, Jun 23, 2015 at 9:01 PM, haosdent <[email protected]<mailto:[email protected]>> wrote: And for your this question: >on instances, that also contain hdfs service, to prevent all Data going over >the network ? If you open HDFS Short-Circuit Local Reads, HDFS would auto read from local machine instead of read from network when the data exists in local machine. On Tue, Jun 23, 2015 at 8:58 PM, haosdent <[email protected]<mailto:[email protected]>> wrote: For your second question, I think unless you already have data replica in new hdfs datanode, otherwise hdfs would copy the block from other exist datanode. On Tue, Jun 23, 2015 at 7:51 PM, Sebastien Brennion <[email protected]<mailto:[email protected]>> wrote: Hi, - I would like to know it there is a way to make mesos dispatch in priority spark jobs, on instances, that also contain hdfs service, to prevent all Data going over the network ? - What I also not sure about, is how to handle the storage, if hdfs is running in mesos ? What happend when hdfs_1 is moved from mesos_worker_1 to mesos_worker_2, do all Data have to be copied ? How are handling this ? Regards Sébastien -- Best Regards, Haosdent Huang -- Best Regards, Haosdent Huang -- Best Regards, Haosdent Huang
