Ankur, Response inline.
On Tue, Mar 31, 2015 at 4:49 PM, Ankur Chauhan <achau...@brightcove.com> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi Haoyuan, > > So on each mesos slave node I should allocate/section off some amount > of memory for tachyon (let's say 50% of the total memory) and the rest > for regular mesos tasks? > > This depends on your machine spec and workload. The high level idea is to give Tachyon the memory size equals to the total memory size of the machine minus other processes' memory needs. > This means, on each slave node I would have tachyon worker (+ hdfs > configuration to talk to s3 or the hdfs datanode) and the mesos slave > process. Is this correct? > On each slave node, you would run a Tachyon worker. For underfs, you can configure it to use S3 or HDFS or others. Best, Haoyuan > > On 31/03/2015 16:43, Haoyuan Li wrote: > > Tachyon should be co-located with Spark in this case. > > > > Best, > > > > Haoyuan > > > > On Tue, Mar 31, 2015 at 4:30 PM, Ankur Chauhan > > <achau...@brightcove.com <mailto:achau...@brightcove.com>> wrote: > > > > Hi, > > > > I am fairly new to the spark ecosystem and I have been trying to > > setup a spark on mesos deployment. I can't seem to figure out the > > "best practices" around HDFS and Tachyon. The documentation about > > Spark's data-locality section seems to point that each of my mesos > > slave nodes should also run a hdfs datanode. This seems fine but I > > can't seem to figure out how I would go about pushing tachyon into > > the mix. > > > > How should i organize my cluster? Should tachyon be colocated on my > > mesos worker nodes? or should all the spark jobs reach out to a > > separate hdfs/tachyon cluster. > > > > -- Ankur Chauhan > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > > <mailto:user-unsubscr...@spark.apache.org> For additional commands, > > e-mail: user-h...@spark.apache.org > > <mailto:user-h...@spark.apache.org> > > > > > > > > > > -- Haoyuan Li AMPLab, EECS, UC Berkeley > > http://www.cs.berkeley.edu/~haoyuan/ > > - -- > - -- Ankur Chauhan > -----BEGIN PGP SIGNATURE----- > > iQEcBAEBAgAGBQJVGzKUAAoJEOSJAMhvLp3L3W4IAIVYiEKIZbC1a36/KWo94xYB > dvE4VXxF7z5FWmpuaHBEa+U1XWrR4cLVsQhocusOFn+oC7bstdltt3cGNAuwFSv6 > Oogs4Sl1J4YZm8omKVdCkwD6Hv71HSntM8llz3qTW+Ljk2aKhfvNtp5nioQAm3e+ > bs4ZKlCBij/xV3LbYYIePSS3lL0d9m1qEDJvi6jFcfm3gnBYeNeL9x92B5ylyth0 > BGHnPN4sV/yopgrqOimLb12gSexHGNP1y6JBYy8NrHRY8SxkZ4sWKuyDnGDCOPOc > HC14Parf5Ly5FEz5g5WjF6HrXRdPlgr2ABxSLWOAB/siXsX9o/4yCy7NtDNcL6Y= > =f2xI > -----END PGP SIGNATURE----- > -- Haoyuan Li AMPLab, EECS, UC Berkeley http://www.cs.berkeley.edu/~haoyuan/
