Hadoop on Mesos use local cdh4 installation instead of tar.gz
Hello, Happy new year 2014 @mesos users. I am trying to get MapReduce cdh4.1.2 running on Mesos. Seams working mostly but few things are still problematic. * MR1 code is already deployed locally with HDFS is there a way to use it instead of tar.gz stored on HDFS to be copied locally and untar. * If not, using tar.gz distribution of cdh4 seams not supporting Snappy compression. is there a way to correct it ? Best regards, -- Damien HARDY signature.asc Description: OpenPGP digital signature
Re: Hadoop on Mesos use local cdh4 installation instead of tar.gz
I'm now able to use snappy compression by adding export JAVA_LIBRARY_PATH=/usr/lib/hadoop/lib/native/ in my /etc/default/mesos-slave (environment variable for mesos-slave process used by my init.d script) This envar is propagated to executor Jvm and so taskTracker can find libsnappy.so to use it. Starting using local deployement of cdh4 ... Reading at the source it seams that something could be done using mapred.mesos.executor.directory and mapred.mesos.executor.command to use local hadoop. Le 31/12/2013 15:08, Damien Hardy a écrit : Hello, Happy new year 2014 @mesos users. I am trying to get MapReduce cdh4.1.2 running on Mesos. Seams working mostly but few things are still problematic. * MR1 code is already deployed locally with HDFS is there a way to use it instead of tar.gz stored on HDFS to be copied locally and untar. * If not, using tar.gz distribution of cdh4 seams not supporting Snappy compression. is there a way to correct it ? Best regards, -- Damien HARDY signature.asc Description: OpenPGP digital signature
Seeking venues for 2014 Mesos hackathons
Hi All, Does your organization have space to host a half-day or all-day Mesos hackathon? Read on! There were several Mesos hackathons in 2013, each increasing the number of new contributors to the project and leading to some great enhancements including the Mesos CLI. In 2014, I'd love to see two improvements to Mesos hackathons: 1) Greater regularity, holding them monthly or every two months. 2) Increased coordination with folks who are remote. I believe the latter is always a challenge, but we'll have a better shot at doing so if hackathons are announced well-in-advance so there's time for coordination. As a first step to organizing hackathons with greater regularity, I'm hoping to compile a list of potential hackathon venues that we could begin to plug into a 2014 calendar. If your organization has available space, please respond to this list to voice your interest. Previous hackathons have been located in the Bay Area since that is where most committers are located geographically, but it would be great to change that in 2014. Also, if you can only offer space and don't have a budget, that's fine -- I'm sure there are folks on this list who don't have space but would be willing to pitch in funds for food and beverages. Lastly, I'm happy to help with future hackathons but would love for a team of folks to help with coordination so this is a community effort. If you're interested, please speak up. Dave
unsubscribe
@vinodkone
