Hi Jean, Please see answers inline.
-Gour On 5/6/15, 6:16 AM, "Jean-Baptiste Note" <[email protected]<mailto:[email protected]>> wrote: Hi folks, Currently we're using Chef in our organization to deploy a lot of infrastructure services around Hadoop. Of course it makes a lot of sense to offer these as self-services on YARN using slider, but i'm looking at a number of challenges. So please forgive the broad range of questions :) I'm specifically intersted in deploying the following applications: * HTTPFS service (see https://github.com/jbnote/httpfs-slider) & helpers (nginx) * Opentsdb & helpers (varnish) * kafka (I had a look at koya) * druid * storm (fine, thanks !) * hbase (fine, thanks !) I'm facing a lot of issues with those services which are not yet packaged correctly: * httpfs/opentsdb are not released as standalone tarballs, contrary to all services currently packaged. So i've butchered a tarball from Cloudera RPMs, which is not satisfactory. How would you go about handling this ? Not sure exactly what you mean, by saying "handling this". If you are referring to a way to create a Slider package of an app in rpm format, then there are challenges, such as rpm install requires root access and YARN does not allow that. If you are referring to an issue you are facing with deploying the Slider app (now that you have created a tarball), can you share what issues you are facing? You might also want to take a look at this tomcat Slider package. Caution: It is not ready for prime-time and has few issues which needs to be resolved. But the scripts and metadata files might be a helpful reference. https://issues.apache.org/jira/browse/SLIDER-809 https://github.com/apache/incubator-slider/tree/feature/SLIDER-809-tomcat-app-package/app-packages/tomcat * KOYA has been talked a lot of, however the source i'm looking at ( https://github.com/DataTorrent/koya) is kind of disappointing, and activity is a bit low -- would anyone know if dataTorrent is still committed to the project ? What issues are you facing with KOYA? DataTorrent gave a presentation of KOYA and Slider seems to have fit their need so far. They wanted few features around data locality (strict placement) which will be there in 0.80.0 release AND unique ids which still needs some work to be done. Last but not least, I'm wondering if there would already be a plan to expose somehow (through an internal or an external service) the registry through DNS (that's what we really use for service location for HTTPFS & OpenTSDB). A bash polling script would certainly be sufficient for our needs for now, but longer-term, we'd need to have a more robust solution. Registry and REST APIs on registry comes directly from YARN - https://issues.apache.org/jira/browse/YARN-913 https://issues.apache.org/jira/browse/YARN-2948 http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/registry/yarn-registry.html Thanks a lot, kind regards, JB
