I have been working on getting various frameworks working on my MapR Cluster that is also running Mesos. Basically, while I know that there is a package from MapR (for Drill) I am trying to find a way to better separate the storage layer from the computer layer.
This isn't a dig on MapR, or any of the Hadoop distributions, it's only I want flexibility to try things, to have an R&D team working with the data in an environment that can try out new frameworks etc. This combination has been very good to me (maybe not to MapR support who received lots of quirky questions from me. They have been helpful in furthering my understanding of this space!) My next project I wanted to play with was Drill. I found https://github.com/mhausenblas/dromedar (Thanks Michael!) as a basic start to a Drill on Mesos approach. I read through the code, I understand it, but I wanted to see it at a more basic level. So I just figured out how to run Drill bits in Marathon (manually for now). Basically, for anyone wanting to play along at home, This actually works VERY well. I used MapR FS to host my package from Drill, I set a conf directory. (Multiple conf directories actually, I set it up so I could launch different "sized" drillbits). I have been able to get things running, and be performant on my small test cluster. For those who may be interested here are some of my notes. - I compiled Drill 1.2.0-SNAPSHOT from source. I ran into some compiling issues that Jacques was able to help me through. Basically, Java 1.8 isn't support for building yet (fails some tests) but there is a work around to that. - I took the built package and placed it in MapR FS. Now, I have every node mounting MapRFS to same NFS location. I could be using a hdfs (maprfs) based tarball but I haven't done that yet. I am just playing around and the NFS mounting of MapRFS sure is handy in this regard. - At first I created a single sized Drill bit, the Marathon JSON is like this: { "cmd": "/mapr/brewpot/mesos/drill/apache-drill-1.2.0-SNAPSHOT/bin/runbit --config /mapr/brewpot/mesos/drill/conf", "cpus": 2.0, "mem": 6144, "id": "drillpot", "instances": 1, "constraints": [["hostname", "UNIQUE"]] } So I can walk you through this. The first is the command obviously. I use runbit instead of drillbit.sh start because I want this process to stay running (from Marathon's perspective). If I used the drillbit.sh, it uses nohup and backgrounds it, Mesos/Marathon thinks it died and tries to start another. cpus: obvious, maybe a bit small, but I have a small cluster. mem: When I set mem to 6144 (6GB) in my drill-env.sh, I set max direct memory to 6GB and max heap to 3GB. I wasn't sure if I needed to set my marathon memory to 9GB or if the heap was used inside the direct memory. I could use some pointers here. id: This is the id of my cluster in the drill-overides.conf. I did this so HA proxy would let me connect to the cluster via drillpot.marathon.mesos and it worked pretty well! instances: I started with one, but could scale up with marathon constrains; I only wanted one drill bit per node because of port conflicts. If I want to be multi tenant and have more than one drill bit per node, I would need to figure out how to abstract the ports. This is something that I could potentially do in a frame work for Mesos. But at the same time, I wonder if if when a drill bit registers with a cluster, it could just "report" it ports in the zookeeper information.. This is intriguing because if it did this, we could allow it to pull random ports offered to it from Mesos, registers the information, and away we go. It would be intriguing. Once I posted this to marathon, all was good, bits started, queries were had by all! It worked well. Some challenges: 1. Ports (as mentioned above) I am not managing those, so port conflicts could occur. 2. I should use a tarball for Marathon, this would allow drill to work on Mesos without the MapR requirement. 3. Logging. I have the default logback.xml in the conf directory and I am getting file not found issues in my stderr on the Mesos tasks. This isn't kill drill, and it still works, but I should organize my logging better. Hopeful for the future: 1. It would be neat to have a frame work that did the actual running of the bits. Perhaps something that could scale up and down based on query usage. I played around with some smaller drillbits (similar to how myriad defines profiles) so I could have a drill cluster of 2 large bits, and 2 small bits on my 5 node cluster. That worked, but lots of manual work. A framework would be handy for managing that. 2. Other? I know this isn't a production thing, but I could see being able to go from this to something a subset of production users could use in MapR/Mesos (or just Mesos) I just wanted to share some of my thought processes and show a way that various tools can integrate. Always happy to talk to shop with folks on this stuff if anyone has any questions. John
