Ted interesting on the Jira on impersonation. Would that model require Drill to handle scheduling within it's allocated resources then? For example, if I have HR "buy" certain amount of storage/and time and then Marketing Buy a different amount, would a single drill bit be able to help with that allocation?
My hope is with something like Mesos, we could allocated the resources as needed/purchased. I guess, how does Drill do scheduling with a lot of disparate users? John On Tue, Apr 28, 2015 at 12:20 PM, Timothy Chen <[email protected]> wrote: > Getting Drill on Mesos is always an interest of me, but just due to > deadlines don't have time to explore yet. > > Very happy to work with someone on this though! > > Tim > > On Tue, Apr 28, 2015 at 8:58 AM, Ted Dunning <[email protected]> > wrote: > > There is a JIRA open right now, I believe to support impersonation. > > > > With this, a single Drill bit can function on behalf of multiple users, > > adopting the permissions of each user so that the file system can enforce > > security constraints. > > > > There is also research work going on to experiment with ways that > quiescent > > drill bits can decrease their memory footprint until Drill needs to scale > > up due to a large query. This avoids startup time while allowing all the > > benefits of elasticity. Getting this to work with Mesos will require > that > > Mesos allow processes to grow and shrink and also to be told when to > shrink > > due to pressure from other processes (the former is possible, the latter > is > > coming Real Soon Now). > > > > > > > > On Tue, Apr 28, 2015 at 7:50 AM, Andrew Brust < > > [email protected]> wrote: > > > >> I have no wisdom to share, but I am also very interested in the topic > and > >> hope others can advise. I think it's really a key question for Drill -- > >> and even for Big Data and the Data Lake architecture. > >> > >> -----Original Message----- > >> From: John Omernik [mailto:[email protected]] > >> Sent: Tuesday, April 28, 2015 8:44 AM > >> To: user > >> Subject: Drill In the Enterprise > >> > >> Hello - > >> > >> I have been playing with Drill and really see the potential in it. I > >> wanted to start some discussion around Drill in how it could be used in > the > >> Enterprise, specifically in a services orientated architecture. > >> > >> We are exploring Apache Mesos right now (MapRFS as the clustered > >> filesystem) So we I started thinking how we could create "clusters" of > >> Drill Bits. > >> > >> What I am thinking is say you want to have some data for a part of the > >> company, you could allocated disk space to them, and you could allocated > >> resources to them so they could run map reduce (via Myriad) or Spark, or > >> others. Basically it allows us to determine who is utilizing what and > scale > >> as needed. > >> > >> In my testing with Drill, I ran Drill natively next to my MapRFS > processes > >> and then Mesos processes. This is getting away from "managing" my > >> resources with Mesos. If my drill bits are setting outside of Mesos, > then I > >> am not accounting for those resources. In addition, Can I even run > >> multiple drill bits in this setup? What if Marketing and HR both have > data > >> on a node, and I want to run a drill cluster for HR and Drill cluster > for > >> Marketing? Should we just use one cluster and users? Or would this be > good > >> for smaller drill bits that can allocated per department? > >> > >> So if the multiple, smaller drill bits is a good idea (this helps us > limit > >> access to data as well, using unix users and file permissions, using > users > >> in the definitions for DB or Mongo connections etc). This really helps > >> with data access/governance) Then, how best to do this? > >> > >> I suppose we could do this in Docker and run multiple groups of drill > bit > >> in Docker containers to sync things up, but what about a native Mesos > >> Framework? Is this even a good idea? I guess the reason for this post > is I > >> see the how it could be beneficial, but I am not schooled enough in > Mesos > >> or Drill to see the pitfalls, and am curious on the thoughts of the > group, > >> and also just some discussion on the topic of Drill in the Enterprise. > >> >
