Arina, There is a v1.0 of the Drill Operator for Kubernetes out, but it is currently tied to the MapR platform. Of the people here, Anton & Denys have tried it out.
The open source version of it that I'm working on is based on that, minus MapR integration and plus Apache integration. My plan is to get a preview out as soon as possible. I don't have an ETA, but I think end of Feb is a timeframe odd like to get something out. Regards, Abhishek On Tue, Feb 4, 2020 at 6:06 PM Arina Yelchiyeva <[email protected]> wrote: > Abhishek, looks like Paul is planning to put on hold his efforts since he > expects you open-source version to be out soon. > Do you have an ETA? > This work is important for the community and it would be nice if is > completed in this release. > > Here Jira to tract the efforts: > https://issues.apache.org/jira/browse/DRILL-7563 < > https://issues.apache.org/jira/browse/DRILL-7563> > > Kind regards, > Arina > > > On Feb 3, 2020, at 10:30 PM, Paul Rogers <[email protected]> > wrote: > > > > Hi Abhishek, > > > > Thanks for the update! Seems to make sense to wait for you to open > source your work than to spend time on duplicating your effort. And, people > who want a solution short term can perhaps work with MapR/HPE as you > suggest. Sounds like you have access to the various systems and have > worked though the myriad details involved in creating a good integration. > > > > Does your work include Helm integration? > > > > The key challenge for any K8s integration is that Drill needs access to > data, which requires some kind of distributed storage. This has long been a > K8s weakness. But, it is, of course, a MapR strength. > > > > Please let us know if you need help with the open source efforts. > > > > Thanks, > > - Paul > > > > > > > > On Monday, February 3, 2020, 3:13:28 AM PST, Abhishek Girish < > [email protected]> wrote: > > > > Hey Ron, > > > > As a part of MapR (now HPE), I've created a native operator for Apache > > Drill and this works on multiple variants of Kubernetes including > > OpenShift. With this, we introduce a new Kind called "DrillCluster" via a > > Custom Resource Definition (CRD) and a Custom Controller (logic to manage > > this DrillCluster kind - written in Golang) for the same. Using this, > users > > can easily deploy Drill clusters by submitting Custom Resource YAML files > > (CRs) for the DrillCluster kind. It supports creation of multiple Drill > > clusters (multiple Drillbits launched in distributed mode), multiple > > versions (such as 1.15.0 and 1.16.0), auto-scaling the number of > Drillbits > > (based on CPU utilization) and more. I can share more details of this if > > anyone's interested. > > > > While Vanilla K8S, and GKE worked out of the box, I had to make some > > changes to support OpenShift (related to Service Accounts, Security > Context > > Constraints, etc). Perhaps you ran into similar issues (I'm yet to read > > this thread fully). > > > > We recently had a v1.0.0 GA release [1], [2] & [3]. One thing to note is > > that the current release has dependencies and integrations with MapR's > > distribution of Apache Drill and is close sourced at the moment (there is > > plan to open source that in the near future). > > > > I have an open source variant of this in the works - to support vanilla > > Apache Drill. In the current state, it has all similar features , it > > removes the MapR specific integration (reliance on MapR-FS instead of > HDFS, > > MapR ZooKeeper and such). I shortly plan to add Apache HDFS and ZooKeeper > > integration instead. Let me know if you're interested - and I can share > the > > GitHub branch. > > > > Regards, > > Abhishek > > > > [1] > > > https://mapr.com/blog/mapr-releases-kubernetes-ecosystem-operators-for-apache-spark-and-apache-drill/ > > [2] > > > https://mapr.com/docs/home/PersistentStorage/running_drillbits_in_compute_space.html > > [3] https://github.com/mapr/mapr-operators > > > > On Wed, Jan 29, 2020 at 11:11 AM Ron Cecchini <[email protected]> > > wrote: > > > >> > >> Hi, all. Drill and OpenShift newbie here. > >> > >> Has anyone successfully deployed a Drill Docker container to an > OpenShift > >> environment? > >> > >> While there is information about Drill Docker, there seems to be zero > >> information about OpenShift in particular. > >> > >> Per the instructions at drill.apache.org/docs/running-drill-on-docker, > I > >> pulled the Drill Docker image from Docker Hub, and then pushed it to our > >> OpenShift environment. But when I tried to deploy it, I immediately ran > >> into an error about /opt/drill/conf/drill-override.conf not being > readable. > >> > >> I understand why the problem is happening (because of who OpenShift runs > >> the container as), so I downloaded the source from GitHub and modified > the > >> Dockerfile to include: > >> > >> RUN chgrp -R 0 /opt/drill && chmod -R g=u /opt/drill > >> > >> so that all of /opt/drill would be available to everyone. But then > >> 'docker build' kept failing, giving the error: > >> > >> Non-resolvable parent POM for > >> org.apache.drill:drill-root:1.18.0-SNAPSHOT: > >> Could not transfer artifact org.apache:apache:pom:21 > >> > >> I tried researching that error but couldn't figure out what was going > on. > >> So I finally decided to start trying to mount persistent volumes, > creating > >> one PV for /opt/drill/conf (and then copying the default > >> drill-override.conf there) and one PV for /opt/drill/log. > >> > >> Now the container gets much further, but eventually fails on something > >> Hadoop related. I'm not trying to do anything with Hadoop, so I don't > know > >> what that's about, but it says I don't have HADOOP_HOME set. > >> > >> Hopefully I can figure out the remaining steps I need (an environment > >> variable? more configs?), but I was wondering if anyone else had > already > >> successfully figured out how to deploy to OpenShift, or might know why > the > >> 'docker build' fails with that error? > >> > >> For what it's worth, I copied over only that drill-override.conf and > >> nothing else. And I did not set any Drill environment variables in > >> OpenShift. I'm basically trying to run the "vanilla" Drill Docker > as-is. > >> > >> Thanks for any help! > >> > >> Ron > >> > >
