Hey Chris, Thus far I've managed to get the MapR ODBC driver working in my Ubuntu container (unpack the redhat RPM and it works with some tweaking) as well as getting caravel running. Where I am struggling is with Sql Alchemy itself. I thought dialects would be easy to copy and paste, but they are much more involved that I would have thought. I tried just applying a MySQL dialect and it doesn't like it. while I can get data working with no issue in python via pyodbc, I am struggling with the next steps. To me getting a good drill dialect that can be used for Sql Alchemy would solve that problem, however, I need to explore more if it's something I can devote some time too.
Note, based on my other thread, the MapR ODBC driver has a bug (I think) related to the zk.root. If you specify a different zk.root in your drill-override, the ODBC doesn't appear to have (or is not well documented) ((or worst: it is well documented and I missed it) a way to use a different zk.root, and insteal defaults to /drill. So if I have a zk cluster named mybits and a drill root of drillprod, I can't access it via zookeeper and odbc (I can do direct). Thoughts on approaching writing a dialect for Drill for SQL alchemy? Anyone here done that before? John On Fri, May 13, 2016 at 3:48 PM, Christopher Matta <[email protected]> wrote: > John, I've got some sample working code in this notebook for connecting to > Drill via pyodbc: https://github.com/cjmatta/DrillPandasReddit > > I'd be interested in helping integrate this as well. Let us know if you > push out a repo. > > -- > Chris Matta > 215-701-3146 > [email protected] > > On Fri, May 13, 2016 at 2:28 PM, Ted Dunning <[email protected]> > wrote: > > > The limit 0 form is highly to be preferred since that will give you > > information about columns for a view. > > > > This can be set in the dialect for SQL Alchemy. > > > > > > > > On Fri, May 13, 2016 at 11:03 AM, Neeraja Rentachintala < > > [email protected]> wrote: > > > > > Great, thanks John. I wil look forward for an update on how drill > queuing > > > part goes : ) > > > Btw with regards to metadata queries, Drill already supports metadata > > (both > > > the limit 0 form and also show tables/show schemas which are served > from > > > the Information_schema). > > > > > > On Friday, May 13, 2016, John Omernik <[email protected]> wrote: > > > > > > > So with that Docker file, I got caravel working easily with test data > > (no > > > > drill yet) that will be weekend fun (and the pyodbc is already > > installed > > > in > > > > the container, so now it's time to play!) > > > > > > > > So I started my docker image with: > > > > > > > > sudo docker run -it --rm --net=host > > > > -v=/mapr/brewpot/apps/prod/caravel/working:/app/working:rw > > > > -v=/mapr/brewpot/apps/prod/caravel/cache:/app/cache:rw zeta/caravel > > > > /bin/bash > > > > > > > > > > > > Now, I passed through a couple of volumes that I am not sure I will > > > need, I > > > > want to play so that my "State" and initialization is saved in those > > > > directories in the running container (this is just early testing) I > > just > > > > run bash, and then run the commands below and it works. I was lazy > here > > > and > > > > just did net host, it would likely work with bridged mode, but I am > in > > an > > > > airport and wanted to see if I could get it working... the fun part > > will > > > be > > > > working with Drill over the weekend. Thanks again Neeraja for sharing > > > this! > > > > > > > > > > > > > > > > > > > > Then I ran these commands(per the docs) and could explore... pretty > > easy > > > > actually! > > > > > > > > # Create an admin userfabmanager create-admin --app caravel > > > > # Initialize the databasecaravel db upgrade > > > > # Create default roles and permissionscaravel init > > > > # Load some data to play withcaravel load_examples > > > > # Start the development web servercaravel runserver -d > > > > > > > > > > > > > > > > > > > > On Fri, May 13, 2016 at 11:27 AM, John Omernik <[email protected]> > > wrote: > > > > > > > > > So, without running this, but having it build successfully, this > > seems > > > > > like a good place to start, it has caravel, and pyodbc all > installed > > > > here. > > > > > I will be playing more this weekend.... > > > > > > > > > > FROM ubuntu > > > > > > > > > > RUN apt-get update && apt-get install -y build-essential libssl-dev > > > > > libffi-dev python-dev python-pip > > > > > > > > > > RUN apt-get install -y unixodbc-dev unixodbc-bin > > > > > > > > > > RUN pip install pyodbc > > > > > > > > > > RUN pip install caravel > > > > > > > > > > CMD ["python -v"] > > > > > > > > > > On Fri, May 13, 2016 at 10:44 AM, John Omernik <[email protected]> > > > wrote: > > > > > > > > > >> A little more googling and I found the pyodbc, that looks > promising. > > > > >> > > > > >> On Fri, May 13, 2016 at 10:41 AM, John Omernik <[email protected]> > > > > wrote: > > > > >> > > > > >>> "SQL Alchemy already understands Drill" I was just looking for > > that, > > > is > > > > >>> there already some docs/blogs on that? I was going to start there > > as > > > > well > > > > >>> to determine how it worked and then look into the dialect writing > > and > > > > see > > > > >>> how big that project was. I didn't find much on the Drill + > > Alchemy, > > > > but I > > > > >>> am in an airport and I blame wifi gremlins. > > > > >>> > > > > >>> > > > > >>> > > > > >>> On Fri, May 13, 2016 at 10:25 AM, Ted Dunning < > > [email protected] > > > > > > > > >>> wrote: > > > > >>> > > > > >>>> SQLAlchemy generates SQL queries and passes them on to Drill. > > Since > > > > >>>> SQLAlchemy already understands Drill, most of what will be > needed > > is > > > > >>>> slight > > > > >>>> tuning for SQL dialect and providing a mechanism for SQLAlchemy > to > > > get > > > > >>>> meta-data from views. Tableau does the meta-data discovery > using > > > > limit > > > > >>>> 0 > > > > >>>> queries to get column names. We would hope that similar methods > > > would > > > > >>>> work. > > > > >>>> > > > > >>>> > > > > >>>> On Fri, May 13, 2016 at 6:13 AM, Erik Antelman < > > [email protected] > > > > > > > > >>>> wrote: > > > > >>>> > > > > >>>> > Isn't this a matter of Drill<->SQLAlchemy. Such a support > could > > > > likely > > > > >>>> > enable other frameworks. > > > > >>>> > > > > > >>>> > Would one think that adaptation of SQLAlchemy to Drill is > > specific > > > > to > > > > >>>> > Caravel? What subset of features from a RDBMS ORM is > > meaningfull, > > > > >>>> feasable > > > > >>>> > and usefull to map to Drill. This sounds like a broad general > > > > >>>> question. I > > > > >>>> > am sure there are orms from other language camps that might > want > > > > Drill > > > > >>>> > backends. > > > > >>>> > On May 13, 2016 7:33 AM, "John Omernik" <[email protected]> > > wrote: > > > > >>>> > > > > > >>>> > > I will be looking into this as well, thanks for sharing! > > > > >>>> > > On May 13, 2016 2:01 AM, "Nirav Shah" < > > [email protected] > > > > > > > > >>>> wrote: > > > > >>>> > > > > > > >>>> > > > I Hi Neeraja, > > > > >>>> > > > > > > > >>>> > > > I am interested in contributing if integration is not > > > available. > > > > >>>> > > > Kindly let me know > > > > >>>> > > > > > > > >>>> > > > Regards, > > > > >>>> > > > Nirav > > > > >>>> > > > > > > > >>>> > > > On Thu, May 12, 2016 at 9:19 PM, Neeraja Rentachintala < > > > > >>>> > > > [email protected]> wrote: > > > > >>>> > > > > > > > >>>> > > > > Hi Folks > > > > >>>> > > > > > > > > >>>> > > > > Caravel is nice visualization tool recently open sourced > > by > > > > >>>> airbnb. > > > > >>>> > Did > > > > >>>> > > > > anyone try to integrate Drill and/or interested in > > > > contributing > > > > >>>> to > > > > >>>> > > making > > > > >>>> > > > > this work with Drill. > > > > >>>> > > > > > > > > >>>> > > > > https://github.com/airbnb/caravel > > > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > -Thanks > > > > >>>> > > > > Neeraja > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>> > > > > >>> > > > > >> > > > > > > > > > > > > > > >
