Hi all, in the hope of helping many other enthusiastic users of Apache Phoneix and Apache Drill, I've just finished to create a dedicated github repository [1] with all the instruction about how to modify the current Apache Drill 1.12.0 in order to make it work with Phoenix 4.13.2 on Cloudera CDH 5.11.2. I've used this version because it is the latest stable one available on Maven Central containing some important fix about namespaces[2]. In the README I've also reported some known issues I've encountered but that probably need some further work in the Apache Drill code base. Since I use Drill just to sample tables this could be enough for me...
Looking forward for the deeper integration provided by Drillix (dreaming in a common effort to integrate its benefint also into the official Drill version)! Best, Flavio [1] https://github.com/okkam-it/drill-phoenix-integration [2] https://issues.apache.org/jira/browse/PHOENIX-4523 On Fri, Feb 2, 2018 at 7:21 PM, Kunal Khatua <kkha...@mapr.com> wrote: > That's great, Flavio! > > You can create a Google doc for review and share it on the user list. > > @Bridget handles the documentation on the Apache website, so she can do > the final touches and help it find a home on the website. > > -----Original Message----- > From: Flavio Pompermaier [mailto:pomperma...@okkam.it] > Sent: Friday, February 02, 2018 9:04 AM > To: user@drill.apache.org > Cc: James Taylor <jamestay...@apache.org> > Subject: Re: Apache Phoenix integration > > Eventually I made it to integrate Phoenix with Drill! I debugged remotely > the drill-embedded via Eclipse and I discovered that the problem was that > you need some extra jars to make it work! > Where can I write some documentation about debugging remotely Drill from > Eclipse and integration with Drill? > > On Fri, Feb 2, 2018 at 5:28 PM, Flavio Pompermaier <pomperma...@okkam.it> > wrote: > > > What is the fastest way to debug the JDBC plugin from Eclipse? I don't > > see anything in the logs that could help... > > Is it possible to connect directly to the external embedded drill > > running on my machine if I enable jmx? > > it seems that the JDBC connection is established correctly but Drill > > throws an Exception (that is not well unwrapped by Jersey): > > > > 2018-02-02 16:54:04,520 [qtp159619134-56] INFO > > o.a.p.q.ConnectionQueryServicesImpl > > - HConnection established. Stacktrace for informational purposes: > > hconnection-0x1b9fe9f8 > > java.lang.Thread.getStackTrace(Thread.java:1552) > > org.apache.phoenix.util.LogUtil.getCallerStackTrace(LogUtil.java:55) > > org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection( > > ConnectionQueryServicesImpl.java:410) > > org.apache.phoenix.query.ConnectionQueryServicesImpl.access$400( > > ConnectionQueryServicesImpl.java:256) > > org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call( > > ConnectionQueryServicesImpl.java:2408) > > org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call( > > ConnectionQueryServicesImpl.java:2384) > > org.apache.phoenix.util.PhoenixContextExecutor.call( > > PhoenixContextExecutor.java:76) > > org.apache.phoenix.query.ConnectionQueryServicesImpl.init( > > ConnectionQueryServicesImpl.java:2384) > > org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices( > > PhoenixDriver.java:255) > > org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection( > > PhoenixEmbeddedDriver.java:150) > > org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221) > > org.apache.commons.dbcp.DriverConnectionFactory.createConnection( > > DriverConnectionFactory.java:38) > > org.apache.commons.dbcp.PoolableConnectionFactory.makeObject( > > PoolableConnectionFactory.java:582) > > org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory( > > BasicDataSource.java:1556) > > org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactor > > y(BasicDataSource.java:1545) > > org.apache.commons.dbcp.BasicDataSource.createDataSource( > > BasicDataSource.java:1388) > > org.apache.commons.dbcp.BasicDataSource.getConnection( > > BasicDataSource.java:1044) > > org.apache.calcite.adapter.jdbc.JdbcUtils$DialectPool. > > get(JdbcUtils.java:73) > > org.apache.calcite.adapter.jdbc.JdbcSchema.createDialect( > > JdbcSchema.java:138) > > org.apache.drill.exec.store.jdbc.JdbcStoragePlugin.<init>( > > JdbcStoragePlugin.java:103) > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > > sun.reflect.NativeConstructorAccessorImpl.newInstance( > > NativeConstructorAccessorImpl.java:62) > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > > DelegatingConstructorAccessorImpl.java:45) > > java.lang.reflect.Constructor.newInstance(Constructor.java:423) > > org.apache.drill.exec.store.StoragePluginRegistryImpl.create( > > StoragePluginRegistryImpl.java:346) > > org.apache.drill.exec.store.StoragePluginRegistryImpl.createOrUpdate( > > StoragePluginRegistryImpl.java:239) > > org.apache.drill.exec.server.rest.PluginConfigWrapper. > > createOrUpdateInStorage(PluginConfigWrapper.java:57) > > org.apache.drill.exec.server.rest.StorageResources. > > createOrUpdatePluginJSON(StorageResources.java:162) > > org.apache.drill.exec.server.rest.StorageResources.createOrUpdatePlugi > > n( > > StorageResources.java:177) > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java: > > 62) > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > > DelegatingMethodAccessorImpl.java:43) > > java.lang.reflect.Method.invoke(Method.java:498) > > org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHan > > dle > > rFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) > > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodD > > isp > > atcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) > > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodD > > isp > > atcher.invoke(AbstractJavaResourceMethodDispatcher.java:171) > > org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatche > > rPr ovider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherPr > > ovider.java:195) > > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodD > > isp > > atcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104) > > org.glassfish.jersey.server.model.ResourceMethodInvoker. > > invoke(ResourceMethodInvoker.java:387) > > org.glassfish.jersey.server.model.ResourceMethodInvoker. > > apply(ResourceMethodInvoker.java:331) > > org.glassfish.jersey.server.model.ResourceMethodInvoker. > > apply(ResourceMethodInvoker.java:103) > > org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269 > > ) > > org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) > > org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) > > org.glassfish.jersey.internal.Errors.process(Errors.java:315) > > org.glassfish.jersey.internal.Errors.process(Errors.java:297) > > org.glassfish.jersey.internal.Errors.process(Errors.java:267) > > org.glassfish.jersey.process.internal.RequestScope. > > runInScope(RequestScope.java:297) > > org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:2 > > 52) org.glassfish.jersey.server.ApplicationHandler.handle( > > ApplicationHandler.java:1023) > > org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:37 > > 2) org.glassfish.jersey.servlet.ServletContainer.service( > > ServletContainer.java:382) > > org.glassfish.jersey.servlet.ServletContainer.service( > > ServletContainer.java:345) > > org.glassfish.jersey.servlet.ServletContainer.service( > > ServletContainer.java:220) > > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738) > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java: > > 551) org.eclipse.jetty.server.session.SessionHandler. > > doHandle(SessionHandler.java:219) > > org.eclipse.jetty.server.handler.ContextHandler. > > doHandle(ContextHandler.java:1111) > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:4 > > 78) org.eclipse.jetty.server.session.SessionHandler. > > doScope(SessionHandler.java:183) > > org.eclipse.jetty.server.handler.ContextHandler. > > doScope(ContextHandler.java:1045) > > org.eclipse.jetty.server.handler.ScopedHandler.handle( > > ScopedHandler.java:141) > > org.eclipse.jetty.server.handler.HandlerWrapper.handle( > > HandlerWrapper.java:97) > > org.eclipse.jetty.server.Server.handle(Server.java:462) > > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) > > org.eclipse.jetty.server.HttpConnection.onFillable( > > HttpConnection.java:232) > > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java: > > 534) org.eclipse.jetty.util.thread.QueuedThreadPool.runJob( > > QueuedThreadPool.java:607) > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run( > > QueuedThreadPool.java:536) > > java.lang.Thread.run(Thread.java:745) > > > > > > On Fri, Feb 2, 2018 at 10:21 AM, Flavio Pompermaier > > <pomperma...@okkam.it> > > wrote: > > > >> Hi Saurabh, > >> I'd like to use Drill because in my use case I have multiple sources > >> (HDFS, HBase, Phoenix, Oracle, Mysql) and I'd like to allow the user > >> to see the data. > >> Without writing a plugin for each storage system I'd like to leverage > >> Apache Drill as a broker towards all of them. > >> > >> Right now I've tried to puth the phoenix-core.jar into the > >> jar/3rdparty folder but when I try to create the Phoenix storage I > >> get the following > >> error: > >> Please retry: error (unable to create/ update storage) > >> > >> I'm trying to debug but I don't see anything wrong.. > >> > >> > >> On Fri, Feb 2, 2018 at 1:42 AM, Saurabh Mahapatra < > >> saurabhmahapatr...@gmail.com> wrote: > >> > >>> Hi Flavio, > >>> > >>> I was curious what your use case is. Phoenix provides SQL access > >>> with secondary indexes on HBase. Why would you want to use Drill? > >>> > >>> Thanks, > >>> Saurabh > >>> > >>> On Thu, Feb 1, 2018 at 3:08 PM, Kunal Khatua <kkha...@mapr.com> wrote: > >>> > >>> > The JDBC storage plugin allows Drill to leverage any SQL system > >>> > that > >>> has > >>> > JDBC drivers, so it should work. > >>> > > >>> > That said, the JDBC storage plugin is a community developed > >>> > storage plugin, so it might not be fully tested. > >>> > > >>> > If you are looking to simply have the Phoenix JDBC driver bundled > >>> > into > >>> the > >>> > Drill 3rd party directory, I (think) the reason it is not bundled > >>> > is because we're not actively tracking compatibility of different > >>> > JDBC > >>> drivers > >>> > with Drill. There could also be license issues, which is why we > >>> > can't bundle drivers for platforms like Oracle, etc. > >>> > > >>> > ~ Kunal > >>> > > >>> > -----Original Message----- > >>> > From: Flavio Pompermaier [mailto:pomperma...@okkam.it] > >>> > Sent: Thursday, February 01, 2018 9:39 AM > >>> > To: user@drill.apache.org > >>> > Subject: Apache Phoenix integration > >>> > > >>> > Hi all, > >>> > looking at https://urldefense.proofpoint. > com/v2/url?u=https-3A__drill. > >>> > apache.org_docs_apache-2Ddrill-2Dcontribution-2Dideas_&d=DwIBaQ&c= > >>> > cskdkSMqhcnjZxdQVpwTXg&r=-cT6otg6lpT_XkmYy7yg3A&m=daBPP_ > >>> > PaenGs721v8erkPwwO543kYzW-gKUv-c1jVWM&s=KMq_T0hogIOIv_ > >>> > ttrCLCxBeEt-AUKKxpcQt3PpeNSbs&e= > >>> > it seems that Apache Phoenix is not integrated with Drill right > >>> > now. In theory the integration should be quite straightforward > >>> > using the JDBC Phoenix connector, isn't it? > >>> > Right now there are just a few problem of class loading when > >>> > putting > >>> the > >>> > phoenix jar into jars/3rdparty but it shouldn't be too hard to > >>> > shade conflicting classes away. > >>> > > >>> > Has anyone attempted to connect to to Apache before? > >>> > > >>> > Best, > >>> > Flavio > >>> > > >>> > >> > >> > >> > >> -- > >> Flavio Pompermaier > >> Development Department > >> > >> OKKAM S.r.l. > >> Tel. +(39) 0461 041809 <+39%200461%20041809> > >> > > > > > > > > > -- > Flavio Pompermaier > Development Department > > OKKAM S.r.l. > Tel. +(39) 0461 041809 > -- Flavio Pompermaier Development Department OKKAM S.r.l. Tel. +(39) 0461 041809
