There's also a much deeper integration between Phoenix + Drill (code named Drillix) underway that should be possible to complete now that Apache Drill is off of it's fork and on a later version of Apache Calcite. I'm hoping that the output of this will be a Phoenix adapter in Drill. See presentation here[1] and WIP code here[2].
Thanks, James [1] http://phoenix.apache.org/presentations/Drillix.pdf [2] https://github.com/jacques-n/drill/tree/phoenix_plugin On Fri, Feb 2, 2018 at 10:21 AM, Kunal Khatua <kkha...@mapr.com> wrote: > That's great, Flavio! > > You can create a Google doc for review and share it on the user list. > > @Bridget handles the documentation on the Apache website, so she can do > the final touches and help it find a home on the website. > > -----Original Message----- > From: Flavio Pompermaier [mailto:pomperma...@okkam.it] > Sent: Friday, February 02, 2018 9:04 AM > To: u...@drill.apache.org > Cc: James Taylor <jamestay...@apache.org> > Subject: Re: Apache Phoenix integration > > Eventually I made it to integrate Phoenix with Drill! I debugged remotely > the drill-embedded via Eclipse and I discovered that the problem was that > you need some extra jars to make it work! > Where can I write some documentation about debugging remotely Drill from > Eclipse and integration with Drill? > > On Fri, Feb 2, 2018 at 5:28 PM, Flavio Pompermaier <pomperma...@okkam.it> > wrote: > > > What is the fastest way to debug the JDBC plugin from Eclipse? I don't > > see anything in the logs that could help... > > Is it possible to connect directly to the external embedded drill > > running on my machine if I enable jmx? > > it seems that the JDBC connection is established correctly but Drill > > throws an Exception (that is not well unwrapped by Jersey): > > > > 2018-02-02 16:54:04,520 [qtp159619134-56] INFO > > o.a.p.q.ConnectionQueryServicesImpl > > - HConnection established. Stacktrace for informational purposes: > > hconnection-0x1b9fe9f8 > > java.lang.Thread.getStackTrace(Thread.java:1552) > > org.apache.phoenix.util.LogUtil.getCallerStackTrace(LogUtil.java:55) > > org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection( > > ConnectionQueryServicesImpl.java:410) > > org.apache.phoenix.query.ConnectionQueryServicesImpl.access$400( > > ConnectionQueryServicesImpl.java:256) > > org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call( > > ConnectionQueryServicesImpl.java:2408) > > org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call( > > ConnectionQueryServicesImpl.java:2384) > > org.apache.phoenix.util.PhoenixContextExecutor.call( > > PhoenixContextExecutor.java:76) > > org.apache.phoenix.query.ConnectionQueryServicesImpl.init( > > ConnectionQueryServicesImpl.java:2384) > > org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices( > > PhoenixDriver.java:255) > > org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection( > > PhoenixEmbeddedDriver.java:150) > > org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221) > > org.apache.commons.dbcp.DriverConnectionFactory.createConnection( > > DriverConnectionFactory.java:38) > > org.apache.commons.dbcp.PoolableConnectionFactory.makeObject( > > PoolableConnectionFactory.java:582) > > org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory( > > BasicDataSource.java:1556) > > org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactor > > y(BasicDataSource.java:1545) > > org.apache.commons.dbcp.BasicDataSource.createDataSource( > > BasicDataSource.java:1388) > > org.apache.commons.dbcp.BasicDataSource.getConnection( > > BasicDataSource.java:1044) > > org.apache.calcite.adapter.jdbc.JdbcUtils$DialectPool. > > get(JdbcUtils.java:73) > > org.apache.calcite.adapter.jdbc.JdbcSchema.createDialect( > > JdbcSchema.java:138) > > org.apache.drill.exec.store.jdbc.JdbcStoragePlugin.<init>( > > JdbcStoragePlugin.java:103) > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > > sun.reflect.NativeConstructorAccessorImpl.newInstance( > > NativeConstructorAccessorImpl.java:62) > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > > DelegatingConstructorAccessorImpl.java:45) > > java.lang.reflect.Constructor.newInstance(Constructor.java:423) > > org.apache.drill.exec.store.StoragePluginRegistryImpl.create( > > StoragePluginRegistryImpl.java:346) > > org.apache.drill.exec.store.StoragePluginRegistryImpl.createOrUpdate( > > StoragePluginRegistryImpl.java:239) > > org.apache.drill.exec.server.rest.PluginConfigWrapper. > > createOrUpdateInStorage(PluginConfigWrapper.java:57) > > org.apache.drill.exec.server.rest.StorageResources. > > createOrUpdatePluginJSON(StorageResources.java:162) > > org.apache.drill.exec.server.rest.StorageResources.createOrUpdatePlugi > > n( > > StorageResources.java:177) > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java: > > 62) > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > > DelegatingMethodAccessorImpl.java:43) > > java.lang.reflect.Method.invoke(Method.java:498) > > org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHan > > dle > > rFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) > > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodD > > isp > > atcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) > > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodD > > isp > > atcher.invoke(AbstractJavaResourceMethodDispatcher.java:171) > > org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatche > > rPr ovider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherPr > > ovider.java:195) > > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodD > > isp > > atcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104) > > org.glassfish.jersey.server.model.ResourceMethodInvoker. > > invoke(ResourceMethodInvoker.java:387) > > org.glassfish.jersey.server.model.ResourceMethodInvoker. > > apply(ResourceMethodInvoker.java:331) > > org.glassfish.jersey.server.model.ResourceMethodInvoker. > > apply(ResourceMethodInvoker.java:103) > > org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269 > > ) > > org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) > > org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) > > org.glassfish.jersey.internal.Errors.process(Errors.java:315) > > org.glassfish.jersey.internal.Errors.process(Errors.java:297) > > org.glassfish.jersey.internal.Errors.process(Errors.java:267) > > org.glassfish.jersey.process.internal.RequestScope. > > runInScope(RequestScope.java:297) > > org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:2 > > 52) org.glassfish.jersey.server.ApplicationHandler.handle( > > ApplicationHandler.java:1023) > > org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:37 > > 2) org.glassfish.jersey.servlet.ServletContainer.service( > > ServletContainer.java:382) > > org.glassfish.jersey.servlet.ServletContainer.service( > > ServletContainer.java:345) > > org.glassfish.jersey.servlet.ServletContainer.service( > > ServletContainer.java:220) > > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738) > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java: > > 551) org.eclipse.jetty.server.session.SessionHandler. > > doHandle(SessionHandler.java:219) > > org.eclipse.jetty.server.handler.ContextHandler. > > doHandle(ContextHandler.java:1111) > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:4 > > 78) org.eclipse.jetty.server.session.SessionHandler. > > doScope(SessionHandler.java:183) > > org.eclipse.jetty.server.handler.ContextHandler. > > doScope(ContextHandler.java:1045) > > org.eclipse.jetty.server.handler.ScopedHandler.handle( > > ScopedHandler.java:141) > > org.eclipse.jetty.server.handler.HandlerWrapper.handle( > > HandlerWrapper.java:97) > > org.eclipse.jetty.server.Server.handle(Server.java:462) > > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) > > org.eclipse.jetty.server.HttpConnection.onFillable( > > HttpConnection.java:232) > > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java: > > 534) org.eclipse.jetty.util.thread.QueuedThreadPool.runJob( > > QueuedThreadPool.java:607) > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run( > > QueuedThreadPool.java:536) > > java.lang.Thread.run(Thread.java:745) > > > > > > On Fri, Feb 2, 2018 at 10:21 AM, Flavio Pompermaier > > <pomperma...@okkam.it> > > wrote: > > > >> Hi Saurabh, > >> I'd like to use Drill because in my use case I have multiple sources > >> (HDFS, HBase, Phoenix, Oracle, Mysql) and I'd like to allow the user > >> to see the data. > >> Without writing a plugin for each storage system I'd like to leverage > >> Apache Drill as a broker towards all of them. > >> > >> Right now I've tried to puth the phoenix-core.jar into the > >> jar/3rdparty folder but when I try to create the Phoenix storage I > >> get the following > >> error: > >> Please retry: error (unable to create/ update storage) > >> > >> I'm trying to debug but I don't see anything wrong.. > >> > >> > >> On Fri, Feb 2, 2018 at 1:42 AM, Saurabh Mahapatra < > >> saurabhmahapatr...@gmail.com> wrote: > >> > >>> Hi Flavio, > >>> > >>> I was curious what your use case is. Phoenix provides SQL access > >>> with secondary indexes on HBase. Why would you want to use Drill? > >>> > >>> Thanks, > >>> Saurabh > >>> > >>> On Thu, Feb 1, 2018 at 3:08 PM, Kunal Khatua <kkha...@mapr.com> wrote: > >>> > >>> > The JDBC storage plugin allows Drill to leverage any SQL system > >>> > that > >>> has > >>> > JDBC drivers, so it should work. > >>> > > >>> > That said, the JDBC storage plugin is a community developed > >>> > storage plugin, so it might not be fully tested. > >>> > > >>> > If you are looking to simply have the Phoenix JDBC driver bundled > >>> > into > >>> the > >>> > Drill 3rd party directory, I (think) the reason it is not bundled > >>> > is because we're not actively tracking compatibility of different > >>> > JDBC > >>> drivers > >>> > with Drill. There could also be license issues, which is why we > >>> > can't bundle drivers for platforms like Oracle, etc. > >>> > > >>> > ~ Kunal > >>> > > >>> > -----Original Message----- > >>> > From: Flavio Pompermaier [mailto:pomperma...@okkam.it] > >>> > Sent: Thursday, February 01, 2018 9:39 AM > >>> > To: u...@drill.apache.org > >>> > Subject: Apache Phoenix integration > >>> > > >>> > Hi all, > >>> > looking at https://urldefense.proofpoint. > com/v2/url?u=https-3A__drill. > >>> > apache.org_docs_apache-2Ddrill-2Dcontribution-2Dideas_&d=DwIBaQ&c= > >>> > cskdkSMqhcnjZxdQVpwTXg&r=-cT6otg6lpT_XkmYy7yg3A&m=daBPP_ > >>> > PaenGs721v8erkPwwO543kYzW-gKUv-c1jVWM&s=KMq_T0hogIOIv_ > >>> > ttrCLCxBeEt-AUKKxpcQt3PpeNSbs&e= > >>> > it seems that Apache Phoenix is not integrated with Drill right > >>> > now. In theory the integration should be quite straightforward > >>> > using the JDBC Phoenix connector, isn't it? > >>> > Right now there are just a few problem of class loading when > >>> > putting > >>> the > >>> > phoenix jar into jars/3rdparty but it shouldn't be too hard to > >>> > shade conflicting classes away. > >>> > > >>> > Has anyone attempted to connect to to Apache before? > >>> > > >>> > Best, > >>> > Flavio > >>> > > >>> > >> > >> > >> > >> -- > >> Flavio Pompermaier > >> Development Department > >> > >> OKKAM S.r.l. > >> Tel. +(39) 0461 041809 <+39%200461%20041809> > >> > > > > > > > > > -- > Flavio Pompermaier > Development Department > > OKKAM S.r.l. > Tel. +(39) 0461 041809 >
