While it should not be a big problem to provide a cdh5 profile to Apache Drill, my current contribution was just a quick and dirty way to integrate Drill with Phoenix. I think it will be much better to avoid a fork of Apache Drill (i.e. Drillix) and try to merge the 2 things. Then I can work on creating a CDH5 profile for Drill without any problem.
What do you think? Is there any possibility to have Drill and Drillix merged soon? Best, Flavio On Mon, Feb 5, 2018 at 9:10 PM, Kunal Khatua <kkha...@mapr.com> wrote: > Hi Flavio > > I'm wondering whether you tried modifying the pom.xml on your private > Github fork to work with CDH 5.11.2 . > > https://github.com/apache/drill/blob/master/pom.xml#L2377 > > We know folks have been using Drill with a variety of Hadoop (and > non-Hadoop) platforms by modifying jars on their private forks, just like > you are. > > However, it benefits the community and Drill when we all can contribute > back such enhancements. Do you think you can try modifying the pom.xml to > successfully build ? If you did hit issues, can you file JIRAs for the > same, so that someone can take a look at it? > > Thanks > Kunal > > -----Original Message----- > From: Flavio Pompermaier [mailto:pomperma...@okkam.it] > Sent: Monday, February 05, 2018 7:29 AM > To: u...@drill.apache.org; user@phoenix.apache.org > Cc: Bridget Bevens <bbev...@mapr.com>; James Taylor < > jamestay...@apache.org>; Pedro Boado <pedro.bo...@gmail.com> > Subject: Re: Apache Phoenix integration > > Hi all, > in the hope of helping many other enthusiastic users of Apache Phoneix and > Apache Drill, I've just finished to create a dedicated github repository > [1] with all the instruction about how to modify the current Apache Drill > 1.12.0 in order to make it work with Phoenix 4.13.2 on Cloudera CDH 5.11.2. > I've used this version because it is the latest stable one available on > Maven Central containing some important fix about namespaces[2]. > In the README I've also reported some known issues I've encountered but > that probably need some further work in the Apache Drill code base. > Since I use Drill just to sample tables this could be enough for me... > > Looking forward for the deeper integration provided by Drillix (dreaming > in a common effort to integrate its benefint also into the official Drill > version)! > > Best, > Flavio > > [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__github. > com_okkam-2Dit_drill-2Dphoenix-2Dintegration&d=DwIBaQ&c= > cskdkSMqhcnjZxdQVpwTXg&r=-cT6otg6lpT_XkmYy7yg3A&m= > BQrc4m6Ki2hXeQbOe8XIbthhiEgygxzD16DmbvRBW-I&s= > cwbSaV0Z9ZgnlzVsRduf6IJwXtjFZwJsHgvKPpKvsbo&e= > [2] https://urldefense.proofpoint.com/v2/url?u=https-3A__issues. > apache.org_jira_browse_PHOENIX-2D4523&d=DwIBaQ&c= > cskdkSMqhcnjZxdQVpwTXg&r=-cT6otg6lpT_XkmYy7yg3A&m= > BQrc4m6Ki2hXeQbOe8XIbthhiEgygxzD16DmbvRBW-I&s=zx8vNOk1VKYBp4KV1Iya4XU_ > JyLRlHKSxQvA1WcFHW4&e= > > > On Fri, Feb 2, 2018 at 7:21 PM, Kunal Khatua <kkha...@mapr.com> wrote: > > > That's great, Flavio! > > > > You can create a Google doc for review and share it on the user list. > > > > @Bridget handles the documentation on the Apache website, so she can > > do the final touches and help it find a home on the website. > > > > -----Original Message----- > > From: Flavio Pompermaier [mailto:pomperma...@okkam.it] > > Sent: Friday, February 02, 2018 9:04 AM > > To: u...@drill.apache.org > > Cc: James Taylor <jamestay...@apache.org> > > Subject: Re: Apache Phoenix integration > > > > Eventually I made it to integrate Phoenix with Drill! I debugged > > remotely the drill-embedded via Eclipse and I discovered that the > > problem was that you need some extra jars to make it work! > > Where can I write some documentation about debugging remotely Drill > > from Eclipse and integration with Drill? > > > > On Fri, Feb 2, 2018 at 5:28 PM, Flavio Pompermaier > > <pomperma...@okkam.it> > > wrote: > > > > > What is the fastest way to debug the JDBC plugin from Eclipse? I > > > don't see anything in the logs that could help... > > > Is it possible to connect directly to the external embedded drill > > > running on my machine if I enable jmx? > > > it seems that the JDBC connection is established correctly but Drill > > > throws an Exception (that is not well unwrapped by Jersey): > > > > > > 2018-02-02 16:54:04,520 [qtp159619134-56] INFO > > > o.a.p.q.ConnectionQueryServicesImpl > > > - HConnection established. Stacktrace for informational purposes: > > > hconnection-0x1b9fe9f8 > > > java.lang.Thread.getStackTrace(Thread.java:1552) > > > org.apache.phoenix.util.LogUtil.getCallerStackTrace(LogUtil.java:55) > > > org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection( > > > ConnectionQueryServicesImpl.java:410) > > > org.apache.phoenix.query.ConnectionQueryServicesImpl.access$400( > > > ConnectionQueryServicesImpl.java:256) > > > org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call( > > > ConnectionQueryServicesImpl.java:2408) > > > org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call( > > > ConnectionQueryServicesImpl.java:2384) > > > org.apache.phoenix.util.PhoenixContextExecutor.call( > > > PhoenixContextExecutor.java:76) > > > org.apache.phoenix.query.ConnectionQueryServicesImpl.init( > > > ConnectionQueryServicesImpl.java:2384) > > > org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices( > > > PhoenixDriver.java:255) > > > org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection( > > > PhoenixEmbeddedDriver.java:150) > > > org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221 > > > ) org.apache.commons.dbcp.DriverConnectionFactory.createConnection( > > > DriverConnectionFactory.java:38) > > > org.apache.commons.dbcp.PoolableConnectionFactory.makeObject( > > > PoolableConnectionFactory.java:582) > > > org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory( > > > BasicDataSource.java:1556) > > > org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFact > > > or > > > y(BasicDataSource.java:1545) > > > org.apache.commons.dbcp.BasicDataSource.createDataSource( > > > BasicDataSource.java:1388) > > > org.apache.commons.dbcp.BasicDataSource.getConnection( > > > BasicDataSource.java:1044) > > > org.apache.calcite.adapter.jdbc.JdbcUtils$DialectPool. > > > get(JdbcUtils.java:73) > > > org.apache.calcite.adapter.jdbc.JdbcSchema.createDialect( > > > JdbcSchema.java:138) > > > org.apache.drill.exec.store.jdbc.JdbcStoragePlugin.<init>( > > > JdbcStoragePlugin.java:103) > > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > > > Method) sun.reflect.NativeConstructorAccessorImpl.newInstance( > > > NativeConstructorAccessorImpl.java:62) > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > > > DelegatingConstructorAccessorImpl.java:45) > > > java.lang.reflect.Constructor.newInstance(Constructor.java:423) > > > org.apache.drill.exec.store.StoragePluginRegistryImpl.create( > > > StoragePluginRegistryImpl.java:346) > > > org.apache.drill.exec.store.StoragePluginRegistryImpl.createOrUpdate > > > ( > > > StoragePluginRegistryImpl.java:239) > > > org.apache.drill.exec.server.rest.PluginConfigWrapper. > > > createOrUpdateInStorage(PluginConfigWrapper.java:57) > > > org.apache.drill.exec.server.rest.StorageResources. > > > createOrUpdatePluginJSON(StorageResources.java:162) > > > org.apache.drill.exec.server.rest.StorageResources.createOrUpdatePlu > > > gi > > > n( > > > StorageResources.java:177) > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > sun.reflect.NativeMethodAccessorImpl.invoke( > > NativeMethodAccessorImpl.java: > > > 62) > > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > > > DelegatingMethodAccessorImpl.java:43) > > > java.lang.reflect.Method.invoke(Method.java:498) > > > org.glassfish.jersey.server.model.internal.ResourceMethodInvocationH > > > an > > > dle > > > rFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) > > > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMetho > > > dD > > > isp > > > atcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) > > > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMetho > > > dD > > > isp > > > atcher.invoke(AbstractJavaResourceMethodDispatcher.java:171) > > > org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatc > > > he rPr > > > ovider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherPr > > > ovider.java:195) > > > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMetho > > > dD > > > isp > > > atcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104) > > > org.glassfish.jersey.server.model.ResourceMethodInvoker. > > > invoke(ResourceMethodInvoker.java:387) > > > org.glassfish.jersey.server.model.ResourceMethodInvoker. > > > apply(ResourceMethodInvoker.java:331) > > > org.glassfish.jersey.server.model.ResourceMethodInvoker. > > > apply(ResourceMethodInvoker.java:103) > > > org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:2 > > > 69 > > > ) > > > org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) > > > org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) > > > org.glassfish.jersey.internal.Errors.process(Errors.java:315) > > > org.glassfish.jersey.internal.Errors.process(Errors.java:297) > > > org.glassfish.jersey.internal.Errors.process(Errors.java:267) > > > org.glassfish.jersey.process.internal.RequestScope. > > > runInScope(RequestScope.java:297) > > > org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java > > > :2 > > > 52) org.glassfish.jersey.server.ApplicationHandler.handle( > > > ApplicationHandler.java:1023) > > > org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java: > > > 37 > > > 2) org.glassfish.jersey.servlet.ServletContainer.service( > > > ServletContainer.java:382) > > > org.glassfish.jersey.servlet.ServletContainer.service( > > > ServletContainer.java:345) > > > org.glassfish.jersey.servlet.ServletContainer.service( > > > ServletContainer.java:220) > > > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:73 > > > 8) > > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java: > > > 551) org.eclipse.jetty.server.session.SessionHandler. > > > doHandle(SessionHandler.java:219) > > > org.eclipse.jetty.server.handler.ContextHandler. > > > doHandle(ContextHandler.java:1111) > > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java > > > :4 > > > 78) org.eclipse.jetty.server.session.SessionHandler. > > > doScope(SessionHandler.java:183) > > > org.eclipse.jetty.server.handler.ContextHandler. > > > doScope(ContextHandler.java:1045) > > > org.eclipse.jetty.server.handler.ScopedHandler.handle( > > > ScopedHandler.java:141) > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle( > > > HandlerWrapper.java:97) > > > org.eclipse.jetty.server.Server.handle(Server.java:462) > > > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) > > > org.eclipse.jetty.server.HttpConnection.onFillable( > > > HttpConnection.java:232) > > > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java: > > > 534) org.eclipse.jetty.util.thread.QueuedThreadPool.runJob( > > > QueuedThreadPool.java:607) > > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run( > > > QueuedThreadPool.java:536) > > > java.lang.Thread.run(Thread.java:745) > > > > > > > > > On Fri, Feb 2, 2018 at 10:21 AM, Flavio Pompermaier > > > <pomperma...@okkam.it> > > > wrote: > > > > > >> Hi Saurabh, > > >> I'd like to use Drill because in my use case I have multiple > > >> sources (HDFS, HBase, Phoenix, Oracle, Mysql) and I'd like to allow > > >> the user to see the data. > > >> Without writing a plugin for each storage system I'd like to > > >> leverage Apache Drill as a broker towards all of them. > > >> > > >> Right now I've tried to puth the phoenix-core.jar into the > > >> jar/3rdparty folder but when I try to create the Phoenix storage I > > >> get the following > > >> error: > > >> Please retry: error (unable to create/ update storage) > > >> > > >> I'm trying to debug but I don't see anything wrong.. > > >> > > >> > > >> On Fri, Feb 2, 2018 at 1:42 AM, Saurabh Mahapatra < > > >> saurabhmahapatr...@gmail.com> wrote: > > >> > > >>> Hi Flavio, > > >>> > > >>> I was curious what your use case is. Phoenix provides SQL access > > >>> with secondary indexes on HBase. Why would you want to use Drill? > > >>> > > >>> Thanks, > > >>> Saurabh > > >>> > > >>> On Thu, Feb 1, 2018 at 3:08 PM, Kunal Khatua <kkha...@mapr.com> > wrote: > > >>> > > >>> > The JDBC storage plugin allows Drill to leverage any SQL system > > >>> > that > > >>> has > > >>> > JDBC drivers, so it should work. > > >>> > > > >>> > That said, the JDBC storage plugin is a community developed > > >>> > storage plugin, so it might not be fully tested. > > >>> > > > >>> > If you are looking to simply have the Phoenix JDBC driver > > >>> > bundled into > > >>> the > > >>> > Drill 3rd party directory, I (think) the reason it is not > > >>> > bundled is because we're not actively tracking compatibility of > > >>> > different JDBC > > >>> drivers > > >>> > with Drill. There could also be license issues, which is why we > > >>> > can't bundle drivers for platforms like Oracle, etc. > > >>> > > > >>> > ~ Kunal > > >>> > > > >>> > -----Original Message----- > > >>> > From: Flavio Pompermaier [mailto:pomperma...@okkam.it] > > >>> > Sent: Thursday, February 01, 2018 9:39 AM > > >>> > To: u...@drill.apache.org > > >>> > Subject: Apache Phoenix integration > > >>> > > > >>> > Hi all, > > >>> > looking at https://urldefense.proofpoint.com/v2/url?u=https-3A__ > urldefense.proofpoint&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=- > cT6otg6lpT_XkmYy7yg3A&m=BQrc4m6Ki2hXeQbOe8XIbthhiEgygxzD16DmbvRBW-I&s= > Zo6W4xGsVq4Xm2yeF9bm7mEgpoNL0xzjM0YbvG6buas&e=. > > com/v2/url?u=https-3A__drill. > > >>> > apache.org_docs_apache-2Ddrill-2Dcontribution-2Dideas_&d=DwIBaQ& > > >>> > c= cskdkSMqhcnjZxdQVpwTXg&r=-cT6otg6lpT_XkmYy7yg3A&m=daBPP_ > > >>> > PaenGs721v8erkPwwO543kYzW-gKUv-c1jVWM&s=KMq_T0hogIOIv_ > > >>> > ttrCLCxBeEt-AUKKxpcQt3PpeNSbs&e= it seems that Apache Phoenix is > > >>> > not integrated with Drill right now. In theory the integration > > >>> > should be quite straightforward using the JDBC Phoenix > > >>> > connector, isn't it? > > >>> > Right now there are just a few problem of class loading when > > >>> > putting > > >>> the > > >>> > phoenix jar into jars/3rdparty but it shouldn't be too hard to > > >>> > shade conflicting classes away. > > >>> > > > >>> > Has anyone attempted to connect to to Apache before? > > >>> > > > >>> > Best, > > >>> > Flavio > > >>> > > > >>> > > >> > > >> > > >> > > >> -- > > >> Flavio Pompermaier > > >> Development Department > > >> > > >> OKKAM S.r.l. > > >> Tel. +(39) 0461 041809 <+39%200461%20041809> > > >> > > > > > > > > > > > > > > > -- > > Flavio Pompermaier > > Development Department > > > > OKKAM S.r.l. > > Tel. +(39) 0461 041809 > > > > > > -- > Flavio Pompermaier > Development Department > > OKKAM S.r.l. > Tel. +(39) 0461 041809 >
