Re: Apache Phoenix integration
While it should not be a big problem to provide a cdh5 profile to Apache Drill, my current contribution was just a quick and dirty way to integrate Drill with Phoenix. I think it will be much better to avoid a fork of Apache Drill (i.e. Drillix) and try to merge the 2 things. Then I can work on creating a CDH5 profile for Drill without any problem. What do you think? Is there any possibility to have Drill and Drillix merged soon? Best, Flavio On Mon, Feb 5, 2018 at 9:10 PM, Kunal Khatua <kkha...@mapr.com> wrote: > Hi Flavio > > I'm wondering whether you tried modifying the pom.xml on your private > Github fork to work with CDH 5.11.2 . > > https://github.com/apache/drill/blob/master/pom.xml#L2377 > > We know folks have been using Drill with a variety of Hadoop (and > non-Hadoop) platforms by modifying jars on their private forks, just like > you are. > > However, it benefits the community and Drill when we all can contribute > back such enhancements. Do you think you can try modifying the pom.xml to > successfully build ? If you did hit issues, can you file JIRAs for the > same, so that someone can take a look at it? > > Thanks > Kunal > > -Original Message- > From: Flavio Pompermaier [mailto:pomperma...@okkam.it] > Sent: Monday, February 05, 2018 7:29 AM > To: u...@drill.apache.org; user@phoenix.apache.org > Cc: Bridget Bevens <bbev...@mapr.com>; James Taylor < > jamestay...@apache.org>; Pedro Boado <pedro.bo...@gmail.com> > Subject: Re: Apache Phoenix integration > > Hi all, > in the hope of helping many other enthusiastic users of Apache Phoneix and > Apache Drill, I've just finished to create a dedicated github repository > [1] with all the instruction about how to modify the current Apache Drill > 1.12.0 in order to make it work with Phoenix 4.13.2 on Cloudera CDH 5.11.2. > I've used this version because it is the latest stable one available on > Maven Central containing some important fix about namespaces[2]. > In the README I've also reported some known issues I've encountered but > that probably need some further work in the Apache Drill code base. > Since I use Drill just to sample tables this could be enough for me... > > Looking forward for the deeper integration provided by Drillix (dreaming > in a common effort to integrate its benefint also into the official Drill > version)! > > Best, > Flavio > > [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__github. > com_okkam-2Dit_drill-2Dphoenix-2Dintegration=DwIBaQ= > cskdkSMqhcnjZxdQVpwTXg=-cT6otg6lpT_XkmYy7yg3A= > BQrc4m6Ki2hXeQbOe8XIbthhiEgygxzD16DmbvRBW-I= > cwbSaV0Z9ZgnlzVsRduf6IJwXtjFZwJsHgvKPpKvsbo= > [2] https://urldefense.proofpoint.com/v2/url?u=https-3A__issues. > apache.org_jira_browse_PHOENIX-2D4523=DwIBaQ= > cskdkSMqhcnjZxdQVpwTXg=-cT6otg6lpT_XkmYy7yg3A= > BQrc4m6Ki2hXeQbOe8XIbthhiEgygxzD16DmbvRBW-I=zx8vNOk1VKYBp4KV1Iya4XU_ > JyLRlHKSxQvA1WcFHW4= > > > On Fri, Feb 2, 2018 at 7:21 PM, Kunal Khatua <kkha...@mapr.com> wrote: > > > That's great, Flavio! > > > > You can create a Google doc for review and share it on the user list. > > > > @Bridget handles the documentation on the Apache website, so she can > > do the final touches and help it find a home on the website. > > > > -Original Message- > > From: Flavio Pompermaier [mailto:pomperma...@okkam.it] > > Sent: Friday, February 02, 2018 9:04 AM > > To: u...@drill.apache.org > > Cc: James Taylor <jamestay...@apache.org> > > Subject: Re: Apache Phoenix integration > > > > Eventually I made it to integrate Phoenix with Drill! I debugged > > remotely the drill-embedded via Eclipse and I discovered that the > > problem was that you need some extra jars to make it work! > > Where can I write some documentation about debugging remotely Drill > > from Eclipse and integration with Drill? > > > > On Fri, Feb 2, 2018 at 5:28 PM, Flavio Pompermaier > > <pomperma...@okkam.it> > > wrote: > > > > > What is the fastest way to debug the JDBC plugin from Eclipse? I > > > don't see anything in the logs that could help... > > > Is it possible to connect directly to the external embedded drill > > > running on my machine if I enable jmx? > > > it seems that the JDBC connection is established correctly but Drill > > > throws an Exception (that is not well unwrapped by Jersey): > > > > > > 2018-02-02 16:54:04,520 [qtp159619134-56] INFO > > > o.a.p.q.ConnectionQueryServicesImpl > > > - HConnection established. Stacktrace for informational purposes: > > > hconnection-0x1b9fe9f8 > > >
Re: Apache Phoenix integration
Hi all, in the hope of helping many other enthusiastic users of Apache Phoneix and Apache Drill, I've just finished to create a dedicated github repository [1] with all the instruction about how to modify the current Apache Drill 1.12.0 in order to make it work with Phoenix 4.13.2 on Cloudera CDH 5.11.2. I've used this version because it is the latest stable one available on Maven Central containing some important fix about namespaces[2]. In the README I've also reported some known issues I've encountered but that probably need some further work in the Apache Drill code base. Since I use Drill just to sample tables this could be enough for me... Looking forward for the deeper integration provided by Drillix (dreaming in a common effort to integrate its benefint also into the official Drill version)! Best, Flavio [1] https://github.com/okkam-it/drill-phoenix-integration [2] https://issues.apache.org/jira/browse/PHOENIX-4523 On Fri, Feb 2, 2018 at 7:21 PM, Kunal Khatua <kkha...@mapr.com> wrote: > That's great, Flavio! > > You can create a Google doc for review and share it on the user list. > > @Bridget handles the documentation on the Apache website, so she can do > the final touches and help it find a home on the website. > > -Original Message- > From: Flavio Pompermaier [mailto:pomperma...@okkam.it] > Sent: Friday, February 02, 2018 9:04 AM > To: u...@drill.apache.org > Cc: James Taylor <jamestay...@apache.org> > Subject: Re: Apache Phoenix integration > > Eventually I made it to integrate Phoenix with Drill! I debugged remotely > the drill-embedded via Eclipse and I discovered that the problem was that > you need some extra jars to make it work! > Where can I write some documentation about debugging remotely Drill from > Eclipse and integration with Drill? > > On Fri, Feb 2, 2018 at 5:28 PM, Flavio Pompermaier <pomperma...@okkam.it> > wrote: > > > What is the fastest way to debug the JDBC plugin from Eclipse? I don't > > see anything in the logs that could help... > > Is it possible to connect directly to the external embedded drill > > running on my machine if I enable jmx? > > it seems that the JDBC connection is established correctly but Drill > > throws an Exception (that is not well unwrapped by Jersey): > > > > 2018-02-02 16:54:04,520 [qtp159619134-56] INFO > > o.a.p.q.ConnectionQueryServicesImpl > > - HConnection established. Stacktrace for informational purposes: > > hconnection-0x1b9fe9f8 > > java.lang.Thread.getStackTrace(Thread.java:1552) > > org.apache.phoenix.util.LogUtil.getCallerStackTrace(LogUtil.java:55) > > org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection( > > ConnectionQueryServicesImpl.java:410) > > org.apache.phoenix.query.ConnectionQueryServicesImpl.access$400( > > ConnectionQueryServicesImpl.java:256) > > org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call( > > ConnectionQueryServicesImpl.java:2408) > > org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call( > > ConnectionQueryServicesImpl.java:2384) > > org.apache.phoenix.util.PhoenixContextExecutor.call( > > PhoenixContextExecutor.java:76) > > org.apache.phoenix.query.ConnectionQueryServicesImpl.init( > > ConnectionQueryServicesImpl.java:2384) > > org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices( > > PhoenixDriver.java:255) > > org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection( > > PhoenixEmbeddedDriver.java:150) > > org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221) > > org.apache.commons.dbcp.DriverConnectionFactory.createConnection( > > DriverConnectionFactory.java:38) > > org.apache.commons.dbcp.PoolableConnectionFactory.makeObject( > > PoolableConnectionFactory.java:582) > > org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory( > > BasicDataSource.java:1556) > > org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactor > > y(BasicDataSource.java:1545) > > org.apache.commons.dbcp.BasicDataSource.createDataSource( > > BasicDataSource.java:1388) > > org.apache.commons.dbcp.BasicDataSource.getConnection( > > BasicDataSource.java:1044) > > org.apache.calcite.adapter.jdbc.JdbcUtils$DialectPool. > > get(JdbcUtils.java:73) > > org.apache.calcite.adapter.jdbc.JdbcSchema.createDialect( > > JdbcSchema.java:138) > > org.apache.drill.exec.store.jdbc.JdbcStoragePlugin.( > > JdbcStoragePlugin.java:103) > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > > sun.reflect.NativeConstructorAccessorImpl.newInstance( > > NativeConstructorAccessorImpl.java:62) >
Re: Apache Phoenix integration
This is a very great news! But what do you mean exactly with "Apache Drill is off of it's fork"? When is this deeper integration going to be released/integrated? Will Phoenix (eventually) become just a Drill storage plugin, with huge benefits for both? On 2 Feb 2018 20:20, "James Taylor" <jamestay...@apache.org> wrote: > Good idea - a joint meetup would be great. > > On Fri, Feb 2, 2018 at 11:15 AM, Saurabh Mahapatra < > saurabhmahapatr...@gmail.com> wrote: > >> This is absolutely great news! Thanks for sharing, James! >> >> We should have this as a presentation in one of our weekly Drill hangout >> sessions. >> >> It’s about time for us to do a meetup. A joint meetup perhaps? >> >> Saurabh >> >> Sent from my iPhone >> >> >> >> > On Feb 2, 2018, at 11:13 AM, James Taylor <jamestay...@apache.org> >> wrote: >> > >> > There's also a much deeper integration between Phoenix + Drill (code >> named >> > Drillix) underway that should be possible to complete now that Apache >> Drill >> > is off of it's fork and on a later version of Apache Calcite. I'm hoping >> > that the output of this will be a Phoenix adapter in Drill. See >> > presentation here[1] and WIP code here[2]. >> > >> > Thanks, >> > James >> > >> > [1] http://phoenix.apache.org/presentations/Drillix.pdf >> > [2] https://github.com/jacques-n/drill/tree/phoenix_plugin >> > >> >> On Fri, Feb 2, 2018 at 10:21 AM, Kunal Khatua <kkha...@mapr.com> >> wrote: >> >> >> >> That's great, Flavio! >> >> >> >> You can create a Google doc for review and share it on the user list. >> >> >> >> @Bridget handles the documentation on the Apache website, so she can do >> >> the final touches and help it find a home on the website. >> >> >> >> -Original Message- >> >> From: Flavio Pompermaier [mailto:pomperma...@okkam.it] >> >> Sent: Friday, February 02, 2018 9:04 AM >> >> To: u...@drill.apache.org >> >> Cc: James Taylor <jamestay...@apache.org> >> >> Subject: Re: Apache Phoenix integration >> >> >> >> Eventually I made it to integrate Phoenix with Drill! I debugged >> remotely >> >> the drill-embedded via Eclipse and I discovered that the problem was >> that >> >> you need some extra jars to make it work! >> >> Where can I write some documentation about debugging remotely Drill >> from >> >> Eclipse and integration with Drill? >> >> >> >> On Fri, Feb 2, 2018 at 5:28 PM, Flavio Pompermaier < >> pomperma...@okkam.it> >> >> wrote: >> >> >> >>> What is the fastest way to debug the JDBC plugin from Eclipse? I don't >> >>> see anything in the logs that could help... >> >>> Is it possible to connect directly to the external embedded drill >> >>> running on my machine if I enable jmx? >> >>> it seems that the JDBC connection is established correctly but Drill >> >>> throws an Exception (that is not well unwrapped by Jersey): >> >>> >> >>> 2018-02-02 16:54:04,520 [qtp159619134-56] INFO >> >>> o.a.p.q.ConnectionQueryServicesImpl >> >>> - HConnection established. Stacktrace for informational purposes: >> >>> hconnection-0x1b9fe9f8 >> >>> java.lang.Thread.getStackTrace(Thread.java:1552) >> >>> org.apache.phoenix.util.LogUtil.getCallerStackTrace(LogUtil.java:55) >> >>> org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection( >> >>> ConnectionQueryServicesImpl.java:410) >> >>> org.apache.phoenix.query.ConnectionQueryServicesImpl.access$400( >> >>> ConnectionQueryServicesImpl.java:256) >> >>> org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call( >> >>> ConnectionQueryServicesImpl.java:2408) >> >>> org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call( >> >>> ConnectionQueryServicesImpl.java:2384) >> >>> org.apache.phoenix.util.PhoenixContextExecutor.call( >> >>> PhoenixContextExecutor.java:76) >> >>> org.apache.phoenix.query.ConnectionQueryServicesImpl.init( >> >>> ConnectionQueryServicesImpl.java:2384) >> >>> org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices( >> >>> PhoenixDrive
Re: Apache Phoenix integration
Good idea - a joint meetup would be great. On Fri, Feb 2, 2018 at 11:15 AM, Saurabh Mahapatra < saurabhmahapatr...@gmail.com> wrote: > This is absolutely great news! Thanks for sharing, James! > > We should have this as a presentation in one of our weekly Drill hangout > sessions. > > It’s about time for us to do a meetup. A joint meetup perhaps? > > Saurabh > > Sent from my iPhone > > > > > On Feb 2, 2018, at 11:13 AM, James Taylor <jamestay...@apache.org> > wrote: > > > > There's also a much deeper integration between Phoenix + Drill (code > named > > Drillix) underway that should be possible to complete now that Apache > Drill > > is off of it's fork and on a later version of Apache Calcite. I'm hoping > > that the output of this will be a Phoenix adapter in Drill. See > > presentation here[1] and WIP code here[2]. > > > > Thanks, > > James > > > > [1] http://phoenix.apache.org/presentations/Drillix.pdf > > [2] https://github.com/jacques-n/drill/tree/phoenix_plugin > > > >> On Fri, Feb 2, 2018 at 10:21 AM, Kunal Khatua <kkha...@mapr.com> wrote: > >> > >> That's great, Flavio! > >> > >> You can create a Google doc for review and share it on the user list. > >> > >> @Bridget handles the documentation on the Apache website, so she can do > >> the final touches and help it find a home on the website. > >> > >> -----Original Message- > >> From: Flavio Pompermaier [mailto:pomperma...@okkam.it] > >> Sent: Friday, February 02, 2018 9:04 AM > >> To: u...@drill.apache.org > >> Cc: James Taylor <jamestay...@apache.org> > >> Subject: Re: Apache Phoenix integration > >> > >> Eventually I made it to integrate Phoenix with Drill! I debugged > remotely > >> the drill-embedded via Eclipse and I discovered that the problem was > that > >> you need some extra jars to make it work! > >> Where can I write some documentation about debugging remotely Drill from > >> Eclipse and integration with Drill? > >> > >> On Fri, Feb 2, 2018 at 5:28 PM, Flavio Pompermaier < > pomperma...@okkam.it> > >> wrote: > >> > >>> What is the fastest way to debug the JDBC plugin from Eclipse? I don't > >>> see anything in the logs that could help... > >>> Is it possible to connect directly to the external embedded drill > >>> running on my machine if I enable jmx? > >>> it seems that the JDBC connection is established correctly but Drill > >>> throws an Exception (that is not well unwrapped by Jersey): > >>> > >>> 2018-02-02 16:54:04,520 [qtp159619134-56] INFO > >>> o.a.p.q.ConnectionQueryServicesImpl > >>> - HConnection established. Stacktrace for informational purposes: > >>> hconnection-0x1b9fe9f8 > >>> java.lang.Thread.getStackTrace(Thread.java:1552) > >>> org.apache.phoenix.util.LogUtil.getCallerStackTrace(LogUtil.java:55) > >>> org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection( > >>> ConnectionQueryServicesImpl.java:410) > >>> org.apache.phoenix.query.ConnectionQueryServicesImpl.access$400( > >>> ConnectionQueryServicesImpl.java:256) > >>> org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call( > >>> ConnectionQueryServicesImpl.java:2408) > >>> org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call( > >>> ConnectionQueryServicesImpl.java:2384) > >>> org.apache.phoenix.util.PhoenixContextExecutor.call( > >>> PhoenixContextExecutor.java:76) > >>> org.apache.phoenix.query.ConnectionQueryServicesImpl.init( > >>> ConnectionQueryServicesImpl.java:2384) > >>> org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices( > >>> PhoenixDriver.java:255) > >>> org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection( > >>> PhoenixEmbeddedDriver.java:150) > >>> org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221) > >>> org.apache.commons.dbcp.DriverConnectionFactory.createConnection( > >>> DriverConnectionFactory.java:38) > >>> org.apache.commons.dbcp.PoolableConnectionFactory.makeObject( > >>> PoolableConnectionFactory.java:582) > >>> org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory( > >>> BasicDataSource.java:1556) > >>> org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactor > >>> y(Basi
Re: Apache Phoenix integration
There's also a much deeper integration between Phoenix + Drill (code named Drillix) underway that should be possible to complete now that Apache Drill is off of it's fork and on a later version of Apache Calcite. I'm hoping that the output of this will be a Phoenix adapter in Drill. See presentation here[1] and WIP code here[2]. Thanks, James [1] http://phoenix.apache.org/presentations/Drillix.pdf [2] https://github.com/jacques-n/drill/tree/phoenix_plugin On Fri, Feb 2, 2018 at 10:21 AM, Kunal Khatua <kkha...@mapr.com> wrote: > That's great, Flavio! > > You can create a Google doc for review and share it on the user list. > > @Bridget handles the documentation on the Apache website, so she can do > the final touches and help it find a home on the website. > > -Original Message- > From: Flavio Pompermaier [mailto:pomperma...@okkam.it] > Sent: Friday, February 02, 2018 9:04 AM > To: u...@drill.apache.org > Cc: James Taylor <jamestay...@apache.org> > Subject: Re: Apache Phoenix integration > > Eventually I made it to integrate Phoenix with Drill! I debugged remotely > the drill-embedded via Eclipse and I discovered that the problem was that > you need some extra jars to make it work! > Where can I write some documentation about debugging remotely Drill from > Eclipse and integration with Drill? > > On Fri, Feb 2, 2018 at 5:28 PM, Flavio Pompermaier <pomperma...@okkam.it> > wrote: > > > What is the fastest way to debug the JDBC plugin from Eclipse? I don't > > see anything in the logs that could help... > > Is it possible to connect directly to the external embedded drill > > running on my machine if I enable jmx? > > it seems that the JDBC connection is established correctly but Drill > > throws an Exception (that is not well unwrapped by Jersey): > > > > 2018-02-02 16:54:04,520 [qtp159619134-56] INFO > > o.a.p.q.ConnectionQueryServicesImpl > > - HConnection established. Stacktrace for informational purposes: > > hconnection-0x1b9fe9f8 > > java.lang.Thread.getStackTrace(Thread.java:1552) > > org.apache.phoenix.util.LogUtil.getCallerStackTrace(LogUtil.java:55) > > org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection( > > ConnectionQueryServicesImpl.java:410) > > org.apache.phoenix.query.ConnectionQueryServicesImpl.access$400( > > ConnectionQueryServicesImpl.java:256) > > org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call( > > ConnectionQueryServicesImpl.java:2408) > > org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call( > > ConnectionQueryServicesImpl.java:2384) > > org.apache.phoenix.util.PhoenixContextExecutor.call( > > PhoenixContextExecutor.java:76) > > org.apache.phoenix.query.ConnectionQueryServicesImpl.init( > > ConnectionQueryServicesImpl.java:2384) > > org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices( > > PhoenixDriver.java:255) > > org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection( > > PhoenixEmbeddedDriver.java:150) > > org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221) > > org.apache.commons.dbcp.DriverConnectionFactory.createConnection( > > DriverConnectionFactory.java:38) > > org.apache.commons.dbcp.PoolableConnectionFactory.makeObject( > > PoolableConnectionFactory.java:582) > > org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory( > > BasicDataSource.java:1556) > > org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactor > > y(BasicDataSource.java:1545) > > org.apache.commons.dbcp.BasicDataSource.createDataSource( > > BasicDataSource.java:1388) > > org.apache.commons.dbcp.BasicDataSource.getConnection( > > BasicDataSource.java:1044) > > org.apache.calcite.adapter.jdbc.JdbcUtils$DialectPool. > > get(JdbcUtils.java:73) > > org.apache.calcite.adapter.jdbc.JdbcSchema.createDialect( > > JdbcSchema.java:138) > > org.apache.drill.exec.store.jdbc.JdbcStoragePlugin.( > > JdbcStoragePlugin.java:103) > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > > sun.reflect.NativeConstructorAccessorImpl.newInstance( > > NativeConstructorAccessorImpl.java:62) > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > > DelegatingConstructorAccessorImpl.java:45) > > java.lang.reflect.Constructor.newInstance(Constructor.java:423) > > org.apache.drill.exec.store.StoragePluginRegistryImpl.create( > > StoragePluginRegistryImpl.java:346) > > org.apache.drill.exec.store.StoragePluginRegistryImpl.createOrUpdate( > > StoragePluginRegistryImpl.java:239) > > org.apache.drill.exec.server.rest.PluginConfigWr
