Thanks for this Charm! I've sent it to my colleagues who are working with Drill. We got a lot better performance by storing the data in Parquet format. Drill wasn't able to distribute the queries because the data was stored as json in HDFS.
2016-05-31 17:29 GMT+02:00 Tom Barber <[email protected]>: > (PPS. Going back to our chat about drill speed the other week, I'm not > saying Drill is actually the most performant or best in terms of what > people need, but in terms of a tool to connect to lots of stuff and offer > pretty useful data federation, its top of my list. I do have Phoenix, Kylin > etc sat on the backburner because the should offer more performance but for > more specific usecases.) > > -------------- > > Director Meteorite.bi - Saiku Analytics Founder > Tel: +44(0)5603641316 > > (Thanks to the Saiku community we reached our Kickstart > <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> > goal, but you can always help by sponsoring the project > <http://www.meteorite.bi/products/saiku/sponsorship>) > > On 31 May 2016 at 16:25, Tom Barber <[email protected]> wrote: > >> (I am pretty stoked about this stuff though because it finally brings the >> connectivity Saiku, Pentaho stuff, Spago etc can all leverage to sit on top >> of the Juju Big Data stuff sanely) >> >> -------------- >> >> Director Meteorite.bi - Saiku Analytics Founder >> Tel: +44(0)5603641316 >> >> (Thanks to the Saiku community we reached our Kickstart >> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> >> goal, but you can always help by sponsoring the project >> <http://www.meteorite.bi/products/saiku/sponsorship>) >> >> On 31 May 2016 at 16:21, Tom Barber <[email protected]> wrote: >> >>> Sorry, forgot I mentioned that. >>> >>> There is an unstable drill charm currently because I'm hacking it around >>> and pushing updates to the store whilst I test adding providers as hooks. >>> Bear with me until tomorrow and I'll dump something into the stable >>> channel. It does work, its very basic, missing auth and stuff, but the guts >>> will spin you up a scalable drill cluster. I'll post some stuff tomorrow >>> morning with details. >>> >>> Tom >>> >>> -------------- >>> >>> Director Meteorite.bi - Saiku Analytics Founder >>> Tel: +44(0)5603641316 >>> >>> (Thanks to the Saiku community we reached our Kickstart >>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> >>> goal, but you can always help by sponsoring the project >>> <http://www.meteorite.bi/products/saiku/sponsorship>) >>> >>> On 31 May 2016 at 16:18, Merlijn Sebrechts <[email protected]> >>> wrote: >>> >>>> The Charms for a Drill cluster? Or did I misread that part? >>>> >>>> 2016-05-31 17:16 GMT+02:00 Tom Barber <[email protected]>: >>>> >>>>> Which charms? >>>>> >>>>> -------------- >>>>> >>>>> Director Meteorite.bi - Saiku Analytics Founder >>>>> Tel: +44(0)5603641316 >>>>> >>>>> (Thanks to the Saiku community we reached our Kickstart >>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> >>>>> goal, but you can always help by sponsoring the project >>>>> <http://www.meteorite.bi/products/saiku/sponsorship>) >>>>> >>>>> On 31 May 2016 at 16:16, Merlijn Sebrechts < >>>>> [email protected]> wrote: >>>>> >>>>>> Cool! I'm interested in these Charms. Are they online somewhere? >>>>>> >>>>>> 2016-05-31 16:59 GMT+02:00 Tom Barber <[email protected]>: >>>>>> >>>>>>> Just to clear up this thread. >>>>>>> >>>>>>> Thanks to Kos we tracked the the condition, where Drill was setting >>>>>>> its config before the whole quorum was available. >>>>>>> >>>>>>> For reference I ended up implementing something a bit like: >>>>>>> >>>>>>> https://github.com/juju-solutions/layer-apache-spark/blob/master/reactive/spark.py#L159 >>>>>>> >>>>>>> Thanks guys, I now have a working basic Apache Drill cluster \o/ >>>>>>> >>>>>>> Tom >>>>>>> >>>>>>> -------------- >>>>>>> >>>>>>> Director Meteorite.bi - Saiku Analytics Founder >>>>>>> Tel: +44(0)5603641316 >>>>>>> >>>>>>> (Thanks to the Saiku community we reached our Kickstart >>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> >>>>>>> goal, but you can always help by sponsoring the project >>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>) >>>>>>> >>>>>>> On 31 May 2016 at 13:34, Konstantinos Tsakalozos < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hm... I do not see anything wrong with this part of the code. >>>>>>>> Looking at [0] you should be getting all three Zookeeper >>>>>>>> units/conversations. >>>>>>>> >>>>>>>> Would it be possible to share the charm you are building so that I >>>>>>>> can test it locally or ping us on IRC channel #juju at freenode for a >>>>>>>> pair >>>>>>>> debugging session? >>>>>>>> >>>>>>>> >>>>>>>> [0] >>>>>>>> https://github.com/juju-solutions/interface-zookeeper/blob/master/requires.py#L40 >>>>>>>> >>>>>>>> On Tue, May 31, 2016 at 3:19 PM, Tom Barber < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Okay, slightly different question: >>>>>>>>> >>>>>>>>> zookeeper/0 active idle 2.0-beta7 8 >>>>>>>>> 2181/tcp,9998/tcp 10.106.143.126 Ready (3 zk units) >>>>>>>>> zookeeper/1 active idle 2.0-beta7 9 >>>>>>>>> 2181/tcp,9998/tcp 10.106.143.234 Ready (3 zk units) >>>>>>>>> zookeeper/2 active idle 2.0-beta7 10 >>>>>>>>> 2181/tcp,9998/tcp 10.106.143.133 Ready (3 zk units) >>>>>>>>> >>>>>>>>> I have 3 ZK units and I used the example here: >>>>>>>>> https://github.com/juju-solutions/interface-zookeeper for the >>>>>>>>> client config so my function looks like: >>>>>>>>> >>>>>>>>> @when('zookeeper.ready') >>>>>>>>> @when_not('charm.configured') >>>>>>>>> def configure(zookeeper): >>>>>>>>> zklist = '' >>>>>>>>> for zk_unit in zookeeper.zookeepers(): >>>>>>>>> zklist += add_zookeeper(zk_unit['host'], zk_unit['port']) >>>>>>>>> zklist = zklist[:-1] >>>>>>>>> >>>>>>>>> But it only iterates once even though 3 are listed: >>>>>>>>> >>>>>>>>> a) did I do something wrong >>>>>>>>> b) is this due to some leader election thing and is actually >>>>>>>>> correct? >>>>>>>>> c) another answer? >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> Tom >>>>>>>>> >>>>>>>>> -------------- >>>>>>>>> >>>>>>>>> Director Meteorite.bi - Saiku Analytics Founder >>>>>>>>> Tel: +44(0)5603641316 >>>>>>>>> >>>>>>>>> (Thanks to the Saiku community we reached our Kickstart >>>>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> >>>>>>>>> goal, but you can always help by sponsoring the project >>>>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>) >>>>>>>>> >>>>>>>>> On 31 May 2016 at 12:37, Tom Barber <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Fair enough, that resolves that then! >>>>>>>>>> >>>>>>>>>> Thanks Kos. >>>>>>>>>> >>>>>>>>>> Tom >>>>>>>>>> >>>>>>>>>> -------------- >>>>>>>>>> >>>>>>>>>> Director Meteorite.bi - Saiku Analytics Founder >>>>>>>>>> Tel: +44(0)5603641316 >>>>>>>>>> >>>>>>>>>> (Thanks to the Saiku community we reached our Kickstart >>>>>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> >>>>>>>>>> goal, but you can always help by sponsoring the project >>>>>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>) >>>>>>>>>> >>>>>>>>>> On 31 May 2016 at 12:25, Konstantinos Tsakalozos < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Ah, it is clear now, thanks. >>>>>>>>>>> >>>>>>>>>>> The Apache Hadoop charms use Zookeeper when in HA mode. The >>>>>>>>>>> namenode is the charm that relates to Zookeeper [0,1]. So to >>>>>>>>>>> answer your question the Hadoop charms use ZK but they do not >>>>>>>>>>> carry ZK under the hood. ZK is treated as separate >>>>>>>>>>> service/charm. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [0] https://github.com/juju-solutions/layer-apache-hadoop- >>>>>>>>>>> namenode/blob/master/metadata.yaml >>>>>>>>>>> [1] https://github.com/juju-solutions/layer-apache-hadoop- >>>>>>>>>>> namenode/blob/master/reactive/namenode_failover.py >>>>>>>>>>> >>>>>>>>>>> On Tue, May 31, 2016 at 1:50 PM, Tom Barber < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> No worries Kos >>>>>>>>>>>> >>>>>>>>>>>> I'm rubbish at explaining stuff properly! >>>>>>>>>>>> >>>>>>>>>>>> My query was more along the lines of "I can see the ZK charms >>>>>>>>>>>> available, but surely the Hadoop charms use ZK under the hood >>>>>>>>>>>> somewhere, do >>>>>>>>>>>> any of them expose the ZK relation so you can hook into that", >>>>>>>>>>>> because it >>>>>>>>>>>> seems like overkill running a second ZK setup if one already >>>>>>>>>>>> exists inside >>>>>>>>>>>> an existing Juju Hadoop deployment. >>>>>>>>>>>> >>>>>>>>>>>> Tom >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -------------- >>>>>>>>>>>> >>>>>>>>>>>> Director Meteorite.bi - Saiku Analytics Founder >>>>>>>>>>>> Tel: +44(0)5603641316 >>>>>>>>>>>> >>>>>>>>>>>> (Thanks to the Saiku community we reached our Kickstart >>>>>>>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> >>>>>>>>>>>> goal, but you can always help by sponsoring the project >>>>>>>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>) >>>>>>>>>>>> >>>>>>>>>>>> On 31 May 2016 at 11:47, Konstantinos Tsakalozos < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I am sorry Tom, I replied to your first email. >>>>>>>>>>>>> >>>>>>>>>>>>> Not sure I fully get your question. Not sure what "this" >>>>>>>>>>>>> refers to in "Is there a way to hook this". >>>>>>>>>>>>> >>>>>>>>>>>>> Charms interacting with a Zookeeper quorum need to use this >>>>>>>>>>>>> interface: http://interfaces.juju.solutions/interface/zookeeper/ >>>>>>>>>>>>> . Through the zookeeper interface you can be informed about >>>>>>>>>>>>> zookeeper units >>>>>>>>>>>>> joining and departing the quorum. >>>>>>>>>>>>> >>>>>>>>>>>>> You can have a single Zookeeper service that you can scale to >>>>>>>>>>>>> an odd number of units. Then you can have any charm "requiring" >>>>>>>>>>>>> the above >>>>>>>>>>>>> relation use the zookeeper service. Typically you would need to >>>>>>>>>>>>> construct a >>>>>>>>>>>>> connection string or edit some configuration file within your >>>>>>>>>>>>> charm in >>>>>>>>>>>>> order to reference the zookeeper units. Have a look at https:// >>>>>>>>>>>>> github.com/juju-solutions/layer-apache >>>>>>>>>>>>> -spark/blob/master/lib/charms/layer/apache_spark.py#L175 >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Konstantinos >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 31, 2016 at 1:28 PM, Konstantinos Tsakalozos < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Tom, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Have you looked at: https://jujucharms.com/apache >>>>>>>>>>>>>> -zookeeper/trusty/3 >>>>>>>>>>>>>> The apache-zookeeper provides the zookeeper interface, see >>>>>>>>>>>>>> https://api.jujucharms.com/charmstore/v5/trusty/apache >>>>>>>>>>>>>> -zookeeper-3/archive/metadata.yaml >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Konstantinos >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, May 31, 2016 at 1:13 PM, Tom Barber < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Morning >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> There is a zookeeper interface and a bucket load of Big Data >>>>>>>>>>>>>>> stuff which all must run ZK under the hood. Is there anything >>>>>>>>>>>>>>> that provides >>>>>>>>>>>>>>> the relationit so my charm can hook into ZK? I can't see >>>>>>>>>>>>>>> anything obvious >>>>>>>>>>>>>>> in the Charm Store. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Tom >>>>>>>>>>>>>>> -------------- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Director Meteorite.bi - Saiku Analytics Founder >>>>>>>>>>>>>>> Tel: +44(0)5603641316 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (Thanks to the Saiku community we reached our Kickstart >>>>>>>>>>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> >>>>>>>>>>>>>>> goal, but you can always help by sponsoring the project >>>>>>>>>>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Juju mailing list >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> Modify settings or unsubscribe at: >>>>>>>>>>>>>>> https://lists.ubuntu.com/mailman/listinfo/juju >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Juju mailing list >>>>>>> [email protected] >>>>>>> Modify settings or unsubscribe at: >>>>>>> https://lists.ubuntu.com/mailman/listinfo/juju >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
-- Juju mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
