Yeah I saw that on the Drill list, apologies for not mentioning that earlier! :)
RAM config in the charm will be available shortly, although I've got a meeting in 5 minutes so it'll probably be an hour or so. -------------- Director Meteorite.bi - Saiku Analytics Founder Tel: +44(0)5603641316 (Thanks to the Saiku community we reached our Kickstart <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> goal, but you can always help by sponsoring the project <http://www.meteorite.bi/products/saiku/sponsorship>) On 1 June 2016 at 09:51, Merlijn Sebrechts <[email protected]> wrote: > Thanks for this Charm! I've sent it to my colleagues who are working with > Drill. We got a lot better performance by storing the data in Parquet > format. Drill wasn't able to distribute the queries because the data was > stored as json in HDFS. > > > > 2016-05-31 17:29 GMT+02:00 Tom Barber <[email protected]>: > >> (PPS. Going back to our chat about drill speed the other week, I'm not >> saying Drill is actually the most performant or best in terms of what >> people need, but in terms of a tool to connect to lots of stuff and offer >> pretty useful data federation, its top of my list. I do have Phoenix, Kylin >> etc sat on the backburner because the should offer more performance but for >> more specific usecases.) >> >> -------------- >> >> Director Meteorite.bi - Saiku Analytics Founder >> Tel: +44(0)5603641316 >> >> (Thanks to the Saiku community we reached our Kickstart >> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> >> goal, but you can always help by sponsoring the project >> <http://www.meteorite.bi/products/saiku/sponsorship>) >> >> On 31 May 2016 at 16:25, Tom Barber <[email protected]> wrote: >> >>> (I am pretty stoked about this stuff though because it finally brings >>> the connectivity Saiku, Pentaho stuff, Spago etc can all leverage to sit on >>> top of the Juju Big Data stuff sanely) >>> >>> -------------- >>> >>> Director Meteorite.bi - Saiku Analytics Founder >>> Tel: +44(0)5603641316 >>> >>> (Thanks to the Saiku community we reached our Kickstart >>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> >>> goal, but you can always help by sponsoring the project >>> <http://www.meteorite.bi/products/saiku/sponsorship>) >>> >>> On 31 May 2016 at 16:21, Tom Barber <[email protected]> wrote: >>> >>>> Sorry, forgot I mentioned that. >>>> >>>> There is an unstable drill charm currently because I'm hacking it >>>> around and pushing updates to the store whilst I test adding providers as >>>> hooks. Bear with me until tomorrow and I'll dump something into the stable >>>> channel. It does work, its very basic, missing auth and stuff, but the guts >>>> will spin you up a scalable drill cluster. I'll post some stuff tomorrow >>>> morning with details. >>>> >>>> Tom >>>> >>>> -------------- >>>> >>>> Director Meteorite.bi - Saiku Analytics Founder >>>> Tel: +44(0)5603641316 >>>> >>>> (Thanks to the Saiku community we reached our Kickstart >>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> >>>> goal, but you can always help by sponsoring the project >>>> <http://www.meteorite.bi/products/saiku/sponsorship>) >>>> >>>> On 31 May 2016 at 16:18, Merlijn Sebrechts <[email protected] >>>> > wrote: >>>> >>>>> The Charms for a Drill cluster? Or did I misread that part? >>>>> >>>>> 2016-05-31 17:16 GMT+02:00 Tom Barber <[email protected]>: >>>>> >>>>>> Which charms? >>>>>> >>>>>> -------------- >>>>>> >>>>>> Director Meteorite.bi - Saiku Analytics Founder >>>>>> Tel: +44(0)5603641316 >>>>>> >>>>>> (Thanks to the Saiku community we reached our Kickstart >>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> >>>>>> goal, but you can always help by sponsoring the project >>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>) >>>>>> >>>>>> On 31 May 2016 at 16:16, Merlijn Sebrechts < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Cool! I'm interested in these Charms. Are they online somewhere? >>>>>>> >>>>>>> 2016-05-31 16:59 GMT+02:00 Tom Barber <[email protected]>: >>>>>>> >>>>>>>> Just to clear up this thread. >>>>>>>> >>>>>>>> Thanks to Kos we tracked the the condition, where Drill was setting >>>>>>>> its config before the whole quorum was available. >>>>>>>> >>>>>>>> For reference I ended up implementing something a bit like: >>>>>>>> >>>>>>>> https://github.com/juju-solutions/layer-apache-spark/blob/master/reactive/spark.py#L159 >>>>>>>> >>>>>>>> Thanks guys, I now have a working basic Apache Drill cluster \o/ >>>>>>>> >>>>>>>> Tom >>>>>>>> >>>>>>>> -------------- >>>>>>>> >>>>>>>> Director Meteorite.bi - Saiku Analytics Founder >>>>>>>> Tel: +44(0)5603641316 >>>>>>>> >>>>>>>> (Thanks to the Saiku community we reached our Kickstart >>>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> >>>>>>>> goal, but you can always help by sponsoring the project >>>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>) >>>>>>>> >>>>>>>> On 31 May 2016 at 13:34, Konstantinos Tsakalozos < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hm... I do not see anything wrong with this part of the code. >>>>>>>>> Looking at [0] you should be getting all three Zookeeper >>>>>>>>> units/conversations. >>>>>>>>> >>>>>>>>> Would it be possible to share the charm you are building so that I >>>>>>>>> can test it locally or ping us on IRC channel #juju at freenode for a >>>>>>>>> pair >>>>>>>>> debugging session? >>>>>>>>> >>>>>>>>> >>>>>>>>> [0] >>>>>>>>> https://github.com/juju-solutions/interface-zookeeper/blob/master/requires.py#L40 >>>>>>>>> >>>>>>>>> On Tue, May 31, 2016 at 3:19 PM, Tom Barber < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Okay, slightly different question: >>>>>>>>>> >>>>>>>>>> zookeeper/0 active idle 2.0-beta7 8 >>>>>>>>>> 2181/tcp,9998/tcp 10.106.143.126 Ready (3 zk units) >>>>>>>>>> zookeeper/1 active idle 2.0-beta7 9 >>>>>>>>>> 2181/tcp,9998/tcp 10.106.143.234 Ready (3 zk units) >>>>>>>>>> zookeeper/2 active idle 2.0-beta7 10 >>>>>>>>>> 2181/tcp,9998/tcp 10.106.143.133 Ready (3 zk units) >>>>>>>>>> >>>>>>>>>> I have 3 ZK units and I used the example here: >>>>>>>>>> https://github.com/juju-solutions/interface-zookeeper for the >>>>>>>>>> client config so my function looks like: >>>>>>>>>> >>>>>>>>>> @when('zookeeper.ready') >>>>>>>>>> @when_not('charm.configured') >>>>>>>>>> def configure(zookeeper): >>>>>>>>>> zklist = '' >>>>>>>>>> for zk_unit in zookeeper.zookeepers(): >>>>>>>>>> zklist += add_zookeeper(zk_unit['host'], zk_unit['port']) >>>>>>>>>> zklist = zklist[:-1] >>>>>>>>>> >>>>>>>>>> But it only iterates once even though 3 are listed: >>>>>>>>>> >>>>>>>>>> a) did I do something wrong >>>>>>>>>> b) is this due to some leader election thing and is actually >>>>>>>>>> correct? >>>>>>>>>> c) another answer? >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> Tom >>>>>>>>>> >>>>>>>>>> -------------- >>>>>>>>>> >>>>>>>>>> Director Meteorite.bi - Saiku Analytics Founder >>>>>>>>>> Tel: +44(0)5603641316 >>>>>>>>>> >>>>>>>>>> (Thanks to the Saiku community we reached our Kickstart >>>>>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> >>>>>>>>>> goal, but you can always help by sponsoring the project >>>>>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>) >>>>>>>>>> >>>>>>>>>> On 31 May 2016 at 12:37, Tom Barber <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Fair enough, that resolves that then! >>>>>>>>>>> >>>>>>>>>>> Thanks Kos. >>>>>>>>>>> >>>>>>>>>>> Tom >>>>>>>>>>> >>>>>>>>>>> -------------- >>>>>>>>>>> >>>>>>>>>>> Director Meteorite.bi - Saiku Analytics Founder >>>>>>>>>>> Tel: +44(0)5603641316 >>>>>>>>>>> >>>>>>>>>>> (Thanks to the Saiku community we reached our Kickstart >>>>>>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> >>>>>>>>>>> goal, but you can always help by sponsoring the project >>>>>>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>) >>>>>>>>>>> >>>>>>>>>>> On 31 May 2016 at 12:25, Konstantinos Tsakalozos < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Ah, it is clear now, thanks. >>>>>>>>>>>> >>>>>>>>>>>> The Apache Hadoop charms use Zookeeper when in HA mode. The >>>>>>>>>>>> namenode is the charm that relates to Zookeeper [0,1]. So to >>>>>>>>>>>> answer your question the Hadoop charms use ZK but they do not >>>>>>>>>>>> carry ZK under the hood. ZK is treated as separate >>>>>>>>>>>> service/charm. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [0] https://github.com/juju-solutions/layer-apache-hadoop- >>>>>>>>>>>> namenode/blob/master/metadata.yaml >>>>>>>>>>>> [1] https://github.com/juju-solutions/layer-apache-hadoop- >>>>>>>>>>>> namenode/blob/master/reactive/namenode_failover.py >>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 31, 2016 at 1:50 PM, Tom Barber < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> No worries Kos >>>>>>>>>>>>> >>>>>>>>>>>>> I'm rubbish at explaining stuff properly! >>>>>>>>>>>>> >>>>>>>>>>>>> My query was more along the lines of "I can see the ZK charms >>>>>>>>>>>>> available, but surely the Hadoop charms use ZK under the hood >>>>>>>>>>>>> somewhere, do >>>>>>>>>>>>> any of them expose the ZK relation so you can hook into that", >>>>>>>>>>>>> because it >>>>>>>>>>>>> seems like overkill running a second ZK setup if one already >>>>>>>>>>>>> exists inside >>>>>>>>>>>>> an existing Juju Hadoop deployment. >>>>>>>>>>>>> >>>>>>>>>>>>> Tom >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -------------- >>>>>>>>>>>>> >>>>>>>>>>>>> Director Meteorite.bi - Saiku Analytics Founder >>>>>>>>>>>>> Tel: +44(0)5603641316 >>>>>>>>>>>>> >>>>>>>>>>>>> (Thanks to the Saiku community we reached our Kickstart >>>>>>>>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> >>>>>>>>>>>>> goal, but you can always help by sponsoring the project >>>>>>>>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>) >>>>>>>>>>>>> >>>>>>>>>>>>> On 31 May 2016 at 11:47, Konstantinos Tsakalozos < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I am sorry Tom, I replied to your first email. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Not sure I fully get your question. Not sure what "this" >>>>>>>>>>>>>> refers to in "Is there a way to hook this". >>>>>>>>>>>>>> >>>>>>>>>>>>>> Charms interacting with a Zookeeper quorum need to use this >>>>>>>>>>>>>> interface: http://interfaces.juju.solutions/interface/zookeeper/ >>>>>>>>>>>>>> . Through the zookeeper interface you can be informed about >>>>>>>>>>>>>> zookeeper units >>>>>>>>>>>>>> joining and departing the quorum. >>>>>>>>>>>>>> >>>>>>>>>>>>>> You can have a single Zookeeper service that you can scale to >>>>>>>>>>>>>> an odd number of units. Then you can have any charm "requiring" >>>>>>>>>>>>>> the above >>>>>>>>>>>>>> relation use the zookeeper service. Typically you would need to >>>>>>>>>>>>>> construct a >>>>>>>>>>>>>> connection string or edit some configuration file within your >>>>>>>>>>>>>> charm in >>>>>>>>>>>>>> order to reference the zookeeper units. Have a look at https:// >>>>>>>>>>>>>> github.com/juju-solutions/layer-apache >>>>>>>>>>>>>> -spark/blob/master/lib/charms/layer/apache_spark.py#L175 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Konstantinos >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, May 31, 2016 at 1:28 PM, Konstantinos Tsakalozos < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Tom, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Have you looked at: https://jujucharms.com/apache >>>>>>>>>>>>>>> -zookeeper/trusty/3 >>>>>>>>>>>>>>> The apache-zookeeper provides the zookeeper interface, see >>>>>>>>>>>>>>> https://api.jujucharms.com/charmstore/v5/trusty/apache >>>>>>>>>>>>>>> -zookeeper-3/archive/metadata.yaml >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Konstantinos >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, May 31, 2016 at 1:13 PM, Tom Barber < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Morning >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> There is a zookeeper interface and a bucket load of Big >>>>>>>>>>>>>>>> Data stuff which all must run ZK under the hood. Is there >>>>>>>>>>>>>>>> anything that >>>>>>>>>>>>>>>> provides the relationit so my charm can hook into ZK? I can't >>>>>>>>>>>>>>>> see anything >>>>>>>>>>>>>>>> obvious in the Charm Store. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Tom >>>>>>>>>>>>>>>> -------------- >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Director Meteorite.bi - Saiku Analytics Founder >>>>>>>>>>>>>>>> Tel: +44(0)5603641316 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> (Thanks to the Saiku community we reached our Kickstart >>>>>>>>>>>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> >>>>>>>>>>>>>>>> goal, but you can always help by sponsoring the project >>>>>>>>>>>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Juju mailing list >>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>> Modify settings or unsubscribe at: >>>>>>>>>>>>>>>> https://lists.ubuntu.com/mailman/listinfo/juju >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Juju mailing list >>>>>>>> [email protected] >>>>>>>> Modify settings or unsubscribe at: >>>>>>>> https://lists.ubuntu.com/mailman/listinfo/juju >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
-- Juju mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
