Re: Apache Drill Charm

2016-06-02 Thread Tom Barber 
Hi folks:

http://spicule.co.uk/2016/06/02/apache-drill-juju.html

I've not slept enough due to ill kids and travel, so it might be complete
nonsense, but here is a brain dump about some of Drills bits and pieces.

The charm works well enough for basic use and you can certainly connect it
to a bunch of stuff, but I need to add a load more config/relation/action
stuff to solve the automation aspect.

Anyway, fill ya boots.

Tom

--

Director Meteorite.bi - Saiku Analytics Founder
Tel: +44(0)5603641316

(Thanks to the Saiku community we reached our Kickstart

goal, but you can always help by sponsoring the project
)

On 1 June 2016 at 13:47, Merlijn Sebrechts 
wrote:

> Yeah, we should see if we can do the same with YARN...
>
> 2016-06-01 14:42 GMT+02:00 Marco Ceppi :
>
>> +1 to using % logic to make it scale across any sized instance. Awesome
>> stuff!
>>
>> On Wed, Jun 1, 2016 at 6:28 AM Tom Barber 
>> wrote:
>>
>>> Okay latest "stable" build has RAM config options.
>>>
>>> Drill ships with defaults of 8GB and 3GB but I didn't want it to die on
>>> EC2 Large etc boxes that dont have that much. So I added a bit of logic,
>>> you can (I hope) add XXG and it will use that fixed amount, or you can, as
>>> it ships, tell it you want XX% MAX and XX% Heap and it will try and figure
>>> that out and stand you up a drill box.
>>>
>>>
>>> Tom
>>>
>>> --
>>>
>>> Director Meteorite.bi - Saiku Analytics Founder
>>> Tel: +44(0)5603641316
>>>
>>> (Thanks to the Saiku community we reached our Kickstart
>>> 
>>> goal, but you can always help by sponsoring the project
>>> )
>>>
>>> On 1 June 2016 at 00:50, Tom Barber  wrote:
>>>
 Oh, also currently the RAM is clamped down real low
 in /opt/drill/conf/drill-env.sh I will set it back to some sane defaults
 tomorrow as soon as I put the RAM limits into the config options, just ran
 out of time this evening!

 --

 Director Meteorite.bi - Saiku Analytics Founder
 Tel: +44(0)5603641316

 (Thanks to the Saiku community we reached our Kickstart
 
 goal, but you can always help by sponsoring the project
 )

 On 31 May 2016 at 23:50, Tom Barber  wrote:

> Here we are then, for Merlijn and anyone else interested in SQL
> interfaces to big data/NOSQL stuff.
>
> This is less than a days effort, so its patchy at best:
>
> https://jujucharms.com/u/spicule/drillbit
>
> For those of you who don't know Apache Drill, it will let you run SQL
> querys over, CSV/JSON data, MongoDB, HBase, Parquet files etc in a number
> of different locations. Basically its a great way for analysts who use
> "traditional" SQL tools to leverage data stored within NOSQL solutions.
>
> Getting something like this into the CS has been high on my list of
> priorities for Saiku Analytics as it suddenly offers up loads of new
> connection prospects(of course I can do this manually in the past, but 
> this
> is what Juju is for, right?)
>
> You need to deploy a ZK node (or 3) and connect it to that and OpenJDK
> to run it. Currently its relations-lite, the only one in there is a 
> MongoDB
> test relation that will connect Drill to your MongoDB cluster if you run
> one, but there will be actions and relations coming shortly for other
> stuff. Also its missing a fat load of config options, again, coming soon.
> You can set all of this stuff pretty simply though and there is a web
> console for queries/connections etc, on http://serverip:8047/
>
> Test it, let me know what you think or its its entirely broken. Adding
> this and a few other NOSQL SQL interfaces is key to people being able to
> sanely consume all this big data stuff that is great on Juju. Thats not a
> knock on the Zeppelin guys, but companies will continue to use SQL for a
> long time to come so we should service that requirement.
>
> Tom
>
> --
>
> Director Meteorite.bi - Saiku Analytics Founder
> Tel: +44(0)5603641316
>
> (Thanks to the Saiku community we reached our Kickstart
> 
> goal, but you can always help by sponsoring the project
> )
>


>>> --
>>> Juju mailing

Re: Apache Drill Charm

2016-06-01 Thread Merlijn Sebrechts 
Yeah, we should see if we can do the same with YARN...

2016-06-01 14:42 GMT+02:00 Marco Ceppi :

> +1 to using % logic to make it scale across any sized instance. Awesome
> stuff!
>
> On Wed, Jun 1, 2016 at 6:28 AM Tom Barber  wrote:
>
>> Okay latest "stable" build has RAM config options.
>>
>> Drill ships with defaults of 8GB and 3GB but I didn't want it to die on
>> EC2 Large etc boxes that dont have that much. So I added a bit of logic,
>> you can (I hope) add XXG and it will use that fixed amount, or you can, as
>> it ships, tell it you want XX% MAX and XX% Heap and it will try and figure
>> that out and stand you up a drill box.
>>
>>
>> Tom
>>
>> --
>>
>> Director Meteorite.bi - Saiku Analytics Founder
>> Tel: +44(0)5603641316
>>
>> (Thanks to the Saiku community we reached our Kickstart
>> 
>> goal, but you can always help by sponsoring the project
>> )
>>
>> On 1 June 2016 at 00:50, Tom Barber  wrote:
>>
>>> Oh, also currently the RAM is clamped down real low
>>> in /opt/drill/conf/drill-env.sh I will set it back to some sane defaults
>>> tomorrow as soon as I put the RAM limits into the config options, just ran
>>> out of time this evening!
>>>
>>> --
>>>
>>> Director Meteorite.bi - Saiku Analytics Founder
>>> Tel: +44(0)5603641316
>>>
>>> (Thanks to the Saiku community we reached our Kickstart
>>> 
>>> goal, but you can always help by sponsoring the project
>>> )
>>>
>>> On 31 May 2016 at 23:50, Tom Barber  wrote:
>>>
 Here we are then, for Merlijn and anyone else interested in SQL
 interfaces to big data/NOSQL stuff.

 This is less than a days effort, so its patchy at best:

 https://jujucharms.com/u/spicule/drillbit

 For those of you who don't know Apache Drill, it will let you run SQL
 querys over, CSV/JSON data, MongoDB, HBase, Parquet files etc in a number
 of different locations. Basically its a great way for analysts who use
 "traditional" SQL tools to leverage data stored within NOSQL solutions.

 Getting something like this into the CS has been high on my list of
 priorities for Saiku Analytics as it suddenly offers up loads of new
 connection prospects(of course I can do this manually in the past, but this
 is what Juju is for, right?)

 You need to deploy a ZK node (or 3) and connect it to that and OpenJDK
 to run it. Currently its relations-lite, the only one in there is a MongoDB
 test relation that will connect Drill to your MongoDB cluster if you run
 one, but there will be actions and relations coming shortly for other
 stuff. Also its missing a fat load of config options, again, coming soon.
 You can set all of this stuff pretty simply though and there is a web
 console for queries/connections etc, on http://serverip:8047/

 Test it, let me know what you think or its its entirely broken. Adding
 this and a few other NOSQL SQL interfaces is key to people being able to
 sanely consume all this big data stuff that is great on Juju. Thats not a
 knock on the Zeppelin guys, but companies will continue to use SQL for a
 long time to come so we should service that requirement.

 Tom

 --

 Director Meteorite.bi - Saiku Analytics Founder
 Tel: +44(0)5603641316

 (Thanks to the Saiku community we reached our Kickstart
 
 goal, but you can always help by sponsoring the project
 )

>>>
>>>
>> --
>> Juju mailing list
>> Juju@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju
>>
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Apache Drill Charm

2016-06-01 Thread Marco Ceppi 
+1 to using % logic to make it scale across any sized instance. Awesome
stuff!

On Wed, Jun 1, 2016 at 6:28 AM Tom Barber  wrote:

> Okay latest "stable" build has RAM config options.
>
> Drill ships with defaults of 8GB and 3GB but I didn't want it to die on
> EC2 Large etc boxes that dont have that much. So I added a bit of logic,
> you can (I hope) add XXG and it will use that fixed amount, or you can, as
> it ships, tell it you want XX% MAX and XX% Heap and it will try and figure
> that out and stand you up a drill box.
>
>
> Tom
>
> --
>
> Director Meteorite.bi - Saiku Analytics Founder
> Tel: +44(0)5603641316
>
> (Thanks to the Saiku community we reached our Kickstart
> 
> goal, but you can always help by sponsoring the project
> )
>
> On 1 June 2016 at 00:50, Tom Barber  wrote:
>
>> Oh, also currently the RAM is clamped down real low
>> in /opt/drill/conf/drill-env.sh I will set it back to some sane defaults
>> tomorrow as soon as I put the RAM limits into the config options, just ran
>> out of time this evening!
>>
>> --
>>
>> Director Meteorite.bi - Saiku Analytics Founder
>> Tel: +44(0)5603641316
>>
>> (Thanks to the Saiku community we reached our Kickstart
>> 
>> goal, but you can always help by sponsoring the project
>> )
>>
>> On 31 May 2016 at 23:50, Tom Barber  wrote:
>>
>>> Here we are then, for Merlijn and anyone else interested in SQL
>>> interfaces to big data/NOSQL stuff.
>>>
>>> This is less than a days effort, so its patchy at best:
>>>
>>> https://jujucharms.com/u/spicule/drillbit
>>>
>>> For those of you who don't know Apache Drill, it will let you run SQL
>>> querys over, CSV/JSON data, MongoDB, HBase, Parquet files etc in a number
>>> of different locations. Basically its a great way for analysts who use
>>> "traditional" SQL tools to leverage data stored within NOSQL solutions.
>>>
>>> Getting something like this into the CS has been high on my list of
>>> priorities for Saiku Analytics as it suddenly offers up loads of new
>>> connection prospects(of course I can do this manually in the past, but this
>>> is what Juju is for, right?)
>>>
>>> You need to deploy a ZK node (or 3) and connect it to that and OpenJDK
>>> to run it. Currently its relations-lite, the only one in there is a MongoDB
>>> test relation that will connect Drill to your MongoDB cluster if you run
>>> one, but there will be actions and relations coming shortly for other
>>> stuff. Also its missing a fat load of config options, again, coming soon.
>>> You can set all of this stuff pretty simply though and there is a web
>>> console for queries/connections etc, on http://serverip:8047/
>>>
>>> Test it, let me know what you think or its its entirely broken. Adding
>>> this and a few other NOSQL SQL interfaces is key to people being able to
>>> sanely consume all this big data stuff that is great on Juju. Thats not a
>>> knock on the Zeppelin guys, but companies will continue to use SQL for a
>>> long time to come so we should service that requirement.
>>>
>>> Tom
>>>
>>> --
>>>
>>> Director Meteorite.bi - Saiku Analytics Founder
>>> Tel: +44(0)5603641316
>>>
>>> (Thanks to the Saiku community we reached our Kickstart
>>> 
>>> goal, but you can always help by sponsoring the project
>>> )
>>>
>>
>>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Apache Drill Charm

2016-06-01 Thread Tom Barber 
Okay latest "stable" build has RAM config options.

Drill ships with defaults of 8GB and 3GB but I didn't want it to die on EC2
Large etc boxes that dont have that much. So I added a bit of logic, you
can (I hope) add XXG and it will use that fixed amount, or you can, as it
ships, tell it you want XX% MAX and XX% Heap and it will try and figure
that out and stand you up a drill box.


Tom

--

Director Meteorite.bi - Saiku Analytics Founder
Tel: +44(0)5603641316

(Thanks to the Saiku community we reached our Kickstart

goal, but you can always help by sponsoring the project
)

On 1 June 2016 at 00:50, Tom Barber  wrote:

> Oh, also currently the RAM is clamped down real low
> in /opt/drill/conf/drill-env.sh I will set it back to some sane defaults
> tomorrow as soon as I put the RAM limits into the config options, just ran
> out of time this evening!
>
> --
>
> Director Meteorite.bi - Saiku Analytics Founder
> Tel: +44(0)5603641316
>
> (Thanks to the Saiku community we reached our Kickstart
> 
> goal, but you can always help by sponsoring the project
> )
>
> On 31 May 2016 at 23:50, Tom Barber  wrote:
>
>> Here we are then, for Merlijn and anyone else interested in SQL
>> interfaces to big data/NOSQL stuff.
>>
>> This is less than a days effort, so its patchy at best:
>>
>> https://jujucharms.com/u/spicule/drillbit
>>
>> For those of you who don't know Apache Drill, it will let you run SQL
>> querys over, CSV/JSON data, MongoDB, HBase, Parquet files etc in a number
>> of different locations. Basically its a great way for analysts who use
>> "traditional" SQL tools to leverage data stored within NOSQL solutions.
>>
>> Getting something like this into the CS has been high on my list of
>> priorities for Saiku Analytics as it suddenly offers up loads of new
>> connection prospects(of course I can do this manually in the past, but this
>> is what Juju is for, right?)
>>
>> You need to deploy a ZK node (or 3) and connect it to that and OpenJDK to
>> run it. Currently its relations-lite, the only one in there is a MongoDB
>> test relation that will connect Drill to your MongoDB cluster if you run
>> one, but there will be actions and relations coming shortly for other
>> stuff. Also its missing a fat load of config options, again, coming soon.
>> You can set all of this stuff pretty simply though and there is a web
>> console for queries/connections etc, on http://serverip:8047/
>>
>> Test it, let me know what you think or its its entirely broken. Adding
>> this and a few other NOSQL SQL interfaces is key to people being able to
>> sanely consume all this big data stuff that is great on Juju. Thats not a
>> knock on the Zeppelin guys, but companies will continue to use SQL for a
>> long time to come so we should service that requirement.
>>
>> Tom
>>
>> --
>>
>> Director Meteorite.bi - Saiku Analytics Founder
>> Tel: +44(0)5603641316
>>
>> (Thanks to the Saiku community we reached our Kickstart
>> 
>> goal, but you can always help by sponsoring the project
>> )
>>
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Apache Drill Charm

2016-05-31 Thread Tom Barber 
Oh, also currently the RAM is clamped down real low
in /opt/drill/conf/drill-env.sh I will set it back to some sane defaults
tomorrow as soon as I put the RAM limits into the config options, just ran
out of time this evening!

--

Director Meteorite.bi - Saiku Analytics Founder
Tel: +44(0)5603641316

(Thanks to the Saiku community we reached our Kickstart

goal, but you can always help by sponsoring the project
)

On 31 May 2016 at 23:50, Tom Barber  wrote:

> Here we are then, for Merlijn and anyone else interested in SQL interfaces
> to big data/NOSQL stuff.
>
> This is less than a days effort, so its patchy at best:
>
> https://jujucharms.com/u/spicule/drillbit
>
> For those of you who don't know Apache Drill, it will let you run SQL
> querys over, CSV/JSON data, MongoDB, HBase, Parquet files etc in a number
> of different locations. Basically its a great way for analysts who use
> "traditional" SQL tools to leverage data stored within NOSQL solutions.
>
> Getting something like this into the CS has been high on my list of
> priorities for Saiku Analytics as it suddenly offers up loads of new
> connection prospects(of course I can do this manually in the past, but this
> is what Juju is for, right?)
>
> You need to deploy a ZK node (or 3) and connect it to that and OpenJDK to
> run it. Currently its relations-lite, the only one in there is a MongoDB
> test relation that will connect Drill to your MongoDB cluster if you run
> one, but there will be actions and relations coming shortly for other
> stuff. Also its missing a fat load of config options, again, coming soon.
> You can set all of this stuff pretty simply though and there is a web
> console for queries/connections etc, on http://serverip:8047/
>
> Test it, let me know what you think or its its entirely broken. Adding
> this and a few other NOSQL SQL interfaces is key to people being able to
> sanely consume all this big data stuff that is great on Juju. Thats not a
> knock on the Zeppelin guys, but companies will continue to use SQL for a
> long time to come so we should service that requirement.
>
> Tom
>
> --
>
> Director Meteorite.bi - Saiku Analytics Founder
> Tel: +44(0)5603641316
>
> (Thanks to the Saiku community we reached our Kickstart
> 
> goal, but you can always help by sponsoring the project
> )
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Apache Drill Charm

2016-05-31 Thread Tom Barber 
Here we are then, for Merlijn and anyone else interested in SQL interfaces
to big data/NOSQL stuff.

This is less than a days effort, so its patchy at best:

https://jujucharms.com/u/spicule/drillbit

For those of you who don't know Apache Drill, it will let you run SQL
querys over, CSV/JSON data, MongoDB, HBase, Parquet files etc in a number
of different locations. Basically its a great way for analysts who use
"traditional" SQL tools to leverage data stored within NOSQL solutions.

Getting something like this into the CS has been high on my list of
priorities for Saiku Analytics as it suddenly offers up loads of new
connection prospects(of course I can do this manually in the past, but this
is what Juju is for, right?)

You need to deploy a ZK node (or 3) and connect it to that and OpenJDK to
run it. Currently its relations-lite, the only one in there is a MongoDB
test relation that will connect Drill to your MongoDB cluster if you run
one, but there will be actions and relations coming shortly for other
stuff. Also its missing a fat load of config options, again, coming soon.
You can set all of this stuff pretty simply though and there is a web
console for queries/connections etc, on http://serverip:8047/

Test it, let me know what you think or its its entirely broken. Adding this
and a few other NOSQL SQL interfaces is key to people being able to sanely
consume all this big data stuff that is great on Juju. Thats not a knock on
the Zeppelin guys, but companies will continue to use SQL for a long time
to come so we should service that requirement.

Tom

--

Director Meteorite.bi - Saiku Analytics Founder
Tel: +44(0)5603641316

(Thanks to the Saiku community we reached our Kickstart

goal, but you can always help by sponsoring the project
)
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju