>> With using puppet and i intend to start a fork of vmbuilder to unify all
>> these projects in making images of virtual machines. Since vmbuilder is
>> basically ubuntu only now with centos support floating about in github.
>> Boxgrinder is kind of gross to get setup because of ruby.
FYI, I'm most of the way through a proof-of-concept for a tool
(coincidentally - in Python) that supports the building of a certain VM in
multiple backends, with Boxgrinder and Kickstarter support right now, and
hopefully VMBuilder support without much difficulty. As I don't know much
about VMBuilder yet, would you perhaps want to collaborate on this? I'll be
posting a patch to BIGTOP-871 in a few days (the one that's there is
Boxgrinder-specific - I'm working on a much more modular approach).
Regarding the choice of language, I think I agree with both arguments so
far - might I suggest Jython? </onlyHalfJoking>. Groovy is already heavily
used within Bigtop, but it also requires more than just the JVM. I
personally prefer Python for these purposes - but I wouldn't strongly
object to anything - as long as it was a well-accepted standard so we don't
end up with a different language for every little project somebody
undertakes.
On Thu, Mar 28, 2013 at 8:04 AM, Philip Herron
<philip.her...@wandisco.com>wrote:
> On 27/03/13 15:37, Roman Shaposhnik wrote:
> > Hi Philip,
> >
> > first of all thanks a bunch for considering to lead with the code --
> that's
> > the best way to make progress. I have just started to look at your code
> > and haven finished reviewing it yet (in fact, I'd rather have somebody
> > on this list who appreciates Python doing that -- Steve?) but let me
> > quickly answer some of the points you've raised.
> >
> > My only metapoint here would be that since we're stuck with Java
> > because 90% of Bigtop components are implemented in it
> > and we've also chosen it as a platform for iTest -- any solution
> > that leverages Java tooling (Gradle, Groovy, etc) would get
> > more of my personal enthusiastic embrace than a solution
> > that introduces yet another platform to care about (Python,
> > Ruby, etc.).
>
> I understand your point here but i think python suits this project much
> better than a jvm based language because of the dependency of the jvm to
> me isn't scripting. Python gets by this because its much simpler and
> installed even in minimal environments of linux by default. And suits
> this work very well.
>
> At the moment we care about:
>
> Make/Bash/java/puppet/ruby/kickstart/ From all i can see at the moment,
> With python we can unify all of bigtop's functionality properly.
>
> With using puppet and i intend to start a fork of vmbuilder to unify all
> these projects in making images of virtual machines. Since vmbuilder is
> basically ubuntu only now with centos support floating about in github.
> Boxgrinder is kind of gross to get setup because of ruby.
>
> >
> > On Wed, Mar 27, 2013 at 6:54 AM, Philip Herron
> > <philip.her...@wandisco.com> wrote:
> >> 1 - Bigtop's logic is handled within gnu/make and bash which is
> >> detrimental for the long term maintainability of bigtop as it may turn
> >> away developers.
> >>
> >> 2 - Make is hard to debug in this way esp for the need of \ for line
> run-on.
> >
> > Could you, please, elaborate on what exact problem you were faced
> > with when you decided that Python is a better option than Make?
> > We all have preferences (for example mine is, honestly, make over
> > python) but of course if the current tooling is inadequate in some
> > respect it needs to be updated.
>
> Yeah I understand you like make i don't mind it but it doesn't have any
> real long term potential. Because i tend to think most people from java
> backgrounds will have next to no background with make. I come from a gcc
> developer/commiter background so i am used to it. But even then this
> isn't a valid use of make in my opinion and i am sure gnu/make
> developers would agree this isn't how make should be used.
>
> Take for example, the next latest and greatest component wants to be a
> *.{zip,tar.xz.tar.bz2,.xz,*.7z} this is a basic example but its still
> valid having the dynamic support for different archives how would we do
> this in python we could figure this out. In Bash were going to have to
> have more hooks and more obscure bash to do something like that.
>
> >
> >> 3 - This new directory structure feels more maintainable already with
> >> bigtop's main function to generate packages for hadoop stack's on
> >> different platforms. Other functions which (don't seem) to be actively
> >> maintained like bigtop-deploy and the test framework with regards to
> >> their documentation.
> >
> > Here's my very strong take on it -- without deployment and test Bigtop
> > might as well pack its stuff and go home. If we're reduced to a
> "framework
> > that builds packages" there's very little left in the project to keep me
> > interested in it.
>
> Deployment and test can still be done and there are 100% important i
> think if i come back with a more complete bigtop with these components
> my fork might be taken more seriously.
>
> >
> > We *all* should be investing pretty heavily in both. In fact, I hoped
> > to make Bigtop 0.6.0 about polishing our tests and deployment
> > code, but Hadoop 2.0.3 story changed that. Hopefully Bigtop 0.7.0
> > will be a release where we really do invest a lot of energy in
> > making both squeaky clean.
> >
>
> I understand this, it will be nice to see them.
>
> >> 5 - The bigtop.mk BOM file was difficult to read and find errors with
> >> eval and ?= as well as having archives set in Makefile and package.mk
> >> doing a lot of tricky bash to get things going.
> >
> > Could you, please elaborate? What exactly made it difficult? It is,
> > essentially, a bunch of key-value pairs (with $substitutions).
>
> Yes its key value you pairs but its horrible to read. Coupled with a
> $eval at the end of each component. I dunno this is a glorified config
> file and i tend to be very pragmatic about this and think this isn't
> what it should look like.
>
> >
> >> One thing to note is this isn't any extra dependency as this is just
> >> stock python code no need for pip install or easy_install of modules.
> >> And almost all distro's of linux/unix come with python so i think its
> fine.
> >
> > That's my biggest worry -- I had *terrible* luck with Python breaking
> > between different Linux distributions.
> >
> > Thanks,
> > Roman.
> >
>
> I understand your worry there, this code is pure python no extra depends
> for now. So that's good and if i get rid of my with usage i can make it
> python >= 2.4 and this is eliminate this worry dead.
>
> Java is by far worse for this problem. example:
> gcj/eclipse/openjdk/oracle...{5,6,7,8}?
>
> And how do you get a standard java version on each linux distro?
>
> I will come back with a more solid implementation with deployment and
> testing.
>
> --Phil
>
