On Fri, Jan 24, 2014 at 03:28PM, Mark Grover wrote: > Hey Jay, > Currently we don't patch things in Bigtop. That means when we download and > include, say Hadoop 2.0.2 in Bigtop 0.8, our maven artifacts for hadoop > (say hadoop-common.jar) would have the version 2.2.0 - exactly the same > version as what upstream hadoop released.
In the interest of full disclosure, we do patch certain things - just grep for sed in the packages source tree. Builds would be one things; we did patch source on a couple of occasions in the past to deal with some idiosyncrasies of the official releases. But yes - we frown upon such occasions ;) > So, now 2 options exist for bigpetstore in my opinion: > 1. Pull upstream Hadoop artifacts from maven. You will rely on Apache > Hadoop artifacts instead of bigtop artifacts. However, since Bigtop doesn't > patch, java artifacts should be exactly the same from Bigtop as compared to > Apache Hadoop. > 2. Pull Bigtop artifacts for maven. For this, we will obviously need Bigtop > to a) start updating pom files with its own versioning scheme b) Upload > them to maven central or equivalent. > > As you can see option #2 is a fairly non-trivial overhead for Bigtop but I > would love to hear if you prefer one of the two options and if so why. I think what would be most helpful from the ease of development in the stack - and I have stepped on it more than a few times myself - is to be able to do maven install for the _whole_ project from the top level pom. As of right now, one needs to do a bit of rain dance in order to get all the bits in place. And that's quite annoying apparently. I guess that'd be the next thing to me to look into it. Just occurred to me, that if package driving Makefile is replaced with Gradle that will give a way better consistency of all the parts of the Bigtop environment and stack. Perhaps, it is my severely under-slept brain is talking now ;) Cos > On Thu, Jan 23, 2014 at 2:25 PM, Jay Vyas <[email protected]> wrote: > > > Well, on bigpetstore which I'm writing to target bigtop, I have trouble > > setting up a maven repo with the right dependencies because there are many > > ecosystem projects. > > > > I want to know that my dev environment matches the cluster deployment > > environment. > > > > To do that id like it if there was a maven archetype I could use to build > > a stub project. I'm actually planning to make bigpetstore into this > > archetype app.... So in order to do that I need a systematic way to define > > the pulled in maven dependencies. > > > > Makes sense? > > > > > On Jan 23, 2014, at 5:04 PM, Konstantin Boudnik <[email protected]> wrote: > > > > > > Jay, > > > > > > I don't think we've been doing bigtop publishing like ever. All use > > cases that > > > we had were clearly covered by simple local installation of the > > artifacts. > > > > > > I think it might be a good time to start doing the publishing of the > > official > > > Bigtop artifacts to mavencentral. > > > > > > Could you get a bit deeper into this whole idea of the applications? I am > > > interested to hear more ;)! > > > > > > Thanks, > > > Cos > > > > > >> On Thu, Jan 23, 2014 at 03:46PM, Jay Vyas wrote: > > >> Hi bigtop. > > >> > > >> If I want to build an app against bigtop.... Is there a maven repo I can > > >> point to that has all the version stuff figured out? > > >> > > >> As we all know: it's hard to build hadoop apps and test them in a dev > > >> environment that matches the classpath of cluster environment... I think > > >> bigtop as a distribution might be able to support something like a > > >> maven-archetype for bigtop based app development.? > >
