> What would be the alternative for a public URL? (for tarballs) Whirr 0.5.0 can publish local artifacts before starting a cluster. You just have to use a local URL for the tarball (e.g. whirr.hadoop.tarball.url=file:///my/local/hadoop-build.tar.gz). For EC2 Whirr will automatically use S3 to create a temporary container.
Keep in mind that when extracted the name of folder should be the same as the name of the archive (without extension). > The best case would be if Apache made the correct version of hadoop available > for each release of HBase. That seems like the responsible thing for them to > do. I believe BigTop will eventually move things in that direction. I also hope this will happen. As you can see we are having a hard time getting this right in Whirr. > Regardless of this, I think it is still interesting to run the configuration > phase for all templates in parallel. I have a patch for this ready, any > interest in this? @Bruno Sounds good to me. Concurrency should not break anything if we are limiting the scope to only one phase of the process. Cheers, -- Andrei On Tue, Jul 12, 2011 at 10:38 AM, Jim R. Wilson <wilson.ji...@gmail.com> wrote: > > > On Tue, Jul 12, 2011 at 12:21 PM, Bruno Dumon <br...@outerthought.org> > wrote: >> >> On Tue, Jul 12, 2011 at 5:51 PM, Jim R. Wilson <wilson.ji...@gmail.com> >> wrote: >>> >>> What version? The de-facto HBase version is 0.89.xxxxxx (as of Whirr >>> 0.5.0). When I change the "whirr.hbase.tarball.url" to point at HBase >>> 0.90.3, I get the Master won't start error of the other email thread. >> >> I didn't specify a version so it was likely the 0.89 one. Maybe I'll give >> it another try tomorrow with 0.90.3. If the problem is the same as with CDH >> (wait with starting the HBase services until the namenode is available), >> then it is easy to apply my fix to the stock hbase configure script as well. > > I'm not sure whether the problems are the same. After failing to get 0.90.3 > working I reached out to the community. Digging through the error logs, it > looked like I had a mismatch between hadoop versions. (The one that HBase > required was newer that the one Whirr uses by default, and the one that > HBase prefers is not publically available anywhere). > >> >> >>> >>> One complication is that HBase 0.90.3 needs an unreleased version of >>> Hadoop. I thought I could use the CDH hadoop 0.20.2 with stock HBase >>> 0.90.3, but that didn't help matters. I thought my problem was just hadoop >>> versioning, but after asking around here, it seems there are other problems. >>> The fact that Whirr requires a public URL to download these things from >>> is also problematic. I could roll my own hadoop at the version level that >>> HBase expects, but then I'd have to host it somewhere. My use-case, >>> however, is that I'm writing a book, and hoping to make it easy for readers >>> to play with a cluster of latest HBase (so I'd prefer not to host a binary >>> since I'd have to make in internet public). >> >> What would be the alternative for a public URL? Upload it from the user's >> workstation? While it would be possible to implement that, it would require >> serious upload capacity from the user, especially when doing a naive upload >> of the .tar.gz to every node that needs it. > > Right, it's not ideal. It /would/ support the case where someone rolls > their own trunk version of HBase/hadoop, but I don't expect this to be a > common use case. > The best case would be if Apache made the correct version of hadoop > available for each release of HBase. That seems like the responsible thing > for them to do. I believe BigTop will eventually move things in that > direction. > In any case, I'm about ready to concede that CDH is my best bet, even though > they're two versions behind. > >> >> >>> >>> For me, CDH (0.90.1) is better than 0.89.x, but I really want 0.90.3. If >>> we can get CDH working, I'd probably be willing to back off of 0.90.3 and >>> change the book to use 0.90.1. I'd really prefer not to use 0.89 though >>> since it's now several releases back. >>> -- Jim >> >> >> -- >> Bruno Dumon >> Outerthought >> http://outerthought.org/ > > -- Jim
