Alan D. Cabrera wrote: > > On Mar 2, 2007, at 5:06 AM, Jim White wrote: > >> Doğacan Güney wrote: >> >>> On 3/2/07, Dan Creswell <[EMAIL PROTECTED]> wrote: >>>> Nigel Daley wrote: >>>> > One more difference... >>>> > ,,, >>>> > OTOH, Hadoop servers and clients are currently pre-configured with >>>> > necessary host/ports. >>>> > >>>> Indeed - I was toying with doing something about removing this >>>> pre-configuration - worthwhile? >>> ... >>> In a data center there are typically 5 chubby instances and >>> their locations are stored in DNS. A client reads their location from >>> DNS and then traverses them to locate the Chubby master. After >>> [snipped the rest] >> >> This manual configuration in Hadoop was a limitation I noticed first >> thing and have suggested Zeroconf (dynamic multicast >> central-server-optional service-discovering DNS) as a solution. >> >> http://www.zeroconf.org/ >> >> I've haven't looked into the details yet, but it is clear to me that >> DNS Service Discovery would work dandy for implementing auto >> configuration for Hadoop. >> >> http://www.dns-sd.org/ >> >> Multicast also has good potential to enhance Hadoop networking >> performance, but that is a separate issue. >> >> And for those concerned that Zeroconf is something exotic, it turns >> out Amazon EC2 is already using it for the parameterized launch. >> That's where the funny "169.254.169.254" address comes from. >> >> http://docs.amazonwebservices.com/AmazonEC2/dg/2006-10-01/TechnicalFAQ.html#d0e14061 >> >> >> A few notes I've collected on Zeroconf: >> >> http://www.ifcx.org/wiki/LocalNetworking.html > > There's another protocol that is similar called Service Location > Protocol, SLP. Thoughts? >
Looks an awful lot like the Jini service location protocol - seems like it got published out of Sun in a similar timeframe too. I'm somewhat biased but I'd be looking at SLP and/or Jini although ZeroConf might be a quicker initial hack. Not sure whether ZeroConf'd be the right solution over the long term or with larger clusters - anybody got experience there? Dan.
