Hi, I'm starting to build a service registry in the now-in-incubation Slider project, with an ultimate goal for a derivative or rework of this to make its way into YARN proper, so that there'll be one we can all use to publish and share information about YARN-deployed and static apps in a Hadoop cluster.
To this end I'm trying to make sure I understand the various uses other projects are making of their ZK-based registries. I see that Twill is using it to publish (hostname, port) pairs, and lets you poll or watch for this, and want to know a bit more 1. What are the common ways you use this? Is it within a distributed app, or between apps? 2. Do you ever use this outside the cluster itself? 3. Do you listen for changes, or just grab it at startup? 4. If you could publish more information, what would you publish? 5. If we had a well-known URL a service registry, one that would support GET lookups and 302 redirects, what would you serve up? A Web UI for people, a REST API for app management, core app functionality... 6. What security model do you rely on? do you assume that whoever publishes a service is being honest, do you rely on ZK permissions, or something else? 7. Finally: did you deliberately choose not to use Apache Curator's discovery mechanism, and if so -why not? Any insight to these or other registry-issues would be really helpful. I don't want to build the "single best ever registry service for humanity" as it would take too long, but I'd like to start with something that we can evolve, which suits YARN app needs and hooks into the user experience and management tools. For now we're using Curator, with some workarounds for hadoop 2.4 jackson versions, a custom payload that publishes the real info, and our AM serving up content -configurations and other artifacts- indexed off the registry entry. -Steve -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
