The YAML I came up with looks like this: # Discovery source config discovery-type : AMBARI discovery-registry : http://c6401.ambari.apache.org:8080
# Provider config reference, the contents of which will be included in, or referenced from, the resulting topology descriptor provider-config-ref : ambari-cluster-policy.xml # The cluster for which the details should be discovered cluster: test # The services to declare in the resulting topology descriptor, whose URLs will be discovered (unless a value is specified) services: - NAMENODE - JOBTRACKER - WEBHDFS - WEBHCAT - OOZIE - WEBHBASE - HIVE - RESOURCEMANAGER - AMBARI : http://c6401.ambari.apache.org:8080 - AMBARIUI : http://c6401.ambari.apache.org:8080 The REST API features I've employed are: To identify all the clusters known to Ambari (for executing a comprehensive discovery of all clusters) http://AMBARI_HOST:PORT/api/v1/clusters To identify all the service components associated with a particular cluster http://AMBARI_HOST:PORT/api/v1/clusters/CLUSTER_NAME/components To determine the service components' respective hosts http://AMBARI_HOST:PORT /api/v1/clusters/CLUSTER_NAME/services?fields=components/host_components/HostRoles For the contents of the active versions of the service component configurations (hive-site, hdfs-site, etc...) http://AMBARI_HOST:PORT /api/v1/clusters/CLUSTER_NAME/configurations/service_config_versions?is_current=true You can check the responses for these using curl (e.g., curl -ivku admin:admin -H 'X-Requested-By:admin' ' http://c6401.ambari.apache.org:8080/api/v1/clusters') >From the responses to these queries, I've managed to construct the necessary service URLs in a topology descriptor. On Thu, Aug 24, 2017 at 5:46 PM, larry mccay <lmc...@apache.org> wrote: > I completely agree, Phil. I also think that is in line with the KIP-1 > item's intent. One thing to keep in mind is how changes to the common > config will need to be dynamically picked up by all referencers. > > I will look into getting you wiki rights for updating the KIP once I am > back at a computer. > > The POC sounds interesting. I'd like to see the YAML and the APIs! > > On Aug 24, 2017 3:45 PM, "Philip Zampino" <pzamp...@gmail.com> wrote: > > I've put together a quick python POC targeting Ambari as the discovery > source, just to prove that we can indeed get the necessary details via > Ambari's REST API. > It generates a proper topology.xml descriptor based on a simple descriptor > (I chose YAML for this POC), which has a reference to what I've called a > policy descriptor (the <gateway/> portion of the topology descriptor). > > I am currently unable to update the KIP, but I can share the REST APIs I've > employed if there is interest. > > As a result, I'm wondering if Knox should support provider configuration > references in topology.xml, rather than having to duplicate it across > descriptors. > So, instead of <gateway><provider>...</gateway> in each topology.xml, have > a single element that points to an external provider config (e.g., > <policy-ref>$GATEWAY_HOME/conf/policy/my-named-provider- > config.xml</policy-ref>). > I've already externalized it for input to the POC, but I'm still copying > the contents into the resulting topology descriptor; I think it would be > better to copy only the reference. > > Thoughts? > > - Phil > > > On Fri, Aug 18, 2017 at 1:55 PM, larry mccay <lmc...@apache.org> wrote: > > > Good to hear, Phil. > > > > Yes, I was looking to go back and add some of the API specifics and > > investigation details for at least Ambari and ZK. > > If others make sense to add such as Consul, etcd, etc that would be good > as > > well and it would help to tease out some of what may be needed for the > > abstraction API and config. > > > > > > On Fri, Aug 18, 2017 at 1:52 PM, Philip Zampino <pzamp...@gmail.com> > > wrote: > > > > > This sounds like a good improvement from a usability perspective. > > > > > > I agree that Knox must continue its support for direct topology (XML) > > > definitions, but for those deployments where there is a service > registry > > > (e.g., Ambari, Zookeeper, Consul, etc...) with knowledge of the > topology, > > > the simplified config coupled with the service discovery could reduce > > > topology (especially ServiceConnectivity) errors. > > > > > > I also agree that consolidating provider configurations into named > sets, > > > which can be referenced from topology configurations, will be a good > > > change, especially if there are commonly grouped providers employed by > > > multiple topologies. > > > > > > Concerning the TODO sections, is your intention is that they contain > the > > > respective API facilities that will provide the information needed to > > > populate a topology definition? > > > > > > - Phil > > > > > > On Fri, Aug 18, 2017 at 12:39 PM, larry mccay <lmc...@apache.org> > wrote: > > > > > > > All - > > > > > > > > I have published an initial stab at a proposal for simplifying > topology > > > > management [1] - as I briefly mentioned on the 0.14.0 planning > thread. > > > > > > > > There are a couple TODO sections that need some investigation for > > Ambari > > > > API and Zookeeper Registry API as discovery plugins. > > > > > > > > Please give the KIP a read and let me know if it makes sense or if > you > > > > would like to take on specific pieces of this work. If you would like > > to > > > > add other options or ideas, etc. > > > > > > > > I think that this will have a huge impact as usability improvements > for > > > > management of the gateway! > > > > > > > > thanks, > > > > > > > > --larry > > > > > > > > 1. > > > > https://cwiki.apache.org/confluence/display/KNOX/KIP-8+ > > > > Service+Discovery+and+Topology+Generation > > > > > > > > > >