The YAML I came up with looks like this:
# Discovery source config
discovery-type : AMBARI
discovery-registry : http://c6401.ambari.apache.org:8080
# Provider config reference, the contents of which will be included in, or
referenced from, the resulting topology descriptor
provider-config-ref : ambari-cluster-policy.xml
# The cluster for which the details should be discovered
cluster: test
# The services to declare in the resulting topology descriptor, whose URLs
will be discovered (unless a value is specified)
services:
- NAMENODE
- JOBTRACKER
- WEBHDFS
- WEBHCAT
- OOZIE
- WEBHBASE
- HIVE
- RESOURCEMANAGER
- AMBARI : http://c6401.ambari.apache.org:8080
- AMBARIUI : http://c6401.ambari.apache.org:8080
The REST API features I've employed are:
To identify all the clusters known to Ambari (for executing a comprehensive
discovery of all clusters)
http://AMBARI_HOST:PORT/api/v1/clusters
To identify all the service components associated with a particular cluster
http://AMBARI_HOST:PORT/api/v1/clusters/CLUSTER_NAME/components
To determine the service components' respective hosts
http://AMBARI_HOST:PORT
/api/v1/clusters/CLUSTER_NAME/services?fields=components/host_components/HostRoles
For the contents of the active versions of the service component
configurations (hive-site, hdfs-site, etc...)
http://AMBARI_HOST:PORT
/api/v1/clusters/CLUSTER_NAME/configurations/service_config_versions?is_current=true
You can check the responses for these using curl (e.g., curl -ivku
admin:admin -H 'X-Requested-By:admin' '
http://c6401.ambari.apache.org:8080/api/v1/clusters')
>From the responses to these queries, I've managed to construct the
necessary service URLs in a topology descriptor.
On Thu, Aug 24, 2017 at 5:46 PM, larry mccay <[email protected]> wrote:
> I completely agree, Phil. I also think that is in line with the KIP-1
> item's intent. One thing to keep in mind is how changes to the common
> config will need to be dynamically picked up by all referencers.
>
> I will look into getting you wiki rights for updating the KIP once I am
> back at a computer.
>
> The POC sounds interesting. I'd like to see the YAML and the APIs!
>
> On Aug 24, 2017 3:45 PM, "Philip Zampino" <[email protected]> wrote:
>
> I've put together a quick python POC targeting Ambari as the discovery
> source, just to prove that we can indeed get the necessary details via
> Ambari's REST API.
> It generates a proper topology.xml descriptor based on a simple descriptor
> (I chose YAML for this POC), which has a reference to what I've called a
> policy descriptor (the <gateway/> portion of the topology descriptor).
>
> I am currently unable to update the KIP, but I can share the REST APIs I've
> employed if there is interest.
>
> As a result, I'm wondering if Knox should support provider configuration
> references in topology.xml, rather than having to duplicate it across
> descriptors.
> So, instead of <gateway><provider>...</gateway> in each topology.xml, have
> a single element that points to an external provider config (e.g.,
> <policy-ref>$GATEWAY_HOME/conf/policy/my-named-provider-
> config.xml</policy-ref>).
> I've already externalized it for input to the POC, but I'm still copying
> the contents into the resulting topology descriptor; I think it would be
> better to copy only the reference.
>
> Thoughts?
>
> - Phil
>
>
> On Fri, Aug 18, 2017 at 1:55 PM, larry mccay <[email protected]> wrote:
>
> > Good to hear, Phil.
> >
> > Yes, I was looking to go back and add some of the API specifics and
> > investigation details for at least Ambari and ZK.
> > If others make sense to add such as Consul, etcd, etc that would be good
> as
> > well and it would help to tease out some of what may be needed for the
> > abstraction API and config.
> >
> >
> > On Fri, Aug 18, 2017 at 1:52 PM, Philip Zampino <[email protected]>
> > wrote:
> >
> > > This sounds like a good improvement from a usability perspective.
> > >
> > > I agree that Knox must continue its support for direct topology (XML)
> > > definitions, but for those deployments where there is a service
> registry
> > > (e.g., Ambari, Zookeeper, Consul, etc...) with knowledge of the
> topology,
> > > the simplified config coupled with the service discovery could reduce
> > > topology (especially ServiceConnectivity) errors.
> > >
> > > I also agree that consolidating provider configurations into named
> sets,
> > > which can be referenced from topology configurations, will be a good
> > > change, especially if there are commonly grouped providers employed by
> > > multiple topologies.
> > >
> > > Concerning the TODO sections, is your intention is that they contain
> the
> > > respective API facilities that will provide the information needed to
> > > populate a topology definition?
> > >
> > > - Phil
> > >
> > > On Fri, Aug 18, 2017 at 12:39 PM, larry mccay <[email protected]>
> wrote:
> > >
> > > > All -
> > > >
> > > > I have published an initial stab at a proposal for simplifying
> topology
> > > > management [1] - as I briefly mentioned on the 0.14.0 planning
> thread.
> > > >
> > > > There are a couple TODO sections that need some investigation for
> > Ambari
> > > > API and Zookeeper Registry API as discovery plugins.
> > > >
> > > > Please give the KIP a read and let me know if it makes sense or if
> you
> > > > would like to take on specific pieces of this work. If you would like
> > to
> > > > add other options or ideas, etc.
> > > >
> > > > I think that this will have a huge impact as usability improvements
> for
> > > > management of the gateway!
> > > >
> > > > thanks,
> > > >
> > > > --larry
> > > >
> > > > 1.
> > > > https://cwiki.apache.org/confluence/display/KNOX/KIP-8+
> > > > Service+Discovery+and+Topology+Generation
> > > >
> > >
> >
>
