> On June 21, 2017, 8:04 a.m., Sebastian Toader wrote: > > ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/kafka.py > > Lines 106-115 (patched) > > <https://reviews.apache.org/r/60249/diff/1/?file=1754688#file1754688line106> > > > > The ```params.all_racks``` provides only access to rack info that was > > set up for hosts within ambari. However there rack info may come from a > > custom script in that case the information in ```params.all_racks``` won't > > have correct data. > > > > Use the topology script file to figure out the rack information for the > > current cluster. The path to the topology script is found in > > ```core-site/net.topology.script.file.name``` > > > > The topology script file accepts one or more host fqdn as input > > parameter and will return to stdout the rack info for the given hosts. > > Ambud Sharma wrote: > The use case here is to populate information of Node rack in Kafka > configuration. Unlike Hadoop (HDFS/YARN) where the topology script is > applicable and supported natively, Kafka takes rack information differently > i.e. via a property file. While this could be an enhancement later, most > users simply want to propagate rack information for a given Kafka broker in > Ambari and publish that to Kafka. > > core-site does not get installed (there is no /etc/hadoop folder) when > you install Kafka, the topology mapping file does not get published; we > shouldn't make this a dependency or publish a file that creates confusion for > the standard Kafka configuration file set.
The ```topology_script.py``` is generated by BEFORE-START hook (see ```/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py```), this ensures that the file is present on all nodes. Note that core-site.xml that has the path to the ```topology_script.py``` is created only of at least one HDP component that requires it is installed to the host. To ensure that core-site.xml is present for Kafka as well you'll need to add core-site to metainfo.xml/<configuration-dependencies> - Sebastian ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/60249/#review178450 ----------------------------------------------------------- On June 21, 2017, 10:33 p.m., Ambud Sharma wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/60249/ > ----------------------------------------------------------- > > (Updated June 21, 2017, 10:33 p.m.) > > > Review request for Ambari and Alejandro Fernandez. > > > Repository: ambari > > > Description > ------- > > Ambari rack awareness for Kafka. > https://issues.apache.org/jira/browse/AMBARI-21234 > > As an operations person it would be nice to manage Kafka rack awareness via > Ambari. Ambari allows node rack information to be configured and this > information can then be pulled in the Kafka stack and populated in the > server.properties file for Kafka. > Design: > This stack change uses the /clusterHostInfo/all_hosts and > /clusterHostInfo/all_racks paths and materializes them to a variable. Then it > uses linear search to find this node in the list of all hosts and it's > corresponding rack id. This information is then stored in a variable called > rack and which is materialized during the configure method of the broker > scripts. > This stack change relies on the node rack information stored in Ambari > therefore will enable both Ambari UI and Blueprints to be used for setting up > Kafka broker rack information. > > > Diffs > ----- > > > ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/kafka.py > 680dd32802 > > ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/params.py > 5b0be54982 > > > Diff: https://reviews.apache.org/r/60249/diff/2/ > > > Testing > ------- > > Manually deployed Kafka cluster and verified the broker.rack property is > correctly populated. > > > Thanks, > > Ambud Sharma > >
