smolnar82 opened a new pull request #236: KNOX-2160 - Introducing Hadoop XML type descriptor format URL: https://github.com/apache/knox/pull/236 ## What changes were proposed in this pull request? Implemented a new descriptor type - using Hadoop XML configuration - as described in [KNOX-2160 ](https://issues.apache.org/jira/browse/KNOX-2160). Two new Maven modules were created: - `gateway-cm-integration` - Clouder Manager integration-related resources should be added here. The first step was to add the new XML descriptor processing here. - `gateway-topology-simple` - simple topology generation related resources were moved here from `gateway-server` So that the new `gateway-cm-integration` can see them too. ## How was this patch tested? Added and updated JUnit tests then executed them: ``` $ mvn clean -Dshellcheck=true -T1C verify -Prelease,package ... [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 20:06 min (Wall Clock) [INFO] Finished at: 2020-01-10T19:02:35+01:00 [INFO] Final Memory: 385M/2054M [INFO] ------------------------------------------------------------------------ ``` Additionally, the following E2E testing took place locally: 1. deployed and started Knox with my changes 2. created `topology1-rpvider` and `topology2-provider` provider configurations 3. saved the following Hadoop XML configuration in Knox's descriptor directory as `testDescriptor.xml`: ``` <configuration> <property> <name>topology1</name> <value> discoveryType=ClouderaManager; discoveryAddress=http://host:123; discoveryUser=user; discoveryPasswordAlias=alias; cluster=Cluster 1; providerConfigRef=topology1-provider; app:knoxauth:param1.name=param1.value; HIVE:url=http://localhost:456; HIVE:version=1.0; HIVE:httpclient.connectionTimeout=5m; HIVE:httpclient.socketTimeout=100m </value> </property> <property> <name>topology2</name> <value> providerConfigRef=topology2-provider; HDFS:url=http://localhost:456; HDFS:httpclient.connectionTimeout=5m; HDFS:httpclient.socketTimeout=100m </value> </property> </configuration> ``` 4. Confirmed that - descriptor files `topology1.json` and `topology2.json` created within Knox's descriptor folder - topology generation finished successfully (based on log messages and Admin UI) - `topology1` and `topology2` descriptors appeared on Admin UI as well as the corresponding topologies Relevant `gateway.log` messages: ``` 2020-01-10 17:28:45,359 INFO knox.gateway (XmlDescriptorParser.java:parse(53)) - Parsing XML descriptor /Users/smolnar/test/knoxGateway/conf/descriptors/testDescriptor.xml 2020-01-10 17:28:45,361 INFO knox.gateway (XmlDescriptorParser.java:parse(59)) - Found descriptors topology1, topology2 in /Users/smolnar/test/knoxGateway/conf/descriptors/testDescriptor.xml 2020-01-10 17:28:45,367 ERROR discovery.cm (DiscoveryApiClient.java:configure(106)) - No password configured for Cloudera Manager service discovery. 2020-01-10 17:28:45,367 INFO discovery.cm (ClouderaManagerServiceDiscovery.java:discoverCluster(170)) - Performing cluster discovery for "Cluster 1" 2020-01-10 17:28:45,369 ERROR discovery.cm (ClouderaManagerServiceDiscovery.java:getClusterServices(218)) - Failed to access the service configurations for cluster (Cluster 1) discovery: com.cloudera.api.swagger.client.ApiException: java.net.UnknownHostException: host: nodename nor servname provided, or not known 2020-01-10 17:28:45,475 INFO knox.gateway (DefaultTopologyService.java:onFileChange(868)) - Generated topology topology1.xml because the associated descriptor topology1.json changed. 2020-01-10 17:28:45,478 ERROR discovery.ambari (AmbariServiceDiscovery.java:discover(237)) - No address for Ambari service discovery has been configured. 2020-01-10 17:28:45,478 ERROR discovery.ambari (AmbariServiceDiscovery.java:discover(247)) - No cluster for Ambari service discovery has been configured. 2020-01-10 17:28:45,480 ERROR topology.simple (SimpleDescriptorHandler.java:handle(107)) - Unable to complete service discovery for cluster topology2. 2020-01-10 17:28:45,586 INFO knox.gateway (DefaultTopologyService.java:onFileChange(868)) - Generated topology topology2.xml because the associated descriptor topology2.json changed. 2020-01-10 17:28:46,321 INFO knox.gateway (GatewayServer.java:handleCreateDeployment(998)) - Deploying topology topology1 to /Users/smolnar/test/knoxGateway/data/deployments/topology1.topo.16f90495e48 2020-01-10 17:28:46,322 INFO knox.gateway (GatewayServer.java:internalDeactivateTopology(917)) - Deactivating topology topology1 2020-01-10 17:28:46,419 INFO knox.gateway (DefaultGatewayServices.java:initializeContribution(164)) - Credential store found for the cluster: topology1 - no need to create one. 2020-01-10 17:28:46,707 INFO knox.gateway (GatewayServer.java:internalActivateTopology(883)) - Activating topology topology1 2020-01-10 17:28:46,707 INFO knox.gateway (GatewayServer.java:internalActivateArchive(893)) - Activating topology topology1 archive %2Fknoxauth 2020-01-10 17:28:46,762 INFO knox.gateway (GatewayServer.java:handleCreateDeployment(998)) - Deploying topology topology2 to /Users/smolnar/test/knoxGateway/data/deployments/topology2.topo.16f90495e48 2020-01-10 17:28:46,762 INFO knox.gateway (GatewayServer.java:internalDeactivateTopology(917)) - Deactivating topology topology2 2020-01-10 17:28:46,850 INFO knox.gateway (GatewayServer.java:internalActivateTopology(883)) - Activating topology topology2 ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
