smolnar82 opened a new pull request #236: KNOX-2160 - Introducing Hadoop XML 
type descriptor format
URL: https://github.com/apache/knox/pull/236
 
 
   ## What changes were proposed in this pull request?
   
   Implemented a new descriptor type - using Hadoop XML configuration - as 
described in [KNOX-2160 ](https://issues.apache.org/jira/browse/KNOX-2160).
   
   Two new Maven modules were created:
   - `gateway-cm-integration` - Clouder Manager integration-related resources 
should be added here. The first step was to add the new XML descriptor 
processing here.
   - `gateway-topology-simple` - simple topology generation related resources 
were moved here from `gateway-server` So that the new `gateway-cm-integration` 
can see them too.
   
   ## How was this patch tested?
   
   Added and updated JUnit tests then executed them:
   ```
   $ mvn clean -Dshellcheck=true -T1C verify -Prelease,package
   ...
   [INFO] 
------------------------------------------------------------------------
   [INFO] BUILD SUCCESS
   [INFO] 
------------------------------------------------------------------------
   [INFO] Total time: 20:06 min (Wall Clock)
   [INFO] Finished at: 2020-01-10T19:02:35+01:00
   [INFO] Final Memory: 385M/2054M
   [INFO] 
------------------------------------------------------------------------
   
   ```
   
   Additionally, the following E2E testing took place locally:
   1. deployed and started Knox with my changes
   2. created `topology1-rpvider` and `topology2-provider` provider 
configurations
   3. saved the following Hadoop XML configuration in Knox's descriptor 
directory as `testDescriptor.xml`:
   ```
   <configuration>
     <property>
       <name>topology1</name>
       <value>
           discoveryType=ClouderaManager;
           discoveryAddress=http://host:123;
           discoveryUser=user;
           discoveryPasswordAlias=alias;
           cluster=Cluster 1;
           providerConfigRef=topology1-provider;
           app:knoxauth:param1.name=param1.value;
           HIVE:url=http://localhost:456;
           HIVE:version=1.0;
           HIVE:httpclient.connectionTimeout=5m;
           HIVE:httpclient.socketTimeout=100m
       </value>
     </property>
     <property>
       <name>topology2</name>
       <value>
           providerConfigRef=topology2-provider;
           HDFS:url=http://localhost:456;
           HDFS:httpclient.connectionTimeout=5m;
           HDFS:httpclient.socketTimeout=100m
       </value>
     </property>
   </configuration>
   
   ```
   4. Confirmed that
   - descriptor files `topology1.json` and `topology2.json` created within 
Knox's descriptor folder
   - topology generation finished successfully (based on log messages and Admin 
UI)
   - `topology1` and `topology2` descriptors appeared on Admin UI as well as 
the corresponding topologies
   
   Relevant `gateway.log` messages:
   ```
   2020-01-10 17:28:45,359 INFO  knox.gateway 
(XmlDescriptorParser.java:parse(53)) - Parsing XML descriptor 
/Users/smolnar/test/knoxGateway/conf/descriptors/testDescriptor.xml
   2020-01-10 17:28:45,361 INFO  knox.gateway 
(XmlDescriptorParser.java:parse(59)) - Found descriptors topology1, topology2 
in /Users/smolnar/test/knoxGateway/conf/descriptors/testDescriptor.xml
   2020-01-10 17:28:45,367 ERROR discovery.cm 
(DiscoveryApiClient.java:configure(106)) - No password configured for Cloudera 
Manager service discovery.
   2020-01-10 17:28:45,367 INFO  discovery.cm 
(ClouderaManagerServiceDiscovery.java:discoverCluster(170)) - Performing 
cluster discovery for "Cluster 1"
   2020-01-10 17:28:45,369 ERROR discovery.cm 
(ClouderaManagerServiceDiscovery.java:getClusterServices(218)) - Failed to 
access the service configurations for cluster (Cluster 1) discovery: 
com.cloudera.api.swagger.client.ApiException: java.net.UnknownHostException: 
host: nodename nor servname provided, or not known
   2020-01-10 17:28:45,475 INFO  knox.gateway 
(DefaultTopologyService.java:onFileChange(868)) - Generated topology 
topology1.xml because the associated descriptor topology1.json changed.
   2020-01-10 17:28:45,478 ERROR discovery.ambari 
(AmbariServiceDiscovery.java:discover(237)) - No address for Ambari service 
discovery has been configured.
   2020-01-10 17:28:45,478 ERROR discovery.ambari 
(AmbariServiceDiscovery.java:discover(247)) - No cluster for Ambari service 
discovery has been configured.
   2020-01-10 17:28:45,480 ERROR topology.simple 
(SimpleDescriptorHandler.java:handle(107)) - Unable to complete service 
discovery for cluster topology2.
   2020-01-10 17:28:45,586 INFO  knox.gateway 
(DefaultTopologyService.java:onFileChange(868)) - Generated topology 
topology2.xml because the associated descriptor topology2.json changed.
   2020-01-10 17:28:46,321 INFO  knox.gateway 
(GatewayServer.java:handleCreateDeployment(998)) - Deploying topology topology1 
to /Users/smolnar/test/knoxGateway/data/deployments/topology1.topo.16f90495e48
   2020-01-10 17:28:46,322 INFO  knox.gateway 
(GatewayServer.java:internalDeactivateTopology(917)) - Deactivating topology 
topology1
   2020-01-10 17:28:46,419 INFO  knox.gateway 
(DefaultGatewayServices.java:initializeContribution(164)) - Credential store 
found for the cluster: topology1 - no need to create one.
   2020-01-10 17:28:46,707 INFO  knox.gateway 
(GatewayServer.java:internalActivateTopology(883)) - Activating topology 
topology1
   2020-01-10 17:28:46,707 INFO  knox.gateway 
(GatewayServer.java:internalActivateArchive(893)) - Activating topology 
topology1 archive %2Fknoxauth
   2020-01-10 17:28:46,762 INFO  knox.gateway 
(GatewayServer.java:handleCreateDeployment(998)) - Deploying topology topology2 
to /Users/smolnar/test/knoxGateway/data/deployments/topology2.topo.16f90495e48
   2020-01-10 17:28:46,762 INFO  knox.gateway 
(GatewayServer.java:internalDeactivateTopology(917)) - Deactivating topology 
topology2
   2020-01-10 17:28:46,850 INFO  knox.gateway 
(GatewayServer.java:internalActivateTopology(883)) - Activating topology 
topology2
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


With regards,
Apache Git Services

Reply via email to