[
https://issues.apache.org/jira/browse/KNOX-2160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sandor Molnar updated KNOX-2160:
--------------------------------
Description:
To support topology management in Cloudera Manager it'd be beneficial if Knox
was able to process a descriptor that CM can generate natively. As of now, CM's
CSD framework is capable of producing a file of its parameters in the following
[formats|https://github.com/cloudera/cm_ext/wiki/Service-Descriptor-Language-Reference#configWriterFormat]:
* Hadoop XML
* properties
* gflags
As the {{gateway-site.xml}} is a Hadoop XML it's quite obvious that the first
option is the one that fits the most.
One XML type descriptor file may contain one or more Knox descriptors using the
following structure:
* the configuration name would indicate the descriptor (topology) name
* the configuration value would list all properties of a Knox descriptor
** service discovery related information (type, address, cluster,
user/password alias)
** services
*** name
*** url
*** version (optional)
*** parameters (optional)
** applications (optional)
*** name
*** parameters (optional)
A sample descriptor file would look like this:
{code:xml}
<configuration>
<property>
<name>topology1</name>
<value>
discoveryType=ClouderaManager;
discoveryAddress=http://host:123;
discoveryUser=user;
discoveryPasswordAlias=alias;
cluster=Cluster 1;
providerConfigRef=topology1-provider;
app:knoxauth:param1.name=param1.value;
app:KNOX;
HIVE:url=hiveUrl;
HIVE:version=1.0;
HIVE:httpclient.connectionTimeout=5m;
HIVE:httpclient.socketTimeout=100m
</value>
</property>
<property>
<name>topology2</name>
<value>
discoveryType=ClouderaManager;
discoveryAddress=http://host:123;
discoveryUser=user;
discoveryPasswordAlias=alias;
cluster=Cluster 1;
providerConfigRef=topology2-provider;
app:KNOX;
HDFS.url=hdfsURL;
HDFS:httpclient.connectionTimeout=5m;
HDFS:httpclient.socketTimeout=100m
</value>
</property>
</configuration>
{code}
Workflow:
# this kind of descriptor should also be placed in Knox's descriptor directory
# once it's added or modified Knox's existing descriptor monitor should parse
the XML and build an instance of
{{org.apache.knox.gateway.topology.simple.SimpleDescriptor}}
# after the Java object got created it should be saved in the Knox descriptor
directory in JSON format. As a result, the same monitor should parse the
new/modified JSON descriptor and re-deploys it using the already existing
mechanism
was:
It'd be beneficial if there was a way to update service parameters in a
descriptor without using the Admin API/UI. The preferred approach is described
as follows. I'd add 2 new configurations:
- {{refreshable.service.parameters.folder}} - optional configuration to
indicate if Knox should monitor the given folder for service parameter changes.
If it's {{null}} -> no monitoring is enabled.
- {{refreshable.service.parameters.folder.monitor.interval}} - indicates the
time period Knox checks if any parameter changes were made; defaults to 60
seconds. This is relevant only if {{refreshable.service.parameters.folder}} is
{{not null}}.
A new monitor would be implemented (triggered only if monitoring is enabled) as
follows:
# checks {{refreshable-service-parameters.xml}} within the configured
{{refreshable.service.parameters.folder}}. The relevant configuration's name is
{{refreshable.service.parameters}}.
# if there is new content (based on file timestamp) -> parse the file and
build a new parameter list -> trigger topology redeployment with the new
parameter list by rewriting the appropriate descriptor file.
> Introduce a Hadoop XML type descriptor format
> ---------------------------------------------
>
> Key: KNOX-2160
> URL: https://issues.apache.org/jira/browse/KNOX-2160
> Project: Apache Knox
> Issue Type: New Feature
> Components: Server
> Affects Versions: 1.1.0, 1.2.0, 1.3.0
> Reporter: Sandor Molnar
> Assignee: Sandor Molnar
> Priority: Major
> Fix For: 1.4.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> To support topology management in Cloudera Manager it'd be beneficial if Knox
> was able to process a descriptor that CM can generate natively. As of now,
> CM's CSD framework is capable of producing a file of its parameters in the
> following
> [formats|https://github.com/cloudera/cm_ext/wiki/Service-Descriptor-Language-Reference#configWriterFormat]:
> * Hadoop XML
> * properties
> * gflags
> As the {{gateway-site.xml}} is a Hadoop XML it's quite obvious that the first
> option is the one that fits the most.
> One XML type descriptor file may contain one or more Knox descriptors using
> the following structure:
> * the configuration name would indicate the descriptor (topology) name
> * the configuration value would list all properties of a Knox descriptor
> ** service discovery related information (type, address, cluster,
> user/password alias)
> ** services
> *** name
> *** url
> *** version (optional)
> *** parameters (optional)
> ** applications (optional)
> *** name
> *** parameters (optional)
> A sample descriptor file would look like this:
> {code:xml}
> <configuration>
> <property>
> <name>topology1</name>
> <value>
> discoveryType=ClouderaManager;
> discoveryAddress=http://host:123;
> discoveryUser=user;
> discoveryPasswordAlias=alias;
> cluster=Cluster 1;
> providerConfigRef=topology1-provider;
> app:knoxauth:param1.name=param1.value;
> app:KNOX;
> HIVE:url=hiveUrl;
> HIVE:version=1.0;
> HIVE:httpclient.connectionTimeout=5m;
> HIVE:httpclient.socketTimeout=100m
> </value>
> </property>
> <property>
> <name>topology2</name>
> <value>
> discoveryType=ClouderaManager;
> discoveryAddress=http://host:123;
> discoveryUser=user;
> discoveryPasswordAlias=alias;
> cluster=Cluster 1;
> providerConfigRef=topology2-provider;
> app:KNOX;
> HDFS.url=hdfsURL;
> HDFS:httpclient.connectionTimeout=5m;
> HDFS:httpclient.socketTimeout=100m
> </value>
> </property>
> </configuration>
> {code}
>
> Workflow:
> # this kind of descriptor should also be placed in Knox's descriptor
> directory
> # once it's added or modified Knox's existing descriptor monitor should
> parse the XML and build an instance of
> {{org.apache.knox.gateway.topology.simple.SimpleDescriptor}}
> # after the Java object got created it should be saved in the Knox
> descriptor directory in JSON format. As a result, the same monitor should
> parse the new/modified JSON descriptor and re-deploys it using the already
> existing mechanism
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
