[jira] [Comment Edited] (KNOX-1014) Service Discovery and Topology Generation Framework
[
https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182980#comment-16182980
]
Phil Zampino edited comment on KNOX-1014 at 9/27/17 7:11 PM:
-
{quote}
1. hive protocol scheme
2. deployment fail for missing services or no?
3. .yaml extension in addition to .yml
4. service level params
{quote}
# KNOX-1064 includes changes to address this
# KNOX-1064 includes changes to address this
# KNOX-1063 will address this
# KNOX-1062 will address this
was (Author: pzampino):
{quote}
1. hive protocol scheme
2. deployment fail for missing services or no?
3. .yaml extension in addition to .yml
4. service level params
1. It was unclear to me whether hive proxying through Knox is supported if the
transport mode is NOT http. If it is unsupported, then the resolution of the
second item in your list comes into play...
2. Prior to supporting service discovery, a topology would deploy just fine
with incorrect/invalid service URLs. However, I could be persuaded that failure
to dynamically determine a URL for a service declared in a descriptor should
cause topology generation to fail (since discovery knows it would be generating
an incorrect topology whereas a human does not necessarily know this). I don't
know if we want to generate/deploy something that the descriptor author did not
intend.
3. This is literally a one-line change; Consider it done.
{quote}
# KNOX-1064 includes changes to address this
# KNOX-1064 includes changes to address this
# KNOX-1063 will address this
# KNOX-1062 will address this
> Service Discovery and Topology Generation Framework
> ---
>
> Key: KNOX-1014
> URL: https://issues.apache.org/jira/browse/KNOX-1014
> Project: Apache Knox
> Issue Type: Sub-task
> Components: Server
>Reporter: Phil Zampino
>Assignee: Phil Zampino
> Labels: kip-8
> Fix For: 0.14.0
>
> Attachments: KNOX-1014-002.patch, KNOX-1014.patch
>
>
> Implement the foundation for Service Discovery and Topology Generation.
> * Define simple descriptor format (YAML, JSON, etc...)
> * Local simple descriptor and shared provider configuration discovery
> ** Monitor conf/shared-providers, conf/descriptors similar to the way
> conf/topologies is currently monitored.
> * Ambari service discovery (REST API interactions and model construction)
> ** Configuration
> *** How to plug-in discovery implementations
> *** How to configure authentication (credentials/trust) with the service
> registries
> * Topology assembly from simple descriptor and discovery details
> * Topology deployment
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KNOX-1014) Service Discovery and Topology Generation Framework
[
https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182980#comment-16182980
]
Phil Zampino edited comment on KNOX-1014 at 9/27/17 7:10 PM:
-
{quote}
1. hive protocol scheme
2. deployment fail for missing services or no?
3. .yaml extension in addition to .yml
4. service level params
1. It was unclear to me whether hive proxying through Knox is supported if the
transport mode is NOT http. If it is unsupported, then the resolution of the
second item in your list comes into play...
2. Prior to supporting service discovery, a topology would deploy just fine
with incorrect/invalid service URLs. However, I could be persuaded that failure
to dynamically determine a URL for a service declared in a descriptor should
cause topology generation to fail (since discovery knows it would be generating
an incorrect topology whereas a human does not necessarily know this). I don't
know if we want to generate/deploy something that the descriptor author did not
intend.
3. This is literally a one-line change; Consider it done.
{quote}
# KNOX-1064 includes changes to address this
# KNOX-1064 includes changes to address this
# KNOX-1063 will address this
# KNOX-1062 will address this
was (Author: pzampino):
{quote}
1. hive protocol scheme
2. deployment fail for missing services or no?
3. .yaml extension in addition to .yml
1. It was unclear to me whether hive proxying through Knox is supported if the
transport mode is NOT http. If it is unsupported, then the resolution of the
second item in your list comes into play...
2. Prior to supporting service discovery, a topology would deploy just fine
with incorrect/invalid service URLs. However, I could be persuaded that failure
to dynamically determine a URL for a service declared in a descriptor should
cause topology generation to fail (since discovery knows it would be generating
an incorrect topology whereas a human does not necessarily know this). I don't
know if we want to generate/deploy something that the descriptor author did not
intend.
3. This is literally a one-line change; Consider it done.
{quote}
# KNOX-1064 includes changes to address this
# KNOX-1064 includes changes to address this
# KNOX-1063 will address this
> Service Discovery and Topology Generation Framework
> ---
>
> Key: KNOX-1014
> URL: https://issues.apache.org/jira/browse/KNOX-1014
> Project: Apache Knox
> Issue Type: Sub-task
> Components: Server
>Reporter: Phil Zampino
>Assignee: Phil Zampino
> Labels: kip-8
> Fix For: 0.14.0
>
> Attachments: KNOX-1014-002.patch, KNOX-1014.patch
>
>
> Implement the foundation for Service Discovery and Topology Generation.
> * Define simple descriptor format (YAML, JSON, etc...)
> * Local simple descriptor and shared provider configuration discovery
> ** Monitor conf/shared-providers, conf/descriptors similar to the way
> conf/topologies is currently monitored.
> * Ambari service discovery (REST API interactions and model construction)
> ** Configuration
> *** How to plug-in discovery implementations
> *** How to configure authentication (credentials/trust) with the service
> registries
> * Topology assembly from simple descriptor and discovery details
> * Topology deployment
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KNOX-1014) Service Discovery and Topology Generation Framework
[
https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178456#comment-16178456
]
Phil Zampino edited comment on KNOX-1014 at 9/25/17 2:28 AM:
-
[~lmccay]
{quote}
1. hive protocol scheme
2. deployment fail for missing services or no?
3. .yaml extension in addition to .yml
4. service level params
{quote}
# It was unclear to me whether hive proxying through Knox is supported if the
transport mode is NOT http. If it is unsupported, then the resolution of the
second item in your list comes into play...
# Prior to supporting service discovery, a topology would deploy just fine with
incorrect/invalid service URLs. However, I could be persuaded that failure to
dynamically determine a URL for a service declared in a descriptor should cause
topology generation to fail (since discovery knows it would be generating an
incorrect topology whereas a human does not necessarily know this). I don't
know if we want to generate/deploy something that the descriptor author did not
intend.
# This is literally a one-line change; Consider it done.
# I'll need to learn about these service params in order to add support for
them.
was (Author: pzampino):
[~lmccay]
{quote}
1. hive protocol scheme
2. deployment fail for missing services or no?
3. .yaml extension in addition to .yml
4. service level params
{quote}
# It was unclear to me whether hive proxying through Knox is supported if the
transport mode is NOT http. If it is unsupported, then the resolution of the
second item in your list comes into play...
# Prior to supporting service discovery, a topology would deploy just fine with
incorrect/invalid service URLs. However, I could be persuaded that failure to
dynamically determine a URL for a service declared in a descriptor should cause
topology generation to fail (since discovery knows it would be generating an
incorrect topology whereas a human does not necessarily know this).
# This is literally a one-line change; Consider it done.
# I'll need to learn about these service params in order to add support for
them.
> Service Discovery and Topology Generation Framework
> ---
>
> Key: KNOX-1014
> URL: https://issues.apache.org/jira/browse/KNOX-1014
> Project: Apache Knox
> Issue Type: Sub-task
> Components: Server
>Reporter: Phil Zampino
>Assignee: Phil Zampino
> Labels: kip-8
> Fix For: 0.14.0
>
> Attachments: KNOX-1014-002.patch, KNOX-1014.patch
>
>
> Implement the foundation for Service Discovery and Topology Generation.
> * Define simple descriptor format (YAML, JSON, etc...)
> * Local simple descriptor and shared provider configuration discovery
> ** Monitor conf/shared-providers, conf/descriptors similar to the way
> conf/topologies is currently monitored.
> * Ambari service discovery (REST API interactions and model construction)
> ** Configuration
> *** How to plug-in discovery implementations
> *** How to configure authentication (credentials/trust) with the service
> registries
> * Topology assembly from simple descriptor and discovery details
> * Topology deployment
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KNOX-1014) Service Discovery and Topology Generation Framework
[ https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177896#comment-16177896 ] Larry McCay edited comment on KNOX-1014 at 9/23/17 5:19 PM: Another consideration that we need to account for is parameters for services. There are a number of tuning parameters for various services like hive and the Authentication services APIs like often KnoxSSO and KnoxToken require configuration. I think this will need a follow up JIRA for sure. Unfortunately, this will make the descriptor less simple but it is needed. 1. hive protocol scheme 2. deployment fail for missing services or no? 3. .yaml extension in addition to .yml 4. service level params was (Author: lmccay): Another consideration that we need to account for is parameters for services. There are a number of tuning parameters for various services like hive and the Authentication services APIs like often KnoxSSO and KnoxToken require configuration. I think this will need a follow up JIRA for sure. Unfortunately, this will make the descriptor less simple but it is needed. > Service Discovery and Topology Generation Framework > --- > > Key: KNOX-1014 > URL: https://issues.apache.org/jira/browse/KNOX-1014 > Project: Apache Knox > Issue Type: Sub-task > Components: Server >Reporter: Phil Zampino >Assignee: Phil Zampino > Labels: kip-8 > Fix For: 0.14.0 > > Attachments: KNOX-1014-002.patch, KNOX-1014.patch > > > Implement the foundation for Service Discovery and Topology Generation. > * Define simple descriptor format (YAML, JSON, etc...) > * Local simple descriptor and shared provider configuration discovery > ** Monitor conf/shared-providers, conf/descriptors similar to the way > conf/topologies is currently monitored. > * Ambari service discovery (REST API interactions and model construction) > ** Configuration > *** How to plug-in discovery implementations > *** How to configure authentication (credentials/trust) with the service > registries > * Topology assembly from simple descriptor and discovery details > * Topology deployment -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KNOX-1014) Service Discovery and Topology Generation Framework
[ https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166550#comment-16166550 ] Phil Zampino edited comment on KNOX-1014 at 9/14/17 7:51 PM: - Updated the attached patch with move of Ambari ServiceDiscovery implementation into its own module. was (Author: pzampino): Updated with move of Ambari ServiceDiscovery implementation into its own module. > Service Discovery and Topology Generation Framework > --- > > Key: KNOX-1014 > URL: https://issues.apache.org/jira/browse/KNOX-1014 > Project: Apache Knox > Issue Type: Sub-task > Components: Server >Reporter: Phil Zampino >Assignee: Phil Zampino > Labels: kip-8 > Fix For: 0.14.0 > > Attachments: KNOX-1014.patch > > > Implement the foundation for Service Discovery and Topology Generation. > * Define simple descriptor format (YAML, JSON, etc...) > * Local simple descriptor and shared provider configuration discovery > ** Monitor conf/shared-providers, conf/descriptors similar to the way > conf/topologies is currently monitored. > * Ambari service discovery (REST API interactions and model construction) > ** Configuration > *** How to plug-in discovery implementations > *** How to configure authentication (credentials/trust) with the service > registries > * Topology assembly from simple descriptor and discovery details > * Topology deployment -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KNOX-1014) Service Discovery and Topology Generation Framework
[
https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163460#comment-16163460
]
Phil Zampino edited comment on KNOX-1014 at 9/14/17 5:00 PM:
-
KNOX-1014.patch (attached) provides the foundation for service discovery and
topology generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance
_bin/knoxcli.sh create-alias AMBARI_USERNAME --value AMBARI_PASSWORD
# Create a simple descriptor (JSON):
_Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org), AMBARI_USERNAME,
YOUR_CLUSTER_NAME with the appropriate values_
{noformat}
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_USERNAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "urls":"[http://YOUR_AMBARI_HOST:8080"]},
{"name":"AMBARIUI", "urls":["http://YOUR_AMBARI_HOST:8080"]}
]
}
{noformat}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# Review the contents of conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that
the service URLs have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will update the full topology so the changes
are reflected.
* If you modify a shared provider configuration, it will update any referencing
descriptors, which will in turn update the associated full topology files.
* If you delete a simple descriptor, the associated topology will be
removed/undeployed.
* If you delete a topology file, the associated simple descriptor will also be
deleted.
* -If you delete a shared provider configuration, any referencing descriptors
(and associated topology files) will also be deleted, and the topologies will
be undeployed.-
was (Author: pzampino):
KNOX-1014.patch (attached) provides the foundation for service discovery and
topology generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance
_bin/knoxcli.sh create-alias AMBARI_USERNAME --value AMBARI_PASSWORD
# Create a simple descriptor (JSON):
_Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org), AMBARI_USERNAME,
YOUR_CLUSTER_NAME with the appropriate values_
{noformat}
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_USERNAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "urls":"[http://YOUR_AMBARI_HOST:8080"]},
{"name":"AMBARIUI", "urls":["http://YOUR_AMBARI_HOST:8080"]}
]
}
{noformat}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# Review the contents of conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that
the service URLs have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will update the full topology so the changes
are reflected.
* If
[jira] [Comment Edited] (KNOX-1014) Service Discovery and Topology Generation Framework
[
https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163460#comment-16163460
]
Phil Zampino edited comment on KNOX-1014 at 9/14/17 4:59 PM:
-
KNOX-1014.patch (attached) provides the foundation for service discovery and
topology generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance
_bin/knoxcli.sh create-alias AMBARI_USERNAME --value AMBARI_PASSWORD
# Create a simple descriptor (JSON):
_Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org), AMBARI_USERNAME,
YOUR_CLUSTER_NAME with the appropriate values_
{noformat}
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_USERNAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "urls":"[http://YOUR_AMBARI_HOST:8080"]},
{"name":"AMBARIUI", "urls":["http://YOUR_AMBARI_HOST:8080"]}
]
}
{noformat}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# Review the contents of conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that
the service URLs have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will update the full topology so the changes
are reflected.
* If you modify a shared provider configuration, it will update any referencing
descriptors, which will in turn update the associated full topology files.
* If you delete a simple descriptor, the associated topology will be
removed/undeployed.
* If you delete a topology file, the associated simple descriptor will also be
deleted.
-* If you delete a shared provider configuration, any referencing descriptors
(and associated topology files) will also be deleted, and the topologies will
be undeployed.-
was (Author: pzampino):
KNOX-1014.patch (attached) provides the foundation for service discovery and
topology generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance
_bin/knoxcli.sh create-alias AMBARI_USERNAME --value AMBARI_PASSWORD
# Create a simple descriptor (JSON):
_Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org), AMBARI_USERNAME,
YOUR_CLUSTER_NAME with the appropriate values_
{noformat}
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_CLUSTER_ADMIN_NAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "url":"http://YOUR_AMBARI_HOST:8080"},
{"name":"AMBARIUI", "url":"http://YOUR_AMBARI_HOST:8080"}
]
}
{noformat}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# Review the contents of conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that
the service URLs have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will update the full topology so the changes
are reflected.
*
[jira] [Comment Edited] (KNOX-1014) Service Discovery and Topology Generation Framework
[
https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163460#comment-16163460
]
Phil Zampino edited comment on KNOX-1014 at 9/13/17 2:52 PM:
-
KNOX-1014.patch (attached) provides the foundation for service discovery and
topology generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance
_bin/knoxcli.sh create-alias AMBARI_USERNAME --value AMBARI_PASSWORD
# Create a simple descriptor (JSON):
_Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org), AMBARI_USERNAME,
YOUR_CLUSTER_NAME with the appropriate values_
{noformat}
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_CLUSTER_ADMIN_NAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "url":"http://YOUR_AMBARI_HOST:8080"},
{"name":"AMBARIUI", "url":"http://YOUR_AMBARI_HOST:8080"}
]
}
{noformat}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# Review the contents of conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that
the service URLs have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will update the full topology so the changes
are reflected.
* If you modify a shared provider configuration, it will update any referencing
descriptors, which will in turn update the associated full topology files.
* If you delete a simple descriptor, the associated topology will be
removed/undeployed.
* If you delete a topology file, the associated simple descriptor will also be
deleted.
* If you delete a shared provider configuration, any referencing descriptors
(and associated topology files) will also be deleted, and the topologies will
be undeployed.
was (Author: pzampino):
KNOX-1014.patch (attached) provides the foundation for service discovery and
topology generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance
_bin/knoxcli.sh create-alias AMBARI_CLUSTER_ADMIN_NAME --value
AMBARI_CLUSTER_ADMIN_PASSWORD_
# Create a simple descriptor (JSON):
_Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org),
AMBARI_CLUSTER_ADMIN_NAME, YOUR_CLUSTER_NAME with the appropriate values_
{noformat}
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_CLUSTER_ADMIN_NAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "url":"http://YOUR_AMBARI_HOST:8080"},
{"name":"AMBARIUI", "url":"http://YOUR_AMBARI_HOST:8080"}
]
}
{noformat}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# Review the contents of conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that
the service URLs have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will update the full
[jira] [Comment Edited] (KNOX-1014) Service Discovery and Topology Generation Framework
[
https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163460#comment-16163460
]
Phil Zampino edited comment on KNOX-1014 at 9/12/17 6:46 PM:
-
KNOX-1014.patch (attached) provides the foundation for service discovery and
topology generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance
_bin/knoxcli.sh create-alias AMBARI_CLUSTER_ADMIN_NAME --value
AMBARI_CLUSTER_ADMIN_PASSWORD_
# Create a simple descriptor (JSON):
_Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org),
AMBARI_CLUSTER_ADMIN_NAME, YOUR_CLUSTER_NAME with the appropriate values_
{noformat}
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_CLUSTER_ADMIN_NAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "url":"http://YOUR_AMBARI_HOST:8080"},
{"name":"AMBARIUI", "url":"http://YOUR_AMBARI_HOST:8080"}
]
}
{noformat}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# cat conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that the service URLs
have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will update the full topology so the changes
are reflected.
* If you modify a shared provider configuration, it will update any referencing
descriptors, which will in turn update the associated full topology files.
* If you delete a simple descriptor, the associated topology will be
removed/undeployed.
* If you delete a topology file, the associated simple descriptor will also be
deleted.
* If you delete a shared provider configuration, any referencing descriptors
(and associated topology files) will also be deleted, and the topologies will
be undeployed.
was (Author: pzampino):
KNOX-1014.patch (attached) provides the foundation for service discovery and
topology generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance (bin/knoxcli.sh
create-alias AMBARI_CLUSTER_ADMIN_NAME --value AMBARI_CLUSTER_ADMIN_PASSWORD)
# Create a simple descriptor (JSON):
Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org),
AMBARI_CLUSTER_ADMIN_NAME, YOUR_CLUSTER_NAME with the appropriate values
{noformat}
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_CLUSTER_ADMIN_NAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "url":"http://YOUR_AMBARI_HOST:8080"},
{"name":"AMBARIUI", "url":"http://YOUR_AMBARI_HOST:8080"}
]
}
{noformat}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# cat conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that the service URLs
have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will update the full topology
[jira] [Comment Edited] (KNOX-1014) Service Discovery and Topology Generation Framework
[
https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163460#comment-16163460
]
Phil Zampino edited comment on KNOX-1014 at 9/12/17 6:44 PM:
-
KNOX-1014.patch (attached) provides the foundation for service discovery and
topology generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance (bin/knoxcli.sh
create-alias AMBARI_CLUSTER_ADMIN_NAME --value AMBARI_CLUSTER_ADMIN_PASSWORD)
# Create a simple descriptor (JSON):
Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org),
AMBARI_CLUSTER_ADMIN_NAME, YOUR_CLUSTER_NAME with the appropriate values
{noformat}
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_CLUSTER_ADMIN_NAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "url":"http://YOUR_AMBARI_HOST:8080"},
{"name":"AMBARIUI", "url":"http://YOUR_AMBARI_HOST:8080"}
]
}
{noformat}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# cat conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that the service URLs
have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will update the full topology so the changes
are reflected.
* If you modify a shared provider configuration, it will update any referencing
descriptors, which will in turn update the associated full topology files.
* If you delete a simple descriptor, the associated topology will be
removed/undeployed.
* If you delete a topology file, the associated simple descriptor will also be
deleted.
* If you delete a shared provider configuration, any referencing descriptors
(and associated topology files) will also be deleted, and the topologies will
be undeployed.
was (Author: pzampino):
KNOX-1014.patch (attached) provides the foundation for service discovery and
topology generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance (bin/knoxcli.sh
create-alias AMBARI_CLUSTER_ADMIN_NAME --value AMBARI_CLUSTER_ADMIN_PASSWORD)
# Create a simple descriptor (JSON):
Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org),
AMBARI_CLUSTER_ADMIN_NAME, YOUR_CLUSTER_NAME with the appropriate values
{noformat}
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_CLUSTER_ADMIN_NAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "url":"http://YOUR_AMBARI_HOST:8080"},
{"name":"AMBARIUI", "url":"http://YOUR_AMBARI_HOST:8080"}
]
}
{noformat}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# cat conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that the service URLs
have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will update the full
[jira] [Comment Edited] (KNOX-1014) Service Discovery and Topology Generation Framework
[
https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163460#comment-16163460
]
Phil Zampino edited comment on KNOX-1014 at 9/12/17 6:44 PM:
-
KNOX-1014.patch (attached) provides the foundation for service discovery and
topology generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance (bin/knoxcli.sh
create-alias AMBARI_CLUSTER_ADMIN_NAME --value AMBARI_CLUSTER_ADMIN_PASSWORD)
# Create a simple descriptor (JSON):
Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org),
AMBARI_CLUSTER_ADMIN_NAME, YOUR_CLUSTER_NAME with the appropriate values
{noformat}
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_CLUSTER_ADMIN_NAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "url":"http://YOUR_AMBARI_HOST:8080"},
{"name":"AMBARIUI", "url":"http://YOUR_AMBARI_HOST:8080"}
]
}
{noformat}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# cat conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that the service URLs
have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will update the full topology so the changes
are reflected.
* If you modify a shared provider configuration, it will update any referencing
descriptors, which will in turn update the associated full topology files.
* If you delete a simple descriptor, the associated topology will be
removed/undeployed.
* If you delete a topology file, the associated simple descriptor will also be
deleted.
* If you delete a shared provider configuration, any referencing descriptors
(and associated topology files) will also be deleted, and the topologies will
be undeployed.
was (Author: pzampino):
KNOX-1014.patch (attached) provides the foundation for service discovery and
topology generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance (bin/knoxcli.sh
create-alias AMBARI_CLUSTER_ADMIN_NAME --value AMBARI_CLUSTER_ADMIN_PASSWORD)
# Create a simple descriptor (JSON):
Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org),
AMBARI_CLUSTER_ADMIN_NAME, YOUR_CLUSTER_NAME with the appropriate values
{noformat}
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_CLUSTER_ADMIN_NAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "url":"http://c6401.ambari.apache.org:8080"},
{"name":"AMBARIUI", "url":"http://c6401.ambari.apache.org:8080"}
]
}
{noformat}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# cat conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that the service URLs
have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will update
[jira] [Comment Edited] (KNOX-1014) Service Discovery and Topology Generation Framework
[
https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163460#comment-16163460
]
Phil Zampino edited comment on KNOX-1014 at 9/12/17 6:44 PM:
-
KNOX-1014.patch (attached) provides the foundation for service discovery and
topology generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance (bin/knoxcli.sh
create-alias AMBARI_CLUSTER_ADMIN_NAME --value AMBARI_CLUSTER_ADMIN_PASSWORD)
# Create a simple descriptor (JSON):
Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org),
AMBARI_CLUSTER_ADMIN_NAME, YOUR_CLUSTER_NAME with the appropriate values
{noformat}
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_CLUSTER_ADMIN_NAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "url":"http://YOUR_AMBARI_HOST:8080"},
{"name":"AMBARIUI", "url":"http://YOUR_AMBARI_HOST:8080"}
]
}
{noformat}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# cat conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that the service URLs
have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will update the full topology so the changes
are reflected.
* If you modify a shared provider configuration, it will update any referencing
descriptors, which will in turn update the associated full topology files.
* If you delete a simple descriptor, the associated topology will be
removed/undeployed.
* If you delete a topology file, the associated simple descriptor will also be
deleted.
* If you delete a shared provider configuration, any referencing descriptors
(and associated topology files) will also be deleted, and the topologies will
be undeployed.
was (Author: pzampino):
KNOX-1014.patch (attached) provides the foundation for service discovery and
topology generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance (bin/knoxcli.sh
create-alias AMBARI_CLUSTER_ADMIN_NAME --value AMBARI_CLUSTER_ADMIN_PASSWORD)
# Create a simple descriptor (JSON):
Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org),
AMBARI_CLUSTER_ADMIN_NAME, YOUR_CLUSTER_NAME with the appropriate values
{noformat}
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_CLUSTER_ADMIN_NAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "url":"http://YOUR_AMBARI_HOST:8080"},
{"name":"AMBARIUI", "url":"http://YOUR_AMBARI_HOST:8080"}
]
}
{noformat}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# cat conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that the service URLs
have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will update the full topology
[jira] [Comment Edited] (KNOX-1014) Service Discovery and Topology Generation Framework
[
https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163460#comment-16163460
]
Phil Zampino edited comment on KNOX-1014 at 9/12/17 6:43 PM:
-
KNOX-1014.patch (attached) provides the foundation for service discovery and
topology generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance (bin/knoxcli.sh
create-alias AMBARI_CLUSTER_ADMIN_NAME --value AMBARI_CLUSTER_ADMIN_PASSWORD)
# Create a simple descriptor (JSON):
Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org),
AMBARI_CLUSTER_ADMIN_NAME, YOUR_CLUSTER_NAME with the appropriate values
{noformat}
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_CLUSTER_ADMIN_NAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "url":"http://c6401.ambari.apache.org:8080"},
{"name":"AMBARIUI", "url":"http://c6401.ambari.apache.org:8080"}
]
}
{noformat}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# cat conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that the service URLs
have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will update the full topology so the changes
are reflected.
* If you modify a shared provider configuration, it will update any referencing
descriptors, which will in turn update the associated full topology files.
* If you delete a simple descriptor, the associated topology will be
removed/undeployed.
* If you delete a topology file, the associated simple descriptor will also be
deleted.
* If you delete a shared provider configuration, any referencing descriptors
(and associated topology files) will also be deleted, and the topologies will
be undeployed.
was (Author: pzampino):
KNOX-1014.patch provides the foundation for service discovery and topology
generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance (bin/knoxcli.sh
create-alias AMBARI_CLUSTER_ADMIN_NAME --value AMBARI_CLUSTER_ADMIN_PASSWORD)
# Create a simple descriptor (JSON):
Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org),
AMBARI_CLUSTER_ADMIN_NAME, YOUR_CLUSTER_NAME with the appropriate values
{noformat}
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_CLUSTER_ADMIN_NAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "url":"http://c6401.ambari.apache.org:8080"},
{"name":"AMBARIUI", "url":"http://c6401.ambari.apache.org:8080"}
]
}
{noformat}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# cat conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that the service URLs
have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will
[jira] [Comment Edited] (KNOX-1014) Service Discovery and Topology Generation Framework
[
https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163460#comment-16163460
]
Phil Zampino edited comment on KNOX-1014 at 9/12/17 6:42 PM:
-
KNOX-1014.patch provides the foundation for service discovery and topology
generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance (bin/knoxcli.sh
create-alias AMBARI_CLUSTER_ADMIN_NAME --value AMBARI_CLUSTER_ADMIN_PASSWORD)
# Create a simple descriptor (JSON):
Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org),
AMBARI_CLUSTER_ADMIN_NAME, YOUR_CLUSTER_NAME with the appropriate values
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_CLUSTER_ADMIN_NAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "url":"http://c6401.ambari.apache.org:8080"},
{"name":"AMBARIUI", "url":"http://c6401.ambari.apache.org:8080"}
]
}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# cat conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that the service URLs
have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will update the full topology so the changes
are reflected.
* If you modify a shared provider configuration, it will update any referencing
descriptors, which will in turn update the associated full topology files.
* If you delete a simple descriptor, the associated topology will be
removed/undeployed.
* If you delete a topology file, the associated simple descriptor will also be
deleted.
* If you delete a shared provider configuration, any referencing descriptors
(and associated topology files) will also be deleted, and the topologies will
be undeployed.
was (Author: pzampino):
KNOX-1014.patch provides the foundation for service discovery and topology
generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance (bin/knoxcli.sh
create-alias AMBARI_CLUSTER_ADMIN_NAME --value AMBARI_CLUSTER_ADMIN_PASSWORD)
# Create a simple descriptor (JSON):
Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org),
AMBARI_CLUSTER_ADMIN_NAME, YOUR_CLUSTER_NAME with the appropriate values
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_CLUSTER_ADMIN_NAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "url":"http://c6401.ambari.apache.org:8080"},
{"name":"AMBARIUI", "url":"http://c6401.ambari.apache.org:8080"}
]
}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# cat conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that the service URLs
have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will update the full topology so the changes
are reflected.
* If you
[jira] [Comment Edited] (KNOX-1014) Service Discovery and Topology Generation Framework
[
https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163460#comment-16163460
]
Phil Zampino edited comment on KNOX-1014 at 9/12/17 6:42 PM:
-
KNOX-1014.patch provides the foundation for service discovery and topology
generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance (bin/knoxcli.sh
create-alias AMBARI_CLUSTER_ADMIN_NAME --value AMBARI_CLUSTER_ADMIN_PASSWORD)
# Create a simple descriptor (JSON):
Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org),
AMBARI_CLUSTER_ADMIN_NAME, YOUR_CLUSTER_NAME with the appropriate values
{noformat}
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_CLUSTER_ADMIN_NAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "url":"http://c6401.ambari.apache.org:8080"},
{"name":"AMBARIUI", "url":"http://c6401.ambari.apache.org:8080"}
]
}
{noformat}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# cat conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that the service URLs
have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will update the full topology so the changes
are reflected.
* If you modify a shared provider configuration, it will update any referencing
descriptors, which will in turn update the associated full topology files.
* If you delete a simple descriptor, the associated topology will be
removed/undeployed.
* If you delete a topology file, the associated simple descriptor will also be
deleted.
* If you delete a shared provider configuration, any referencing descriptors
(and associated topology files) will also be deleted, and the topologies will
be undeployed.
was (Author: pzampino):
KNOX-1014.patch provides the foundation for service discovery and topology
generation.
After applying the patch, if you have access to an Ambari cluster, you can try
the following:
# Locate or
[create|https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide]
an Ambari cluster
# Extract the element and its contents from
conf/topologies/sandbox.xml, and save it in
conf/shared-providers/sandbox-providers.xml
# Provision the username/password for your Ambari instance (bin/knoxcli.sh
create-alias AMBARI_CLUSTER_ADMIN_NAME --value AMBARI_CLUSTER_ADMIN_PASSWORD)
# Create a simple descriptor (JSON):
Replace YOUR_AMBARI_HOST(e.g., c6401.ambari.apache.org),
AMBARI_CLUSTER_ADMIN_NAME, YOUR_CLUSTER_NAME with the appropriate values
{
"discovery-type":"AMBARI",
"discovery-address":"http://YOUR_AMBARI_HOST:8080;,
"discovery-user":"AMBARI_CLUSTER_ADMIN_NAME",
"provider-config-ref":"sandbox-providers.xml",
"cluster":"YOUR_CLUSTER_NAME",
"services":[
{"name":"NAMENODE"},
{"name":"JOBTRACKER"},
{"name":"WEBHDFS"},
{"name":"WEBHCAT"},
{"name":"OOZIE"},
{"name":"WEBHBASE"},
{"name":"HIVE"},
{"name":"RESOURCEMANAGER"},
{"name":"AMBARI", "url":"http://c6401.ambari.apache.org:8080"},
{"name":"AMBARIUI", "url":"http://c6401.ambari.apache.org:8080"}
]
}
# Place this simple descriptor in conf/descriptors/YOUR_DESCRIPTOR_NAME.json
# Allow the TopologyService to notice the simple descriptor, generate and
deploy the full topology.
# Check conf/topologies, and notice YOUR_DESCRIPTOR_NAME.xml there
# cat conf/topologies/YOUR_DESCRIPTOR_NAME.xml to see that the service URLs
have been populated from the Ambari cluster details
# Verify the WEBHDFS service URL was correctly discovered
curl -ivku guest:guest-password
https://localhost:8443/gateway/YOUR_DESCRIPTOR_NAME/webhdfs/v1/tmp?op=LISTSTATUS
Now, you can try adding/modifying/removing combinations of provider
configurations and descriptions to see the results.
* If you modify a descriptor, it will update the full topology so the changes
