[jira] [Commented] (KNOX-1014) Service Discovery and Topology Generation Framework
[
https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182980#comment-16182980
]
Phil Zampino commented on KNOX-1014:
{quote}
1. hive protocol scheme
2. deployment fail for missing services or no?
3. .yaml extension in addition to .yml
1. It was unclear to me whether hive proxying through Knox is supported if the
transport mode is NOT http. If it is unsupported, then the resolution of the
second item in your list comes into play...
2. Prior to supporting service discovery, a topology would deploy just fine
with incorrect/invalid service URLs. However, I could be persuaded that failure
to dynamically determine a URL for a service declared in a descriptor should
cause topology generation to fail (since discovery knows it would be generating
an incorrect topology whereas a human does not necessarily know this). I don't
know if we want to generate/deploy something that the descriptor author did not
intend.
3. This is literally a one-line change; Consider it done.
{quote}
# KNOX-1064 includes changes to address this
# KNOX-1064 includes changes to address this
# KNOX-1063 will address this
> Service Discovery and Topology Generation Framework
> ---
>
> Key: KNOX-1014
> URL: https://issues.apache.org/jira/browse/KNOX-1014
> Project: Apache Knox
> Issue Type: Sub-task
> Components: Server
>Reporter: Phil Zampino
>Assignee: Phil Zampino
> Labels: kip-8
> Fix For: 0.14.0
>
> Attachments: KNOX-1014-002.patch, KNOX-1014.patch
>
>
> Implement the foundation for Service Discovery and Topology Generation.
> * Define simple descriptor format (YAML, JSON, etc...)
> * Local simple descriptor and shared provider configuration discovery
> ** Monitor conf/shared-providers, conf/descriptors similar to the way
> conf/topologies is currently monitored.
> * Ambari service discovery (REST API interactions and model construction)
> ** Configuration
> *** How to plug-in discovery implementations
> *** How to configure authentication (credentials/trust) with the service
> registries
> * Topology assembly from simple descriptor and discovery details
> * Topology deployment
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KNOX-1014) Service Discovery and Topology Generation Framework
[
https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179797#comment-16179797
]
Phil Zampino commented on KNOX-1014:
{quote}1. I notice a lot of hardcoded service names and some mappings - these
would be better done more dynamically so as not to require a compile to add
support for new services (can be a follow up JIRA){quote}
I believe KNOX-1064 addresses this concern.
> Service Discovery and Topology Generation Framework
> ---
>
> Key: KNOX-1014
> URL: https://issues.apache.org/jira/browse/KNOX-1014
> Project: Apache Knox
> Issue Type: Sub-task
> Components: Server
>Reporter: Phil Zampino
>Assignee: Phil Zampino
> Labels: kip-8
> Fix For: 0.14.0
>
> Attachments: KNOX-1014-002.patch, KNOX-1014.patch
>
>
> Implement the foundation for Service Discovery and Topology Generation.
> * Define simple descriptor format (YAML, JSON, etc...)
> * Local simple descriptor and shared provider configuration discovery
> ** Monitor conf/shared-providers, conf/descriptors similar to the way
> conf/topologies is currently monitored.
> * Ambari service discovery (REST API interactions and model construction)
> ** Configuration
> *** How to plug-in discovery implementations
> *** How to configure authentication (credentials/trust) with the service
> registries
> * Topology assembly from simple descriptor and discovery details
> * Topology deployment
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KNOX-1014) Service Discovery and Topology Generation Framework
[ https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179713#comment-16179713 ] ASF subversion and git services commented on KNOX-1014: --- Commit a841e2656b359e4f9bd6e819eba2b9723faec1bb in knox's branch refs/heads/KNOX-998-Package_Restructuring from [~lmccay] [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=a841e26 ] KNOX-1014 - remove extraneous directory > Service Discovery and Topology Generation Framework > --- > > Key: KNOX-1014 > URL: https://issues.apache.org/jira/browse/KNOX-1014 > Project: Apache Knox > Issue Type: Sub-task > Components: Server >Reporter: Phil Zampino >Assignee: Phil Zampino > Labels: kip-8 > Fix For: 0.14.0 > > Attachments: KNOX-1014-002.patch, KNOX-1014.patch > > > Implement the foundation for Service Discovery and Topology Generation. > * Define simple descriptor format (YAML, JSON, etc...) > * Local simple descriptor and shared provider configuration discovery > ** Monitor conf/shared-providers, conf/descriptors similar to the way > conf/topologies is currently monitored. > * Ambari service discovery (REST API interactions and model construction) > ** Configuration > *** How to plug-in discovery implementations > *** How to configure authentication (credentials/trust) with the service > registries > * Topology assembly from simple descriptor and discovery details > * Topology deployment -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1014) Service Discovery and Topology Generation Framework
[
https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178456#comment-16178456
]
Phil Zampino commented on KNOX-1014:
[~lmccay]
{quote}
1. hive protocol scheme
2. deployment fail for missing services or no?
3. .yaml extension in addition to .yml
4. service level params
{quote}
# It was unclear to me whether hive proxying through Knox is supported if the
transport mode is NOT http. If it is unsupported, then the resolution of the
second item in your list comes into play...
# Prior to supporting service discovery, a topology would deploy just fine with
incorrect/invalid service URLs. However, I could be persuaded that failure to
dynamically determine a URL for a service declared in a descriptor should cause
topology generation to fail (since discovery knows it would be generating an
incorrect topology whereas a human does not necessarily know this).
# This is literally a one-line change; Consider it done.
# I'll need to learn about these service params in order to add support for
them.
> Service Discovery and Topology Generation Framework
> ---
>
> Key: KNOX-1014
> URL: https://issues.apache.org/jira/browse/KNOX-1014
> Project: Apache Knox
> Issue Type: Sub-task
> Components: Server
>Reporter: Phil Zampino
>Assignee: Phil Zampino
> Labels: kip-8
> Fix For: 0.14.0
>
> Attachments: KNOX-1014-002.patch, KNOX-1014.patch
>
>
> Implement the foundation for Service Discovery and Topology Generation.
> * Define simple descriptor format (YAML, JSON, etc...)
> * Local simple descriptor and shared provider configuration discovery
> ** Monitor conf/shared-providers, conf/descriptors similar to the way
> conf/topologies is currently monitored.
> * Ambari service discovery (REST API interactions and model construction)
> ** Configuration
> *** How to plug-in discovery implementations
> *** How to configure authentication (credentials/trust) with the service
> registries
> * Topology assembly from simple descriptor and discovery details
> * Topology deployment
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KNOX-1014) Service Discovery and Topology Generation Framework
[ https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177896#comment-16177896 ] Larry McCay commented on KNOX-1014: --- Another consideration that we need to account for is parameters for services. There are a number of tuning parameters for various services like hive and the Authentication services APIs like often KnoxSSO and KnoxToken require configuration. I think this will need a follow up JIRA for sure. Unfortunately, this will make the descriptor less simple but it is needed. > Service Discovery and Topology Generation Framework > --- > > Key: KNOX-1014 > URL: https://issues.apache.org/jira/browse/KNOX-1014 > Project: Apache Knox > Issue Type: Sub-task > Components: Server >Reporter: Phil Zampino >Assignee: Phil Zampino > Labels: kip-8 > Fix For: 0.14.0 > > Attachments: KNOX-1014-002.patch, KNOX-1014.patch > > > Implement the foundation for Service Discovery and Topology Generation. > * Define simple descriptor format (YAML, JSON, etc...) > * Local simple descriptor and shared provider configuration discovery > ** Monitor conf/shared-providers, conf/descriptors similar to the way > conf/topologies is currently monitored. > * Ambari service discovery (REST API interactions and model construction) > ** Configuration > *** How to plug-in discovery implementations > *** How to configure authentication (credentials/trust) with the service > registries > * Topology assembly from simple descriptor and discovery details > * Topology deployment -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1014) Service Discovery and Topology Generation Framework
[ https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177835#comment-16177835 ] Larry McCay commented on KNOX-1014: --- I have just committed this to master and it will be in the 0.14.0 release. Thank you for this contribution, [~pzampino]! > Service Discovery and Topology Generation Framework > --- > > Key: KNOX-1014 > URL: https://issues.apache.org/jira/browse/KNOX-1014 > Project: Apache Knox > Issue Type: Sub-task > Components: Server >Reporter: Phil Zampino >Assignee: Phil Zampino > Labels: kip-8 > Fix For: 0.14.0 > > Attachments: KNOX-1014-002.patch, KNOX-1014.patch > > > Implement the foundation for Service Discovery and Topology Generation. > * Define simple descriptor format (YAML, JSON, etc...) > * Local simple descriptor and shared provider configuration discovery > ** Monitor conf/shared-providers, conf/descriptors similar to the way > conf/topologies is currently monitored. > * Ambari service discovery (REST API interactions and model construction) > ** Configuration > *** How to plug-in discovery implementations > *** How to configure authentication (credentials/trust) with the service > registries > * Topology assembly from simple descriptor and discovery details > * Topology deployment -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1014) Service Discovery and Topology Generation Framework
[ https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177834#comment-16177834 ] ASF subversion and git services commented on KNOX-1014: --- Commit c2ca443262a848ae0d56e03e92ecba32fbf149f2 in knox's branch refs/heads/master from [~lmccay] [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=c2ca443 ] KNOX-1014 - Service Discovery and Topology Generation Framework (Phil Zampino via lmccay) > Service Discovery and Topology Generation Framework > --- > > Key: KNOX-1014 > URL: https://issues.apache.org/jira/browse/KNOX-1014 > Project: Apache Knox > Issue Type: Sub-task > Components: Server >Reporter: Phil Zampino >Assignee: Phil Zampino > Labels: kip-8 > Fix For: 0.14.0 > > Attachments: KNOX-1014-002.patch, KNOX-1014.patch > > > Implement the foundation for Service Discovery and Topology Generation. > * Define simple descriptor format (YAML, JSON, etc...) > * Local simple descriptor and shared provider configuration discovery > ** Monitor conf/shared-providers, conf/descriptors similar to the way > conf/topologies is currently monitored. > * Ambari service discovery (REST API interactions and model construction) > ** Configuration > *** How to plug-in discovery implementations > *** How to configure authentication (credentials/trust) with the service > registries > * Topology assembly from simple descriptor and discovery details > * Topology deployment -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1014) Service Discovery and Topology Generation Framework
[ https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177832#comment-16177832 ] Larry McCay commented on KNOX-1014: --- It seems that until you put hiveserver2 in http mode, you will get null. We should probably check for that and default to http. Not a blocker now though. So follow up topics: 1. hive protocol scheme 2. deployment fail for missing services or no? 3. .yaml extension in addition to .yml I am going to commit this as is but would like to have follow up JIRAs for the above - if you agree. > Service Discovery and Topology Generation Framework > --- > > Key: KNOX-1014 > URL: https://issues.apache.org/jira/browse/KNOX-1014 > Project: Apache Knox > Issue Type: Sub-task > Components: Server >Reporter: Phil Zampino >Assignee: Phil Zampino > Labels: kip-8 > Fix For: 0.14.0 > > Attachments: KNOX-1014-002.patch, KNOX-1014.patch > > > Implement the foundation for Service Discovery and Topology Generation. > * Define simple descriptor format (YAML, JSON, etc...) > * Local simple descriptor and shared provider configuration discovery > ** Monitor conf/shared-providers, conf/descriptors similar to the way > conf/topologies is currently monitored. > * Ambari service discovery (REST API interactions and model construction) > ** Configuration > *** How to plug-in discovery implementations > *** How to configure authentication (credentials/trust) with the service > registries > * Topology assembly from simple descriptor and discovery details > * Topology deployment -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1014) Service Discovery and Topology Generation Framework
[ https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177828#comment-16177828 ] Larry McCay commented on KNOX-1014: --- I also notice that only the .yml extension is getting picked up for YAML descriptors. Should we also allow for .yaml? I believe that is also a valid extension for YAML. Again, not a blocker. The hive protocol issue may be though. > Service Discovery and Topology Generation Framework > --- > > Key: KNOX-1014 > URL: https://issues.apache.org/jira/browse/KNOX-1014 > Project: Apache Knox > Issue Type: Sub-task > Components: Server >Reporter: Phil Zampino >Assignee: Phil Zampino > Labels: kip-8 > Fix For: 0.14.0 > > Attachments: KNOX-1014-002.patch, KNOX-1014.patch > > > Implement the foundation for Service Discovery and Topology Generation. > * Define simple descriptor format (YAML, JSON, etc...) > * Local simple descriptor and shared provider configuration discovery > ** Monitor conf/shared-providers, conf/descriptors similar to the way > conf/topologies is currently monitored. > * Ambari service discovery (REST API interactions and model construction) > ** Configuration > *** How to plug-in discovery implementations > *** How to configure authentication (credentials/trust) with the service > registries > * Topology assembly from simple descriptor and discovery details > * Topology deployment -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1014) Service Discovery and Topology Generation Framework
[
https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177820#comment-16177820
]
Larry McCay commented on KNOX-1014:
---
Materialized topology is missing the protocol scheme for the HIVE service.
All the others seem good.
{code}
HIVE
null://c6401.ambari.apache.org:10001/cliservice
{code}
I did have a couple services in the descriptor that were not deployed in ambari
and the topology did fail to materialize as intended.
I'm wondering whether it would be better to just leave them out of the
materialized topology but to allow those that are discovered to be deployed.
Would be good to be able to add comments to it indicated that they were removed
but that is probably not doable.
If you think about what needs to be done to make it deploy there are two
options:
1. remove the undiscoverable service
2. add the service through ambari
#1 we can do for them and if the service ends up being deployed in ambari then
it will magically show up.
#2 we have no control over.
What do you think?
I'm not thinking that either of these necessarily have to block the commit.
The missing HIVE scheme is interesting though - you didn't see that happen in
your deployments?
Maybe there is something special in my cluster?
> Service Discovery and Topology Generation Framework
> ---
>
> Key: KNOX-1014
> URL: https://issues.apache.org/jira/browse/KNOX-1014
> Project: Apache Knox
> Issue Type: Sub-task
> Components: Server
>Reporter: Phil Zampino
>Assignee: Phil Zampino
> Labels: kip-8
> Fix For: 0.14.0
>
> Attachments: KNOX-1014-002.patch, KNOX-1014.patch
>
>
> Implement the foundation for Service Discovery and Topology Generation.
> * Define simple descriptor format (YAML, JSON, etc...)
> * Local simple descriptor and shared provider configuration discovery
> ** Monitor conf/shared-providers, conf/descriptors similar to the way
> conf/topologies is currently monitored.
> * Ambari service discovery (REST API interactions and model construction)
> ** Configuration
> *** How to plug-in discovery implementations
> *** How to configure authentication (credentials/trust) with the service
> registries
> * Topology assembly from simple descriptor and discovery details
> * Topology deployment
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KNOX-1014) Service Discovery and Topology Generation Framework
[ https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177801#comment-16177801 ] Larry McCay commented on KNOX-1014: --- That sounds perfect. Going to do a detailed review now! > Service Discovery and Topology Generation Framework > --- > > Key: KNOX-1014 > URL: https://issues.apache.org/jira/browse/KNOX-1014 > Project: Apache Knox > Issue Type: Sub-task > Components: Server >Reporter: Phil Zampino >Assignee: Phil Zampino > Labels: kip-8 > Fix For: 0.14.0 > > Attachments: KNOX-1014-002.patch, KNOX-1014.patch > > > Implement the foundation for Service Discovery and Topology Generation. > * Define simple descriptor format (YAML, JSON, etc...) > * Local simple descriptor and shared provider configuration discovery > ** Monitor conf/shared-providers, conf/descriptors similar to the way > conf/topologies is currently monitored. > * Ambari service discovery (REST API interactions and model construction) > ** Configuration > *** How to plug-in discovery implementations > *** How to configure authentication (credentials/trust) with the service > registries > * Topology assembly from simple descriptor and discovery details > * Topology deployment -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1014) Service Discovery and Topology Generation Framework
[ https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170728#comment-16170728 ] Larry McCay commented on KNOX-1014: --- [~pzampino] - This is looking really good! Couple things for as far as I have gotten in the review process - may have more later: 1. I notice a lot of hardcoded service names and some mappings - these would be better done more dynamically so as not to require a compile to add support for new services (can be a follow up JIRA) 2. I notice a credential lookup from the alias service - this is a great idea but I think we should probably use a property name sort of key rather than user name. Too easy to have multiple passwords for the same user for different contexts. I see this as more of a config item stored in a credential store than a user based thing. Might be able to be convinced otherwise though. :) 3. question: if a service is in the simple descriptor but unknown by ambari what happens to the materialized topology? 4. question: if a service is in the simple descriptor but just not installed in the deployment what happens to the materialized topology? > Service Discovery and Topology Generation Framework > --- > > Key: KNOX-1014 > URL: https://issues.apache.org/jira/browse/KNOX-1014 > Project: Apache Knox > Issue Type: Sub-task > Components: Server >Reporter: Phil Zampino >Assignee: Phil Zampino > Labels: kip-8 > Fix For: 0.14.0 > > Attachments: KNOX-1014.patch > > > Implement the foundation for Service Discovery and Topology Generation. > * Define simple descriptor format (YAML, JSON, etc...) > * Local simple descriptor and shared provider configuration discovery > ** Monitor conf/shared-providers, conf/descriptors similar to the way > conf/topologies is currently monitored. > * Ambari service discovery (REST API interactions and model construction) > ** Configuration > *** How to plug-in discovery implementations > *** How to configure authentication (credentials/trust) with the service > registries > * Topology assembly from simple descriptor and discovery details > * Topology deployment -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1014) Service Discovery and Topology Generation Framework
[ https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163878#comment-16163878 ] Larry McCay commented on KNOX-1014: --- [~pzampino] - that completely makes sense. Thank you. I will review this within the next day or so. > Service Discovery and Topology Generation Framework > --- > > Key: KNOX-1014 > URL: https://issues.apache.org/jira/browse/KNOX-1014 > Project: Apache Knox > Issue Type: Sub-task > Components: Server >Reporter: Phil Zampino >Assignee: Phil Zampino > Labels: kip-8 > Fix For: 0.14.0 > > Attachments: KNOX-1014.patch > > > Implement the foundation for Service Discovery and Topology Generation. > * Define simple descriptor format (YAML, JSON, etc...) > * Local simple descriptor and shared provider configuration discovery > ** Monitor conf/shared-providers, conf/descriptors similar to the way > conf/topologies is currently monitored. > * Ambari service discovery (REST API interactions and model construction) > ** Configuration > *** How to plug-in discovery implementations > *** How to configure authentication (credentials/trust) with the service > registries > * Topology assembly from simple descriptor and discovery details > * Topology deployment -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KNOX-1014) Service Discovery and Topology Generation Framework
[ https://issues.apache.org/jira/browse/KNOX-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163761#comment-16163761 ] Phil Zampino commented on KNOX-1014: [~lmccay] I went back and forth on that behavior wrt to shared provider configuration deletions. I've modified it such that deleting a shared provider config that is referenced by one or more simple descriptors no longer results in those descriptors being deleted. Once nice thing about this change is that one is able to recover from an unintentional shared provider configuration deletion. If one of these is deleted, the simple descriptor processing will fail, and the topology generation/redeployment will be aborted. This means 1) Your topologies will still be running and working as they were 2) You can recover the shared provider configuration by copying it from a referencing topology in conf/topologies. I've also enabled support for YAML descriptors. While this would not be a good wire format for the API, it is a good config file format to support hand-editing descriptors. The associated APIs will use the JSON format, but the YAML will be a convenience for local manual deployments. I've updated the attached patch file with both of these changes. > Service Discovery and Topology Generation Framework > --- > > Key: KNOX-1014 > URL: https://issues.apache.org/jira/browse/KNOX-1014 > Project: Apache Knox > Issue Type: Sub-task > Components: Server >Reporter: Phil Zampino >Assignee: Phil Zampino > Labels: kip-8 > Fix For: 0.14.0 > > Attachments: KNOX-1014.patch > > > Implement the foundation for Service Discovery and Topology Generation. > * Define simple descriptor format (YAML, JSON, etc...) > * Local simple descriptor and shared provider configuration discovery > ** Monitor conf/shared-providers, conf/descriptors similar to the way > conf/topologies is currently monitored. > * Ambari service discovery (REST API interactions and model construction) > ** Configuration > *** How to plug-in discovery implementations > *** How to configure authentication (credentials/trust) with the service > registries > * Topology assembly from simple descriptor and discovery details > * Topology deployment -- This message was sent by Atlassian JIRA (v6.4.14#64029)
