Re: [openstack-dev] Configuration Discovery - Satori
On Tue, 2014-01-28 at 22:36 +, Adrian Otto wrote: OpenStack Devs, I'd like to introduce you to a team working on an interesting problem space. We would like to know what you think about Configuration Discovery. We plan to build a tool that aids in the process of automated configuration discovery. At Rackspace we work with customers who have pre-existing infrastructure and software running in production. To help them with changes, migrations, or configuration management, we often have to determine what they have in-place already. We have some tools and knowledge in-house that we use now, but we would like to create something even better in the open that would integrate well with OpenStack ecosystem projects like Heat, Solum, Murano, etc… The concept for the tool is simple. The inputs are facts we know about the current system (a username, API Key, URL, etc.) and the output is a set of structured configuration information. That information can be used for: - Generating Heat Templates - Solum Application creation/import - Creation of Chef recipes/cookbooks, Puppet modules, Ansible playbooks, setup scripts, etc.. - Configuration analysis (compare this config with a catalog of best practices) - Configuration monitoring (has the configuration changed?) - Troubleshooting We would like to hear from anyone interested in this problem domain, start a conversation on how this fits in the OpenStack ecosystem, and start exploring possible use cases that would be useful for users and operators of OpenStack clouds. For more information about the team, implementation, and prior art in this area, please see: https://wiki.openstack.org/wiki/Satori In the Related Work section, you list: Devstructure Blueprint (https://github.com/devstructure/blueprint) That is precisely what I would recommend. I don't see value in having a separate OpenStack project that does this. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Configuration Discovery - Satori
On 01/28/2014 06:02 PM, Jay Pipes wrote: On Tue, 2014-01-28 at 22:36 +, Adrian Otto wrote: OpenStack Devs, I'd like to introduce you to a team working on an interesting problem space. We would like to know what you think about Configuration Discovery. We plan to build a tool that aids in the process of automated configuration discovery. At Rackspace we work with customers who have pre-existing infrastructure and software running in production. To help them with changes, migrations, or configuration management, we often have to determine what they have in-place already. We have some tools and knowledge in-house that we use now, but we would like to create something even better in the open that would integrate well with OpenStack ecosystem projects like Heat, Solum, Murano, etc… The concept for the tool is simple. The inputs are facts we know about the current system (a username, API Key, URL, etc.) and the output is a set of structured configuration information. That information can be used for: - Generating Heat Templates - Solum Application creation/import - Creation of Chef recipes/cookbooks, Puppet modules, Ansible playbooks, setup scripts, etc.. - Configuration analysis (compare this config with a catalog of best practices) - Configuration monitoring (has the configuration changed?) - Troubleshooting We would like to hear from anyone interested in this problem domain, start a conversation on how this fits in the OpenStack ecosystem, and start exploring possible use cases that would be useful for users and operators of OpenStack clouds. For more information about the team, implementation, and prior art in this area, please see: https://wiki.openstack.org/wiki/Satori In the Related Work section, you list: Devstructure Blueprint (https://github.com/devstructure/blueprint) That is precisely what I would recommend. I don't see value in having a separate OpenStack project that does this. Sometimes the right answer is to join in and help extend an existing project. :-) If that's *not* the answer, there should be a compelling reason why. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Configuration Discovery - Satori
On 01/28/2014 06:02 PM, Jay Pipes wrote: On Tue, 2014-01-28 at 22:36 +, Adrian Otto wrote: OpenStack Devs, I'd like to introduce you to a team working on an interesting problem space. We would like to know what you think about Configuration Discovery. We plan to build a tool that aids in the process of automated configuration discovery. At Rackspace we work with customers who have pre-existing infrastructure and software running in production. To help them with changes, migrations, or configuration management, we often have to determine what they have in-place already. We have some tools and knowledge in-house that we use now, but we would like to create something even better in the open that would integrate well with OpenStack ecosystem projects like Heat, Solum, Murano, etc… The concept for the tool is simple. The inputs are facts we know about the current system (a username, API Key, URL, etc.) and the output is a set of structured configuration information. That information can be used for: - Generating Heat Templates - Solum Application creation/import - Creation of Chef recipes/cookbooks, Puppet modules, Ansible playbooks, setup scripts, etc.. - Configuration analysis (compare this config with a catalog of best practices) - Configuration monitoring (has the configuration changed?) - Troubleshooting We would like to hear from anyone interested in this problem domain, start a conversation on how this fits in the OpenStack ecosystem, and start exploring possible use cases that would be useful for users and operators of OpenStack clouds. For more information about the team, implementation, and prior art in this area, please see: https://wiki.openstack.org/wiki/Satori In the Related Work section, you list: Devstructure Blueprint (https://github.com/devstructure/blueprint) That is precisely what I would recommend. I don't see value in having a separate OpenStack project that does this. ACK. Also, thanks for bringing attention to such a cool project. :) -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Configuration Discovery - Satori
Andrian, Looks an interesting idea. Would you envision Satori primarily as an migration tool? Let's use the following scenarios as an example to explore the scope in your mind. Server1 is a physical server running a mail server service, and we would like migrate it onto a virtual server provisioned by our OpenStack cluster. In this case, would Satori be helpful? Server2 is a VM that I manually installed after OpenStack provides me a VM with a basic Ubuntu 12.10 image (as I manually did a lot of things and did not follow the Infrastructure as code philosophy, I do not know where I am at now), I want Satori to examine the system and create a cookbook or a heat template for me. Is this the Satori's primary target case? Server3 is an EC2 instance, and I want to migrate that instance to my OpenStack cluster. In this case, would Satori be helpful? BTW, what does Configuration monitoring mean? Can you elaborate it? Love to hear more about you thoughts. Thanks, Shuo -Original Message- From: Adrian Otto [mailto:adrian.o...@rackspace.com] Sent: Tuesday, January 28, 2014 2:36 PM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] Configuration Discovery - Satori OpenStack Devs, I'd like to introduce you to a team working on an interesting problem space. We would like to know what you think about Configuration Discovery. We plan to build a tool that aids in the process of automated configuration discovery. At Rackspace we work with customers who have pre-existing infrastructure and software running in production. To help them with changes, migrations, or configuration management, we often have to determine what they have in-place already. We have some tools and knowledge in-house that we use now, but we would like to create something even better in the open that would integrate well with OpenStack ecosystem projects like Heat, Solum, Murano, etc... The concept for the tool is simple. The inputs are facts we know about the current system (a username, API Key, URL, etc.) and the output is a set of structured configuration information. That information can be used for: - Generating Heat Templates - Solum Application creation/import - Creation of Chef recipes/cookbooks, Puppet modules, Ansible playbooks, setup scripts, etc.. - Configuration analysis (compare this config with a catalog of best practices) - Configuration monitoring (has the configuration changed?) - Troubleshooting We would like to hear from anyone interested in this problem domain, start a conversation on how this fits in the OpenStack ecosystem, and start exploring possible use cases that would be useful for users and operators of OpenStack clouds. For more information about the team, implementation, and prior art in this area, please see: https://wiki.openstack.org/wiki/Satori Thanks, Adrian Otto ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Configuration Discovery - Satori
On January 28, 2014 at 5:05:56 PM, Jay Pipes (jaypi...@gmail.com) wrote: In the Related Work section, you list: Devstructure Blueprint (https://github.com/devstructure/blueprint) That is precisely what I would recommend. I don't see value in having a separate OpenStack project that does this. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Hi Jay, Devstructure Blueprint is scoped to gathering information about a single server. We listed it as related because its a handy way to handle reverse engineering once you’re logged into a server instance. However, Satori has greater aims such as discovering the topology of resources (load balancer, its configuration, nova instances behind the load balancer, connected cinder instances, etc). For each relevant server instance we could gather deep system knowledge by using the projects listed in the Related section. Caleb ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Configuration Discovery - Satori
On January 28, 2014 at 5:18:42 PM, Yang Shuo (Shuo) (shuo.y...@huawei.com(mailto://shuo.y...@huawei.com)) wrote: Andrian, Looks an interesting idea. Would you envision Satori primarily as an migration tool? Let's use the following scenarios as an example to explore the scope in your mind. Migrations are not the primary use case that we are targeting. Primarily, Satori aims to discover as much detail as possible for an existing configuration. Imagine walking into a clients office with the mission to optimize the performance of domain.com. One of the first steps is to understand how the current environment is configured. Satori would give you a quick snapshot of all of the related systems in use. This includes: - DNS settings - SSL certificate details - Load balancer settings and status of the connection pool - Nova instances in the connection pool and piles of relevant info (OS, software installed, active connections to other nova/trove instances, etc). Imagine the client handing you a Visio diagram of the discovered configuration for the entire site that isn’t 6 months old and several major changes behind reality :) Server1 is a physical server running a mail server service, and we would like migrate it onto a virtual server provisioned by our OpenStack cluster. In this case, would Satori be helpful? The hardest part of the migration is dealing with the existing data. Satori could inspect the server and provide details of how it is configured but it wouldn’t seek to promise you a one-click migration that includes moving all of your data. Server2 is a VM that I manually installed after OpenStack provides me a VM with a basic Ubuntu 12.10 image (as I manually did a lot of things and did not follow the Infrastructure as code philosophy, I do not know where I am at now), I want Satori to examine the system and create a cookbook or a heat template for me. Is this the Satori's primary target case? This is a valid use case that we would target. However, a single server probe is handled very well by Devstructure’s Blueprint already. We would provide value in more complicated scenarios where several OpenStack resources and services are in use. The generated Heat template would provision the servers and the associated resources (queue, cinder volumes, trove database, etc). Server3 is an EC2 instance, and I want to migrate that instance to my OpenStack cluster. In this case, would Satori be helpful? This feels very similar to your first use case. BTW, what does Configuration monitoring mean? Can you elaborate it? Configuration monitoring is the passive detection of changes between discovery attempts. Monitoring is probably the wrong word as it implies constant polling. We could store and diff the results of multiple discovery runs. Love to hear more about you thoughts. Thanks, Shuo I hope this helps distinguish Satori and a single server reverse engineering project like Blueprint. Caleb ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Configuration Discovery - Satori
On Tue, 2014-01-28 at 17:20 -0600, Caleb Groom wrote: On January 28, 2014 at 5:05:56 PM, Jay Pipes (jaypi...@gmail.com) wrote: In the Related Work section, you list: Devstructure Blueprint (https://github.com/devstructure/blueprint) That is precisely what I would recommend. I don't see value in having a separate OpenStack project that does this. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Hi Jay, Devstructure Blueprint is scoped to gathering information about a single server. We listed it as related because its a handy way to handle reverse engineering once you’re logged into a server instance. However, Satori has greater aims such as discovering the topology of resources (load balancer, its configuration, nova instances behind the load balancer, connected cinder instances, etc). For each relevant server instance we could gather deep system knowledge by using the projects listed in the Related section. So why not improve Devstructure Blueprint and add in some plugins that, given some OpenStack creds, query things like the Neutron and Nova API endpoints for such information. I don't think this needs to be a separate OpenStack service. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Configuration Discovery - Satori
On January 28, 2014 at 6:23:14 PM, Jay Pipes (jaypi...@gmail.com(mailto://jaypi...@gmail.com)) wrote: On Tue, 2014-01-28 at 17:20 -0600, Caleb Groom wrote: On January 28, 2014 at 5:05:56 PM, Jay Pipes (jaypi...@gmail.com) wrote: In the Related Work section, you list: Devstructure Blueprint (https://github.com/devstructure/blueprint) That is precisely what I would recommend. I don't see value in having a separate OpenStack project that does this. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Hi Jay, Devstructure Blueprint is scoped to gathering information about a single server. We listed it as related because its a handy way to handle reverse engineering once you’re logged into a server instance. However, Satori has greater aims such as discovering the topology of resources (load balancer, its configuration, nova instances behind the load balancer, connected cinder instances, etc). For each relevant server instance we could gather deep system knowledge by using the projects listed in the Related section. So why not improve Devstructure Blueprint and add in some plugins that, given some OpenStack creds, query things like the Neutron and Nova API endpoints for such information. That's a good question. Here's my current thinking: - Blueprint is one method to obtain system information but there are alternatives. Chef's Ohai is particularly interesting because of its plugin structure and active community. Ohai is a standalone application that can be used without needing a server to be actively managed by Chef. My team has recently started to experiment with using Ohai without Chef [1]. Like Blueprint, it provides nice JSON output. - Blueprint isn't under active development [2]. - The configuration management world is rapidly evolving. I believe that Satori should interact with them in a pluggable fashion so that the project's success isn't tied to any one specific vendor. - Blueprint doesn't support Windows or Linux distributions outside of the Red Hat and Debian families. - The design of Blueprint assumes that you're executing commands against localhost only, making a server the center of the universe. However, I believe that there are several use cases for discovery where logging into the servers to fetch data would be nice to have but not required. The center of the universe could be the tenant (show me everything about this tenant) or a website (How is domain.com configured?). Here's an example of discovery without accessing the servers: -- domain.com resolves to 1.2.3.4. -- That IP belongs to LBaaS instance xyz. -- That LBaaS instance supports SSL termination and has 3 nova instances in the connection pool. -- 2 of the 3 instances in the connection pool are offline. -- A security group allows external connections to the public IPs of the nova instances. -- The offline servers are build from a glance image that was updated today but the online server is 7 days old. I believe this is valuable information that can be provided to users and isn't near the current scope of Blueprint. I don't think this needs to be a separate OpenStack service. Best, -jay That's fair. Our intention today is to share our interest in the domain of discovery and our intention to work on this problem using open design, open development and open community. In time we'll solve some use cases that will hopefully make it more clear why we find this problem so interesting! We'll start writing Launchpad blueprints that will get get into more of the details. Caleb [1] https://github.com/rackerlabs/ohai-plugins [2] https://github.com/devstructure/blueprint/graphs/commit-activity ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Configuration Discovery - Satori
Hi Jay, On Jan 28, 2014, at 4:23 PM, Jay Pipes jaypi...@gmail.com wrote: On Tue, 2014-01-28 at 17:20 -0600, Caleb Groom wrote: On January 28, 2014 at 5:05:56 PM, Jay Pipes (jaypi...@gmail.com) wrote: In the Related Work section, you list: Devstructure Blueprint (https://github.com/devstructure/blueprint) That is precisely what I would recommend. I don't see value in having a separate OpenStack project that does this. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Hi Jay, Devstructure Blueprint is scoped to gathering information about a single server. We listed it as related because its a handy way to handle reverse engineering once you’re logged into a server instance. However, Satori has greater aims such as discovering the topology of resources (load balancer, its configuration, nova instances behind the load balancer, connected cinder instances, etc). For each relevant server instance we could gather deep system knowledge by using the projects listed in the Related section. So why not improve Devstructure Blueprint and add in some plugins that, given some OpenStack creds, query things like the Neutron and Nova API endpoints for such information. If that approach would yield the right balance of open collaboration that we are seeking, that's an option that can certainly be revisited. We have learned from experience that organizing development around an open collaboration can yield some very good results. Contributing to existing projects may help us advance on the mission, but it may not be as effective as leveraging all of our lessons learned about how to work in the open. I think this particular concept is more ambitious than the existing work, and may demand a good deal of attention from the current maintainers. I don't think this needs to be a separate OpenStack service. Oh, neither does the Satori team! We are in complete agreement. This is a vision for a tool intended to help OpenStack Cloud operators work more easily, troubleshoot more quickly, etc. It does not need to be a service at all to achieve that. Cheers, Adrian Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
