Re: [openstack-dev] [Fuel] CLI api for working with granular deployment model
Also very important to understand that if task is mapped to role controller, but node where you want to apply that task doesn't have this role - it wont be executed. Is there a particular reason why we want to restrict a user to run an arbitrary task on a server, even if server doesn't have a role assigned? I think we should be flexible here - if I'm hacking something, I'd like to run arbitrary things. The reason it is not supported is that such behaviour will require two different endpoints, with quite similar functionality. In most cases developer will benefit from relying on role mappings, for instance right now one will be able to test dependent tasks on different nodes by next commands: fuel node --node 1,2,3 --tasks corosync_primary corosync_slave fuel node --node 1,2 --tasks controller_service compute_service IMO it is reasonable requirement for developer to ensure that task is properly inserted into deployment configuration. Also there was a discussion to implement an api that will bypass all nailgun logic and will allow to communicate directly with orchestrator hooks, like: fuel exec file_with_tasks.yaml Where file_with_tasks filled with data consumable directly by orchestrator fuel node --node 1,2,3 --end netconfig I would replace --end - --end-on, in order to show that task specified will run as well (to avoid ambiguity) This is separate question probably about CLI UX, but still - are we Ok with missing an action verb, like deploy? So it might be better to have, in my opinion: fuel deploy --node 1,2,3 --end netconfig We may want to put everything that is related to deployment under one CLI namespace, but IMO we need to be consistent and regular deploy/provision should be migrated as well. ' For example if one want to execute only netconfig successors: fuel node --node 1,2,3 --start netconfig --skip netconfig I would come up with shortcut for one task. To me, it would be way better to specify comma-separated tasks: fuel deploy --node 1,2,3 --task netconfig[,task2] I dont like comma-separted notation at all, if majority will think that it is more readable than whitespace - lets do it. Question here: if netconfig depends on other tasks, will those be executed prior to netconfig? I want both options, execute with prior deps, and execute just one particular task. When tasks provided with --tasks flag - no additional dependencies will be included. Traversal will be performed only with --end and --start flags. As a separate note here, a few question: 1. If particular task fails to execute for some reason, what is the error handling? Will I be able to see puppet/deployment tool exception right in the same console, or should I check out some logs? We need to have perfect UX for errors. Those who will be using CLI to run particular tasks, will be dealing with errors for 95% of their time. There will be UI message that something with that id is failed. But developer will need to go for logs (astute preferably). What you are suggesting is doable, but not that trivial.. We will check how much time this will take, and maybe there is other ways to improve deployment feedback. 1. I'd love to have some guidance on slave node as well. For instance, I want to run just netconfig on slave node. How can I do it? You mean completely bypassing fuel control plane? Developer will be able to use underlying tools directly, puppet apply, python, ruby, whatever that task is using. We may add a helper, to show all tasks endpoints in single place, but they can be found easily by usual grep.. 1. If I stuck with error in task execution, which is in puppet. Can I modify puppet module on master node, and re-run the task? (assuming that updated module will be rsynced to slaves under deployment first) Rsync puppet is separate task, so one will need to execute: fuel node --node 1,2,3 --tasks rsync_core_puppet netconfig __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] CLI api for working with granular deployment model
On Sat, Feb 7, 2015 at 9:42 AM, Andrew Woodward xar...@gmail.com wrote: Dmitry, thanks for sharing CLI options. I'd like to clarify a few things. Also very important to understand that if task is mapped to role controller, but node where you want to apply that task doesn't have this role - it wont be executed. Is there a particular reason why we want to restrict a user to run an arbitrary task on a server, even if server doesn't have a role assigned? I think we should be flexible here - if I'm hacking something, I'd like to run arbitrary things. The way I've seen this work so far is the missing role in the graph simply wont be executed, not the requested role Hi Andrew, What do you mean by requested role? If you want to add new role to fuel, lets say redis - adding new group into deployment configuration is mandatory, here is what it looks like [0] Then one will need to add tasks that are required for this group (both custom and basic tasks like hiera netconfig), lets say custom task is install_redis. After this is done user will be able to use cli: fuel node --node 5 --tasks install_redis OR --end install_redis [0] https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/deployment_groups/tasks.yaml __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] CLI api for working with granular deployment model
Dmitriy, Thank you for the excellent run-down of the CLI commands. I assume this will make its way into the developer documentation? I would like to know if you could point me to more information about the inner workings of granular deployment. Currently it's challenging to debug issues related to granular deployment. As an example there is a bug [0] where tasks appear to be run in the wrong order based on which combination of roles exist in the environment. However, it's not clear how to determine what decides which tasks to run and when (is it astute, fuel-library, etc.), where the data comes from. etc. Again, thanks for your (and everybody else's) work on granular deployment. This is an awesome feature. [0] https://bugs.launchpad.net/fuel/+bug/1411660 -Ryan On Fri, Feb 6, 2015 at 6:37 AM, Dmitriy Shulyak dshul...@mirantis.com wrote: Hello folks, Not long ago we added necessary commands in fuel client to work with granular deployment configuration and API. So, you may know that configuration is stored in fuel-library, and uploaded into database during bootstrap of fuel master. If you want to change/add some tasks right on master node, just add tasks.yaml and appropriate manifests in folder for release that you are interested in. Then apply this command: fuel rel --sync-deployment-tasks --dir /etc/puppet Also you may want to overwrite deployment tasks for any specific release/cluster by next commands: fuel rel --rel id --deployment-tasks --download fuel rel --rel id --deployment-tasks --upload fuel env --env id --deployment-tasks --download fuel env --env id --deployment-tasks --upload After this is done - you will be able to run customized graph of tasks: The most basic command: fuel node --node 1,2,3 --tasks upload_repos netconfig Developer will need to specify nodes that should be used in deployment and tasks ids. Order in which they are provided doesn't matter, it will be computed from dependencies specified in database. Also very important to understand that if task is mapped to role controller, but node where you want to apply that task doesn't have this role - it wont be executed. Skipping of tasks fuel node --node 1,2,3 --skip netconfig hiera List of task that are provided with this parameter will be skipped during graph traversal in nailgun. The main question is - should we skip other task that have provided tasks as dependencies? In my opinion we can leave this flag as simple as it is, and use following commands for smarter traversal. Specify start and end nodes in graph: fuel node --node 1,2,3 --end netconfig Will deploy everything up to netconfig task, including netconfig. This is: all tasks that we are considering as pre_deployment (keys generation, rsync manifests, sync time, upload repos), and such tasks as hiera setup, globals computation and maybe some other basic preparatory tasks. fuel node --node 1,2,3 --start netconfig Start from netconfig, including netconfig, deploy all other tasks, tasks that we are considering as post_deployment. For example if one want to execute only netconfig successors: fuel node --node 1,2,3 --start netconfig --skip netconfig And user will be able to use start and end at the same time: fuel node --node 1,2,3 --start netconfig --end upload_cirros Nailgun will build path that includes only necessary tasks to join this two points. However start flag is not merged yet, but i think it will be by Monday. Also we are working on deployment graph visualization, it will be static (i mean there is no progress tracking of any kind), but it will help a lot to understand what is going to be executed. Thank you for reading, i would like to hear more thoughts about this, and answer any questions __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] CLI api for working with granular deployment model
Thank you for the excellent run-down of the CLI commands. I assume this will make its way into the developer documentation? I would like to know if you could point me to more information about the inner workings of granular deployment. Currently it's challenging to debug issues related to granular deployment. All tasks that are in scope of role are serialized right into deployment configuration that is consumed by astute. So it can be traced in the logs (nailgun or astute) or in astute.yaml that is stored on node itself. Here is what it looks like [0]. Some other internals described in spec - https://review.openstack.org/#/c/113491/. For developer it makes sense to get familiar with networkx data structures [1], and then dive into debuging of [2]. But it is not an option for a day-to-day usage, and UX will be improved by graph visualizer [3]. One more option that can improve understanding is human-readable planner.. For example it can output smth like this: fuel deployment plan --start hiera --end netconfig Manifest hiera.pp will be executed on nodes [1,2,3] Manifest netconfig will be executed on nodes [1,2] But i am not sure is this thing really necessary, dependencies are trivial in comparison to puppet, and i hope it will take very little time to understand how things are working :) As an example there is a bug [0] where tasks appear to be run in the wrong order based on which combination of roles exist in the environment. However, it's not clear how to determine what decides which tasks to run and when (is it astute, fuel-library, etc.), where the data comes from. etc. As for the bug - it may be a duplicate for https://launchpad.net/bugs/1417579, which was fixed by https://review.openstack.org/#/c/152511/ [0] http://paste.openstack.org/show/168298/ [1] http://networkx.github.io/documentation/latest/tutorial/tutorial.html#directed-graphs [2] https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/orchestrator/deployment_graph.py#L29 [3] https://review.openstack.org/#/c/152434/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] CLI api for working with granular deployment model
On Fri, Feb 6, 2015 at 11:16 PM, Mike Scherbakov mscherba...@mirantis.com wrote: Dmitry, thanks for sharing CLI options. I'd like to clarify a few things. Also very important to understand that if task is mapped to role controller, but node where you want to apply that task doesn't have this role - it wont be executed. Is there a particular reason why we want to restrict a user to run an arbitrary task on a server, even if server doesn't have a role assigned? I think we should be flexible here - if I'm hacking something, I'd like to run arbitrary things. The way I've seen this work so far is the missing role in the graph simply wont be executed, not the requested role fuel node --node 1,2,3 --end netconfig I would replace --end - --end-on, in order to show that task specified will run as well (to avoid ambiguity) This is separate question probably about CLI UX, but still - are we Ok with missing an action verb, like deploy? So it might be better to have, in my opinion: fuel deploy --node 1,2,3 --end netconfig provision and deploy tasks are already under node so it makes some sense to keep them here unless everything moves For example if one want to execute only netconfig successors: fuel node --node 1,2,3 --start netconfig --skip netconfig I would come up with shortcut for one task. To me, it would be way better to specify comma-separated tasks: fuel deploy --node 1,2,3 --task netconfig[,task2] this already appears to work as fuel node --node 1,2,3 --task netconfig compute Question here: if netconfig depends on other tasks, will those be executed prior to netconfig? I want both options, execute with prior deps, and execute just one particular task. Also we are working on deployment graph visualization yes, this would be awesome to have. When I specify --start and --end, I'll want to know what is going to be executed in reality, before it gets executed. So something like dry-run which shows execution graph would be very helpful. We need to start with a better graph of everything that will run on each node and in which order, I've yet to see something that renders the whole graph including the requested deps on each node. It will be very useful for debugging. As a separate note here, a few question: 1. If particular task fails to execute for some reason, what is the error handling? Will I be able to see puppet/deployment tool exception right in the same console, or should I check out some logs? We need to have perfect UX for errors. Those who will be using CLI to run particular tasks, will be dealing with errors for 95% of their time. The puppet based tasks run and should show errors the same as legacy deployments or plugin tasks 1. I'd love to have some guidance on slave node as well. For instance, I want to run just netconfig on slave node. How can I do it? fuel node --node 1 --tasks netconfig 1. If I stuck with error in task execution, which is in puppet. Can I modify puppet module on master node, and re-run the task? (assuming that updated module will be rsynced to slaves under deployment first) that's exactly how it works Thanks Dmitry! On Sat, Feb 7, 2015 at 12:16 AM, Dmitriy Shulyak dshul...@mirantis.com wrote: Thank you for the excellent run-down of the CLI commands. I assume this will make its way into the developer documentation? I would like to know if you could point me to more information about the inner workings of granular deployment. Currently it's challenging to debug issues related to granular deployment. All tasks that are in scope of role are serialized right into deployment configuration that is consumed by astute. So it can be traced in the logs (nailgun or astute) or in astute.yaml that is stored on node itself. Here is what it looks like [0]. Some other internals described in spec - https://review.openstack.org/#/c/113491/. For developer it makes sense to get familiar with networkx data structures [1], and then dive into debuging of [2]. But it is not an option for a day-to-day usage, and UX will be improved by graph visualizer [3]. One more option that can improve understanding is human-readable planner.. For example it can output smth like this: fuel deployment plan --start hiera --end netconfig Manifest hiera.pp will be executed on nodes [1,2,3] Manifest netconfig will be executed on nodes [1,2] But i am not sure is this thing really necessary, dependencies are trivial in comparison to puppet, and i hope it will take very little time to understand how things are working :) As an example there is a bug [0] where tasks appear to be run in the wrong order based on which combination of roles exist in the environment. However, it's not clear how to determine what decides which tasks to run and when (is it astute, fuel-library, etc.), where the data comes from. etc. As for the bug - it may be a duplicate for
Re: [openstack-dev] [Fuel] CLI api for working with granular deployment model
Dmitry, thanks for sharing CLI options. I'd like to clarify a few things. Also very important to understand that if task is mapped to role controller, but node where you want to apply that task doesn't have this role - it wont be executed. Is there a particular reason why we want to restrict a user to run an arbitrary task on a server, even if server doesn't have a role assigned? I think we should be flexible here - if I'm hacking something, I'd like to run arbitrary things. fuel node --node 1,2,3 --end netconfig I would replace --end - --end-on, in order to show that task specified will run as well (to avoid ambiguity) This is separate question probably about CLI UX, but still - are we Ok with missing an action verb, like deploy? So it might be better to have, in my opinion: fuel deploy --node 1,2,3 --end netconfig For example if one want to execute only netconfig successors: fuel node --node 1,2,3 --start netconfig --skip netconfig I would come up with shortcut for one task. To me, it would be way better to specify comma-separated tasks: fuel deploy --node 1,2,3 --task netconfig[,task2] Question here: if netconfig depends on other tasks, will those be executed prior to netconfig? I want both options, execute with prior deps, and execute just one particular task. Also we are working on deployment graph visualization yes, this would be awesome to have. When I specify --start and --end, I'll want to know what is going to be executed in reality, before it gets executed. So something like dry-run which shows execution graph would be very helpful. As a separate note here, a few question: 1. If particular task fails to execute for some reason, what is the error handling? Will I be able to see puppet/deployment tool exception right in the same console, or should I check out some logs? We need to have perfect UX for errors. Those who will be using CLI to run particular tasks, will be dealing with errors for 95% of their time. 2. I'd love to have some guidance on slave node as well. For instance, I want to run just netconfig on slave node. How can I do it? 3. If I stuck with error in task execution, which is in puppet. Can I modify puppet module on master node, and re-run the task? (assuming that updated module will be rsynced to slaves under deployment first) Thanks Dmitry! On Sat, Feb 7, 2015 at 12:16 AM, Dmitriy Shulyak dshul...@mirantis.com wrote: Thank you for the excellent run-down of the CLI commands. I assume this will make its way into the developer documentation? I would like to know if you could point me to more information about the inner workings of granular deployment. Currently it's challenging to debug issues related to granular deployment. All tasks that are in scope of role are serialized right into deployment configuration that is consumed by astute. So it can be traced in the logs (nailgun or astute) or in astute.yaml that is stored on node itself. Here is what it looks like [0]. Some other internals described in spec - https://review.openstack.org/#/c/113491/. For developer it makes sense to get familiar with networkx data structures [1], and then dive into debuging of [2]. But it is not an option for a day-to-day usage, and UX will be improved by graph visualizer [3]. One more option that can improve understanding is human-readable planner.. For example it can output smth like this: fuel deployment plan --start hiera --end netconfig Manifest hiera.pp will be executed on nodes [1,2,3] Manifest netconfig will be executed on nodes [1,2] But i am not sure is this thing really necessary, dependencies are trivial in comparison to puppet, and i hope it will take very little time to understand how things are working :) As an example there is a bug [0] where tasks appear to be run in the wrong order based on which combination of roles exist in the environment. However, it's not clear how to determine what decides which tasks to run and when (is it astute, fuel-library, etc.), where the data comes from. etc. As for the bug - it may be a duplicate for https://launchpad.net/bugs/1417579, which was fixed by https://review.openstack.org/#/c/152511/ [0] http://paste.openstack.org/show/168298/ [1] http://networkx.github.io/documentation/latest/tutorial/tutorial.html#directed-graphs [2] https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/orchestrator/deployment_graph.py#L29 [3] https://review.openstack.org/#/c/152434/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Mike Scherbakov #mihgen __ OpenStack Development Mailing List (not for usage questions) Unsubscribe:
[openstack-dev] [Fuel] CLI api for working with granular deployment model
Hello folks, Not long ago we added necessary commands in fuel client to work with granular deployment configuration and API. So, you may know that configuration is stored in fuel-library, and uploaded into database during bootstrap of fuel master. If you want to change/add some tasks right on master node, just add tasks.yaml and appropriate manifests in folder for release that you are interested in. Then apply this command: fuel rel --sync-deployment-tasks --dir /etc/puppet Also you may want to overwrite deployment tasks for any specific release/cluster by next commands: fuel rel --rel id --deployment-tasks --download fuel rel --rel id --deployment-tasks --upload fuel env --env id --deployment-tasks --download fuel env --env id --deployment-tasks --upload After this is done - you will be able to run customized graph of tasks: The most basic command: fuel node --node 1,2,3 --tasks upload_repos netconfig Developer will need to specify nodes that should be used in deployment and tasks ids. Order in which they are provided doesn't matter, it will be computed from dependencies specified in database. Also very important to understand that if task is mapped to role controller, but node where you want to apply that task doesn't have this role - it wont be executed. Skipping of tasks fuel node --node 1,2,3 --skip netconfig hiera List of task that are provided with this parameter will be skipped during graph traversal in nailgun. The main question is - should we skip other task that have provided tasks as dependencies? In my opinion we can leave this flag as simple as it is, and use following commands for smarter traversal. Specify start and end nodes in graph: fuel node --node 1,2,3 --end netconfig Will deploy everything up to netconfig task, including netconfig. This is: all tasks that we are considering as pre_deployment (keys generation, rsync manifests, sync time, upload repos), and such tasks as hiera setup, globals computation and maybe some other basic preparatory tasks. fuel node --node 1,2,3 --start netconfig Start from netconfig, including netconfig, deploy all other tasks, tasks that we are considering as post_deployment. For example if one want to execute only netconfig successors: fuel node --node 1,2,3 --start netconfig --skip netconfig And user will be able to use start and end at the same time: fuel node --node 1,2,3 --start netconfig --end upload_cirros Nailgun will build path that includes only necessary tasks to join this two points. However start flag is not merged yet, but i think it will be by Monday. Also we are working on deployment graph visualization, it will be static (i mean there is no progress tracking of any kind), but it will help a lot to understand what is going to be executed. Thank you for reading, i would like to hear more thoughts about this, and answer any questions __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
