Re: [openstack-dev] [openstack-ansible] To NTP, or not to NTP, that is the question
On 22 September 2015 at 01:45, Major Hayden wrote: > On 09/21/2015 07:14 PM, Sergii Golovatiuk wrote: > > Are any chance to configure chrony instead of ntpd? It acts more > predictable on virtual environments. > > That's my plan, if I can find an upstream Ansible galaxy role to use. ;) > Now that we have the spec for independent role repositories approved [1], an option is for us to register a role which implements chrony as the network time mechanism if there isn't a suitable one already in Ansible Galaxy. This role can be an optional add-on to the supported use-cases in OpenStack-Ansible and can also be registered in Ansible Galaxy once it's ready. :) [1] https://review.openstack.org/213779 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [openstack-ansible] To NTP, or not to NTP, that is the question
On 09/21/2015 07:14 PM, Sergii Golovatiuk wrote: > Are any chance to configure chrony instead of ntpd? It acts more predictable > on virtual environments. That's my plan, if I can find an upstream Ansible galaxy role to use. ;) -- Major Hayden __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [openstack-ansible] To NTP, or not to NTP, that is the question
Hi, Are any chance to configure chrony instead of ntpd? It acts more predictable on virtual environments. -- Best regards, Sergii Golovatiuk, Skype #golserge IRC #holser On Mon, Sep 21, 2015 at 4:11 PM, Jesse Pretorius wrote: > On 18 September 2015 at 14:03, Major Hayden wrote: > >> Hey there, >> >> I start working on a bug[1] last night about adding a managed NTP >> configuration to openstack-ansible hosts. My patch[2] gets chrony up and >> running with configurable NTP servers, but I'm still struggling to meet the >> "Proposal" section of the bug where the author has asked for non-infra >> physical nodes to get their time from the infra nodes. I can't figure out >> how to make it work for AIO builds when one physical host is part of all of >> the groups. ;) >> >> I'd argue that time synchronization is critical for a few areas: >> >> 1) Security/auditing when comparing logs >> 2) Troubleshooting when comparing logs >> 3) I've been told swift is time-sensitive >> 4) MySQL/Galera don't like time drift >> >> However, there's a strong argument that this should be done by deployers, >> and not via openstack-ansible. I'm still *very* new to the project and I'd >> like to hear some feedback from other folks. >> >> [1] https://bugs.launchpad.net/openstack-ansible/+bug/1413018 >> [2] https://review.openstack.org/#/c/225006/ > > > We have historically taken the stance of leaving something like this as a > deployer concern - much like setting up host networking and setting host > repositories. That said, there's value in opinionation based on best > practices learned from hard-won lessons in the trenches. > > I'm somewhat on the fence with this. As-is I don't think the review should > go in. That said, I'd be more open to an individual role being used to > implement an appropriate network time configuration - whether that role be > something that exists within Ansible Galaxy, or whether it's a new role in > the current repository, or as its own repository in the OpenStack-Ansible > 'big tent' as proposed in https://review.openstack.org/213779 > > I do definitely think that there's value in preparing some documentation > which will help prospective deployers understand how they can consume roles > from Ansible Galaxy (or some role in an arbitrary repository) to solve > common problems like this. The tooling is already in the OpenStack-Ansible > repository, so all it needs is a guiding document which describes how to > use it. > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [openstack-ansible] To NTP, or not to NTP, that is the question
On 18 September 2015 at 14:03, Major Hayden wrote: > Hey there, > > I start working on a bug[1] last night about adding a managed NTP > configuration to openstack-ansible hosts. My patch[2] gets chrony up and > running with configurable NTP servers, but I'm still struggling to meet the > "Proposal" section of the bug where the author has asked for non-infra > physical nodes to get their time from the infra nodes. I can't figure out > how to make it work for AIO builds when one physical host is part of all of > the groups. ;) > > I'd argue that time synchronization is critical for a few areas: > > 1) Security/auditing when comparing logs > 2) Troubleshooting when comparing logs > 3) I've been told swift is time-sensitive > 4) MySQL/Galera don't like time drift > > However, there's a strong argument that this should be done by deployers, > and not via openstack-ansible. I'm still *very* new to the project and I'd > like to hear some feedback from other folks. > > [1] https://bugs.launchpad.net/openstack-ansible/+bug/1413018 > [2] https://review.openstack.org/#/c/225006/ We have historically taken the stance of leaving something like this as a deployer concern - much like setting up host networking and setting host repositories. That said, there's value in opinionation based on best practices learned from hard-won lessons in the trenches. I'm somewhat on the fence with this. As-is I don't think the review should go in. That said, I'd be more open to an individual role being used to implement an appropriate network time configuration - whether that role be something that exists within Ansible Galaxy, or whether it's a new role in the current repository, or as its own repository in the OpenStack-Ansible 'big tent' as proposed in https://review.openstack.org/213779 I do definitely think that there's value in preparing some documentation which will help prospective deployers understand how they can consume roles from Ansible Galaxy (or some role in an arbitrary repository) to solve common problems like this. The tooling is already in the OpenStack-Ansible repository, so all it needs is a guiding document which describes how to use it. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [openstack-ansible] To NTP, or not to NTP, that is the question
> On Sep 18, 2015, at 9:38 AM, Jay Pipes wrote: > >> On 09/18/2015 11:04 AM, Ian Cordasco wrote: >>> On 9/18/15, 08:03, "Major Hayden" wrote: >>> >>> Hey there, >>> >>> I start working on a bug[1] last night about adding a managed NTP >>> configuration to openstack-ansible hosts. My patch[2] gets chrony up and >>> running with configurable NTP servers, but I'm still struggling to meet >>> the "Proposal" section of the bug where the author has asked for >>> non-infra physical nodes to get their time from the infra nodes. I can't >>> figure out how to make it work for AIO builds when one physical host is >>> part of all of the groups. ;) >>> >>> I'd argue that time synchronization is critical for a few areas: >>> >>> 1) Security/auditing when comparing logs >>> 2) Troubleshooting when comparing logs >>> 3) I've been told swift is time-sensitive >>> 4) MySQL/Galera don't like time drift >>> >>> However, there's a strong argument that this should be done by deployers, >>> and not via openstack-ansible. I'm still *very* new to the project and >>> I'd like to hear some feedback from other folks. >> >> Personally, I fall into the camp of "this is a deployer concern". >> Specifically, there is already an ansible-galaxy role to enable NTP on >> your deployment hosts (https://galaxy.ansible.com/list#/roles/464) which >> *could* be expanded to do this very work that you're talking about. Using >> specialized roles to achieve this (and contributing back to the larger >> ansible community) seems like a bigger win than trying to reimplement some >> of this in OSA instead of reusing other roles that already exist. >> >> Compare it to a hypothetical situation where Keystone wrote its own >> backing libraries to implement Fernet instead of using the cryptography >> library. In that case there would be absolutely no argument that Keystone >> should use cryptography (even if it uses cffi and has bindings to OpenSSL >> which our infra team doesn't like and some deployers find difficult to >> manage when using pure-python deployment tooling). Why should OSA be any >> different from another OpenStack project? > > Have to agree with Ian here. NTP, as Major wrote, is a critical piece of the > deployment puzzle, but I don't think it's necessary to put anything in OSA > specifically to configure NTP. As Ian wrote, better to contribute to upstream > ansible-galaxy playbooks/roles that do this well. I have a nuanced agreement with this which borders on disagreement. An agreed-upon time tick is as crucial to a distributed system as oxygen is to a human. It's not only those components that care, it's the humans who have to understand and operate it. As such, an OpenStack cloud should come with a time source that all services listen to; even if it's wildly off from the real world, the value of all services sharing the same tick is immeasurable. For me, it's part of "batteries included." I'd argue that we should pick a tool and configuration for this by default and allow others to change it. And, while I love Major*, I don't think the deployment tools are the right place for this. --j * and I do. Been too long, Major. We should fix that. =] __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [openstack-ansible] To NTP, or not to NTP, that is the question
On 09/18/2015 11:04 AM, Ian Cordasco wrote: On 9/18/15, 08:03, "Major Hayden" wrote: Hey there, I start working on a bug[1] last night about adding a managed NTP configuration to openstack-ansible hosts. My patch[2] gets chrony up and running with configurable NTP servers, but I'm still struggling to meet the "Proposal" section of the bug where the author has asked for non-infra physical nodes to get their time from the infra nodes. I can't figure out how to make it work for AIO builds when one physical host is part of all of the groups. ;) I'd argue that time synchronization is critical for a few areas: 1) Security/auditing when comparing logs 2) Troubleshooting when comparing logs 3) I've been told swift is time-sensitive 4) MySQL/Galera don't like time drift However, there's a strong argument that this should be done by deployers, and not via openstack-ansible. I'm still *very* new to the project and I'd like to hear some feedback from other folks. Personally, I fall into the camp of "this is a deployer concern". Specifically, there is already an ansible-galaxy role to enable NTP on your deployment hosts (https://galaxy.ansible.com/list#/roles/464) which *could* be expanded to do this very work that you're talking about. Using specialized roles to achieve this (and contributing back to the larger ansible community) seems like a bigger win than trying to reimplement some of this in OSA instead of reusing other roles that already exist. Compare it to a hypothetical situation where Keystone wrote its own backing libraries to implement Fernet instead of using the cryptography library. In that case there would be absolutely no argument that Keystone should use cryptography (even if it uses cffi and has bindings to OpenSSL which our infra team doesn't like and some deployers find difficult to manage when using pure-python deployment tooling). Why should OSA be any different from another OpenStack project? Have to agree with Ian here. NTP, as Major wrote, is a critical piece of the deployment puzzle, but I don't think it's necessary to put anything in OSA specifically to configure NTP. As Ian wrote, better to contribute to upstream ansible-galaxy playbooks/roles that do this well. Best, -jay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [openstack-ansible] To NTP, or not to NTP, that is the question
My $0.02 Support configuring ntp. Have a flag that can turn that piece off. Default it on. Profit. :) Thanks, Kevin From: Major Hayden [ma...@mhtx.net] Sent: Friday, September 18, 2015 6:03 AM To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [openstack-ansible] To NTP, or not to NTP, that is the question Hey there, I start working on a bug[1] last night about adding a managed NTP configuration to openstack-ansible hosts. My patch[2] gets chrony up and running with configurable NTP servers, but I'm still struggling to meet the "Proposal" section of the bug where the author has asked for non-infra physical nodes to get their time from the infra nodes. I can't figure out how to make it work for AIO builds when one physical host is part of all of the groups. ;) I'd argue that time synchronization is critical for a few areas: 1) Security/auditing when comparing logs 2) Troubleshooting when comparing logs 3) I've been told swift is time-sensitive 4) MySQL/Galera don't like time drift However, there's a strong argument that this should be done by deployers, and not via openstack-ansible. I'm still *very* new to the project and I'd like to hear some feedback from other folks. [1] https://bugs.launchpad.net/openstack-ansible/+bug/1413018 [2] https://review.openstack.org/#/c/225006/ -- Major Hayden __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [openstack-ansible] To NTP, or not to NTP, that is the question
On 9/18/15, 08:03, "Major Hayden" wrote: >Hey there, > >I start working on a bug[1] last night about adding a managed NTP >configuration to openstack-ansible hosts. My patch[2] gets chrony up and >running with configurable NTP servers, but I'm still struggling to meet >the "Proposal" section of the bug where the author has asked for >non-infra physical nodes to get their time from the infra nodes. I can't >figure out how to make it work for AIO builds when one physical host is >part of all of the groups. ;) > >I'd argue that time synchronization is critical for a few areas: > > 1) Security/auditing when comparing logs > 2) Troubleshooting when comparing logs > 3) I've been told swift is time-sensitive > 4) MySQL/Galera don't like time drift > >However, there's a strong argument that this should be done by deployers, >and not via openstack-ansible. I'm still *very* new to the project and >I'd like to hear some feedback from other folks. Personally, I fall into the camp of "this is a deployer concern". Specifically, there is already an ansible-galaxy role to enable NTP on your deployment hosts (https://galaxy.ansible.com/list#/roles/464) which *could* be expanded to do this very work that you're talking about. Using specialized roles to achieve this (and contributing back to the larger ansible community) seems like a bigger win than trying to reimplement some of this in OSA instead of reusing other roles that already exist. Compare it to a hypothetical situation where Keystone wrote its own backing libraries to implement Fernet instead of using the cryptography library. In that case there would be absolutely no argument that Keystone should use cryptography (even if it uses cffi and has bindings to OpenSSL which our infra team doesn't like and some deployers find difficult to manage when using pure-python deployment tooling). Why should OSA be any different from another OpenStack project? Cheers, Ian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [openstack-ansible] To NTP, or not to NTP, that is the question
Major, in Fuel, we've dealt with this problem for a long time in its varying degrees of unpleasantness. Some virtualization platforms, such as VirtualBox, are very prone to time drift. Hardware nodes, thankfully, don't suffer so badly. Time sync is very important for RabbitMQ, Corosync, and Ceph, in addition to those items you mentioned above. I haven't seen swift itself break due to time issues, but you may be right. The most ideal situation is to point all hosts to public NTP pool servers. Barring that, elect 1 host to base its time by its hardware clock, and then direct all other hosts to sync time against that one host. This has major issues when you're doing virtual deployments with snapshot/revert and experiencing major time skew, so you may need extra VM management scripts to manually sync time again after revert. Best Regards, Matthew Mosesohn On Fri, Sep 18, 2015 at 4:03 PM, Major Hayden wrote: > Hey there, > > I start working on a bug[1] last night about adding a managed NTP > configuration to openstack-ansible hosts. My patch[2] gets chrony up and > running with configurable NTP servers, but I'm still struggling to meet the > "Proposal" section of the bug where the author has asked for non-infra > physical nodes to get their time from the infra nodes. I can't figure out > how to make it work for AIO builds when one physical host is part of all of > the groups. ;) > > I'd argue that time synchronization is critical for a few areas: > > 1) Security/auditing when comparing logs > 2) Troubleshooting when comparing logs > 3) I've been told swift is time-sensitive > 4) MySQL/Galera don't like time drift > > However, there's a strong argument that this should be done by deployers, > and not via openstack-ansible. I'm still *very* new to the project and I'd > like to hear some feedback from other folks. > > [1] https://bugs.launchpad.net/openstack-ansible/+bug/1413018 > [2] https://review.openstack.org/#/c/225006/ > > -- > Major Hayden > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [openstack-ansible] To NTP, or not to NTP, that is the question
Hey there, I start working on a bug[1] last night about adding a managed NTP configuration to openstack-ansible hosts. My patch[2] gets chrony up and running with configurable NTP servers, but I'm still struggling to meet the "Proposal" section of the bug where the author has asked for non-infra physical nodes to get their time from the infra nodes. I can't figure out how to make it work for AIO builds when one physical host is part of all of the groups. ;) I'd argue that time synchronization is critical for a few areas: 1) Security/auditing when comparing logs 2) Troubleshooting when comparing logs 3) I've been told swift is time-sensitive 4) MySQL/Galera don't like time drift However, there's a strong argument that this should be done by deployers, and not via openstack-ansible. I'm still *very* new to the project and I'd like to hear some feedback from other folks. [1] https://bugs.launchpad.net/openstack-ansible/+bug/1413018 [2] https://review.openstack.org/#/c/225006/ -- Major Hayden __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev