[Yahoo-eng-team] [Bug 1319232] Re: Periodic tasks run too frequently
Cinder was fixed back in Juno: https://review.openstack.org/#/c/96512/ ** Changed in: cinder Status: In Progress = Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1319232 Title: Periodic tasks run too frequently Status in Cinder: Fix Released Status in neutron: Fix Released Status in OpenStack Compute (nova): Fix Released Status in oslo-incubator: Fix Released Bug description: Each periodic task can have a spacing, which defines the minimum amount of time between executions of that task. For example, a task with periodic_spacing=120 would execute no more often than once every 2 minutes. Tasks that do not define an explicit spacing will be run every time the periodic task processor runs. This is commonly loosely interpreted as every 60 seconds, but in reality it's more complicated than that. As a result of these complications, we can actually end up running these tasks more frequently -- I've regularly observed them running every 20-30 seconds, and in several cases I've seen a task running just 1-2 seconds after it previously ran. This consumes extra resources (CPU, database access, etc) without providing any real value. The reason for these extra runs has to do with how the periodic task processor is implemented. When there are multiple tasks with a defined spacing, they can get somewhat staggered and force the periodic task processor to run additional iterations. Since tasks with no spacing run every time the periodic task processor runs, they get run more frequently than one would expect. My proposed solution is to redefine the behavior of periodic tasks with no explicit spacing so that they run with the default interval (60 seconds). The code change is simple -- in nova/openstack/common/periodic_task.py, change this code: # A periodic spacing of zero indicates that this task should # be run every pass if task._periodic_spacing == 0: task._periodic_spacing = None to: # A periodic spacing of zero indicates that this task should # be run at the default interval if task._periodic_spacing == 0: task._periodic_spacing = DEFAULT_INTERVAL The actual runtime task processing code doesn't change -- this fix is basically the equivalent of finding every @periodic_task that doesn't have an explicit spacing, and setting spacing=60. So it's very low risk. Some may argue that this change in behavior could cause some task to behave differently than it used to. However, there was never any guarantee that the task would run more often than every 60 seconds, and in many cases the tasks may already run less frequently than that (due to other long-running tasks). So this change should not introduce any new issues related to the timing of task execution; it would only serve to make the timing more regular. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1319232/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1319232] Re: Periodic tasks run too frequently
** Changed in: neutron Status: Fix Committed = Fix Released ** Changed in: neutron Milestone: None = juno-1 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1319232 Title: Periodic tasks run too frequently Status in Cinder: In Progress Status in OpenStack Neutron (virtual network service): Fix Released Status in OpenStack Compute (Nova): Fix Released Status in Oslo - a Library of Common OpenStack Code: Fix Released Bug description: Each periodic task can have a spacing, which defines the minimum amount of time between executions of that task. For example, a task with periodic_spacing=120 would execute no more often than once every 2 minutes. Tasks that do not define an explicit spacing will be run every time the periodic task processor runs. This is commonly loosely interpreted as every 60 seconds, but in reality it's more complicated than that. As a result of these complications, we can actually end up running these tasks more frequently -- I've regularly observed them running every 20-30 seconds, and in several cases I've seen a task running just 1-2 seconds after it previously ran. This consumes extra resources (CPU, database access, etc) without providing any real value. The reason for these extra runs has to do with how the periodic task processor is implemented. When there are multiple tasks with a defined spacing, they can get somewhat staggered and force the periodic task processor to run additional iterations. Since tasks with no spacing run every time the periodic task processor runs, they get run more frequently than one would expect. My proposed solution is to redefine the behavior of periodic tasks with no explicit spacing so that they run with the default interval (60 seconds). The code change is simple -- in nova/openstack/common/periodic_task.py, change this code: # A periodic spacing of zero indicates that this task should # be run every pass if task._periodic_spacing == 0: task._periodic_spacing = None to: # A periodic spacing of zero indicates that this task should # be run at the default interval if task._periodic_spacing == 0: task._periodic_spacing = DEFAULT_INTERVAL The actual runtime task processing code doesn't change -- this fix is basically the equivalent of finding every @periodic_task that doesn't have an explicit spacing, and setting spacing=60. So it's very low risk. Some may argue that this change in behavior could cause some task to behave differently than it used to. However, there was never any guarantee that the task would run more often than every 60 seconds, and in many cases the tasks may already run less frequently than that (due to other long-running tasks). So this change should not introduce any new issues related to the timing of task execution; it would only serve to make the timing more regular. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1319232/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1319232] Re: Periodic tasks run too frequently
** Changed in: oslo Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1319232 Title: Periodic tasks run too frequently Status in Cinder: In Progress Status in OpenStack Neutron (virtual network service): Fix Released Status in OpenStack Compute (Nova): Fix Released Status in Oslo - a Library of Common OpenStack Code: Fix Released Bug description: Each periodic task can have a spacing, which defines the minimum amount of time between executions of that task. For example, a task with periodic_spacing=120 would execute no more often than once every 2 minutes. Tasks that do not define an explicit spacing will be run every time the periodic task processor runs. This is commonly loosely interpreted as every 60 seconds, but in reality it's more complicated than that. As a result of these complications, we can actually end up running these tasks more frequently -- I've regularly observed them running every 20-30 seconds, and in several cases I've seen a task running just 1-2 seconds after it previously ran. This consumes extra resources (CPU, database access, etc) without providing any real value. The reason for these extra runs has to do with how the periodic task processor is implemented. When there are multiple tasks with a defined spacing, they can get somewhat staggered and force the periodic task processor to run additional iterations. Since tasks with no spacing run every time the periodic task processor runs, they get run more frequently than one would expect. My proposed solution is to redefine the behavior of periodic tasks with no explicit spacing so that they run with the default interval (60 seconds). The code change is simple -- in nova/openstack/common/periodic_task.py, change this code: # A periodic spacing of zero indicates that this task should # be run every pass if task._periodic_spacing == 0: task._periodic_spacing = None to: # A periodic spacing of zero indicates that this task should # be run at the default interval if task._periodic_spacing == 0: task._periodic_spacing = DEFAULT_INTERVAL The actual runtime task processing code doesn't change -- this fix is basically the equivalent of finding every @periodic_task that doesn't have an explicit spacing, and setting spacing=60. So it's very low risk. Some may argue that this change in behavior could cause some task to behave differently than it used to. However, there was never any guarantee that the task would run more often than every 60 seconds, and in many cases the tasks may already run less frequently than that (due to other long-running tasks). So this change should not introduce any new issues related to the timing of task execution; it would only serve to make the timing more regular. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1319232/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1319232] Re: Periodic tasks run too frequently
** Changed in: nova Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1319232 Title: Periodic tasks run too frequently Status in Cinder: In Progress Status in OpenStack Neutron (virtual network service): Fix Committed Status in OpenStack Compute (Nova): Fix Released Status in Oslo - a Library of Common OpenStack Code: Fix Committed Bug description: Each periodic task can have a spacing, which defines the minimum amount of time between executions of that task. For example, a task with periodic_spacing=120 would execute no more often than once every 2 minutes. Tasks that do not define an explicit spacing will be run every time the periodic task processor runs. This is commonly loosely interpreted as every 60 seconds, but in reality it's more complicated than that. As a result of these complications, we can actually end up running these tasks more frequently -- I've regularly observed them running every 20-30 seconds, and in several cases I've seen a task running just 1-2 seconds after it previously ran. This consumes extra resources (CPU, database access, etc) without providing any real value. The reason for these extra runs has to do with how the periodic task processor is implemented. When there are multiple tasks with a defined spacing, they can get somewhat staggered and force the periodic task processor to run additional iterations. Since tasks with no spacing run every time the periodic task processor runs, they get run more frequently than one would expect. My proposed solution is to redefine the behavior of periodic tasks with no explicit spacing so that they run with the default interval (60 seconds). The code change is simple -- in nova/openstack/common/periodic_task.py, change this code: # A periodic spacing of zero indicates that this task should # be run every pass if task._periodic_spacing == 0: task._periodic_spacing = None to: # A periodic spacing of zero indicates that this task should # be run at the default interval if task._periodic_spacing == 0: task._periodic_spacing = DEFAULT_INTERVAL The actual runtime task processing code doesn't change -- this fix is basically the equivalent of finding every @periodic_task that doesn't have an explicit spacing, and setting spacing=60. So it's very low risk. Some may argue that this change in behavior could cause some task to behave differently than it used to. However, there was never any guarantee that the task would run more often than every 60 seconds, and in many cases the tasks may already run less frequently than that (due to other long-running tasks). So this change should not introduce any new issues related to the timing of task execution; it would only serve to make the timing more regular. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1319232/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1319232] Re: Periodic tasks run too frequently
There was also some related discussion in the mailing list here: https://www.mail-archive.com/openstack- dev%40lists.openstack.org/msg15922.html ** Also affects: oslo Importance: Undecided Status: New ** Tags added: compute oslo ** Changed in: nova Status: New = Confirmed ** Changed in: nova Importance: Undecided = High ** Changed in: nova Importance: High = Medium -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1319232 Title: Periodic tasks run too frequently Status in OpenStack Compute (Nova): Confirmed Status in Oslo - a Library of Common OpenStack Code: New Bug description: Each periodic task can have a spacing, which defines the minimum amount of time between executions of that task. For example, a task with periodic_spacing=120 would execute no more often than once every 2 minutes. Tasks that do not define an explicit spacing will be run every time the periodic task processor runs. This is commonly loosely interpreted as every 60 seconds, but in reality it's more complicated than that. As a result of these complications, we can actually end up running these tasks more frequently -- I've regularly observed them running every 20-30 seconds, and in several cases I've seen a task running just 1-2 seconds after it previously ran. This consumes extra resources (CPU, database access, etc) without providing any real value. The reason for these extra runs has to do with how the periodic task processor is implemented. When there are multiple tasks with a defined spacing, they can get somewhat staggered and force the periodic task processor to run additional iterations. Since tasks with no spacing run every time the periodic task processor runs, they get run more frequently than one would expect. My proposed solution is to redefine the behavior of periodic tasks with no explicit spacing so that they run with the default interval (60 seconds). The code change is simple -- in nova/openstack/common/periodic_task.py, change this code: # A periodic spacing of zero indicates that this task should # be run every pass if task._periodic_spacing == 0: task._periodic_spacing = None to: # A periodic spacing of zero indicates that this task should # be run at the default interval if task._periodic_spacing == 0: task._periodic_spacing = DEFAULT_INTERVAL The actual runtime task processing code doesn't change -- this fix is basically the equivalent of finding every @periodic_task that doesn't have an explicit spacing, and setting spacing=60. So it's very low risk. Some may argue that this change in behavior could cause some task to behave differently than it used to. However, there was never any guarantee that the task would run more often than every 60 seconds, and in many cases the tasks may already run less frequently than that (due to other long-running tasks). So this change should not introduce any new issues related to the timing of task execution; it would only serve to make the timing more regular. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1319232/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
