On Mon, Feb 25, 2013 at 6:14 PM, Sam Morrison <sorri...@gmail.com> wrote:
> Hi Joe, > > On 26/02/2013, at 11:19 AM, Joe Gordon <j...@cloudscaling.com> wrote: > > On Sun, Feb 24, 2013 at 3:31 PM, Sam Morrison <sorri...@gmail.com> wrote: > >> I have been playing with the AggregateInstanceExtraSpecs filter and can't >> get it to work. >> >> In our staging environment it works fine with 4 compute nodes, I have 2 >> aggregates to split them into 2. >> >> When I try to do the same in our production environment which has 80 >> compute nodes (splitting them again into 2 aggregates) it doesn't work. >> >> nova-scheduler starts to go very slow, I scheduled an instance and gave >> up after 5 minutes, it seemed to be taking ages and the host was at 100% >> cpu. Also got about 500 messages in rabbit that were unacknowledged. >> >> > what does the nova-scheduler log say? Where is the unacknowledged > rabbitmq messages sent from? > > > Logs are below. Note the large time gap between selecting a host, this is > pretty much instantaneous without this filter. > > Can't figure out how to see an unacknowledged message in rabbit but my > guess is it is the compute service updates from all the compute nodes. > These aren't happening and I think this is the reason that the attempts to > schedule further down are rejected with "is disabled or has not been heard > from in a while" > > Do you see anything that could be an issue? Flags we use for scheduler are > below also: > > Thanks for your help, > Sam > > > # Scheduler Flags > compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler > ram_allocation_ratio=1.0 > cpu_allocation_ratio=0.92 > reserved_host_memory_mb=1024 > reserved_host_disk_mb=0 > > scheduler_default_filters=RetryFilter,AggregateInstanceExtraSpecsFilter,RamFilter,CoreFilter,ComputeFilter > compute_fill_first_cost_fn_weight=1.0 > > > > 2013-02-25 10:01:35 DEBUG nova.scheduler.filter_scheduler > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Attempting to build 1 > instance(s) schedule_run_instance /usr/lib/python2.7/dist-packages/nova/sc > heduler/filter_scheduler.py:66 > 2013-02-25 10:01:35 DEBUG nova.scheduler.filters.retry_filter > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: > []. (host=qh2-rcc27) host_passes /usr/lib/python2.7/dist-packages/n > ova/scheduler/filters/retry_filter.py:39 > 2013-02-25 10:02:13 DEBUG nova.scheduler.host_manager > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter passes for > qh2-rcc27 passes_filters > /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:178 > 2013-02-25 10:02:13 DEBUG nova.scheduler.filters.retry_filter > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: > []. (host=qh2-rcc26) host_passes > /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39 > 2013-02-25 10:02:51 DEBUG nova.scheduler.host_manager > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function > <bound method CoreFilter.host_passes of > <nova.scheduler.filters.core_filter.CoreFilter object at 0x43f7a50>> failed > for qh2-rcc26 passes_filters > /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175 > 2013-02-25 10:02:51 DEBUG nova.scheduler.filters.retry_filter > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: > []. (host=qh2-rcc25) host_passes > /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39 > 2013-02-25 10:03:28 DEBUG nova.scheduler.filters.compute_filter > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc25': > free_ram_mb:71086 free_disk_mb:3035136 is disabled or has not been heard > from in a while host_passes > /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37 > 2013-02-25 10:03:28 DEBUG nova.scheduler.host_manager > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function > <bound method ComputeFilter.host_passes of > <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>> > failed for qh2-rcc25 passes_filters > /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175 > 2013-02-25 10:03:28 DEBUG nova.scheduler.filters.retry_filter > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: > []. (host=qh2-rcc24) host_passes > /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39 > 2013-02-25 10:04:05 DEBUG nova.scheduler.filters.compute_filter > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc24': > free_ram_mb:99758 free_disk_mb:3296256 is disabled or has not been heard > from in a while host_passes > /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37 > 2013-02-25 10:04:05 DEBUG nova.scheduler.host_manager > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function > <bound method ComputeFilter.host_passes of > <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>> > failed for qh2-rcc24 passes_filters > /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175 > 2013-02-25 10:04:05 DEBUG nova.scheduler.filters.retry_filter > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: > []. (host=qh2-rcc23) host_passes > /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39 > 2013-02-25 10:04:43 DEBUG nova.scheduler.filters.compute_filter > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc23': > free_ram_mb:71086 free_disk_mb:3012608 is disabled or has not been heard > from in a while host_passes > /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37 > 2013-02-25 10:04:43 DEBUG nova.scheduler.host_manager > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function > <bound method ComputeFilter.host_passes of > <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>> > failed for qh2-rcc23 passes_filters > /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175 > 2013-02-25 10:04:43 DEBUG nova.scheduler.filters.retry_filter > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: > []. (host=qh2-rcc22) host_passes > /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39 > 2013-02-25 10:05:22 DEBUG nova.scheduler.filters.compute_filter > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc22': > free_ram_mb:91566 free_disk_mb:3196928 is disabled or has not been heard > from in a while host_passes > /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37 > 2013-02-25 10:05:22 DEBUG nova.scheduler.host_manager > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function > <bound method ComputeFilter.host_passes of > <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>> > failed for qh2-rcc22 passes_filters > /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175 > 2013-02-25 10:05:22 DEBUG nova.scheduler.filters.retry_filter > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: > []. (host=qh2-rcc21) host_passes > /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39 > 2013-02-25 10:06:01 DEBUG nova.scheduler.filters.compute_filter > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc21': > free_ram_mb:91566 free_disk_mb:3216384 is disabled or has not been heard > from in a while host_passes > /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37 > 2013-02-25 10:06:01 DEBUG nova.scheduler.host_manager > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function > <bound method ComputeFilter.host_passes of > <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>> > failed for qh2-rcc21 passes_filters > /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175 > 2013-02-25 10:06:01 DEBUG nova.scheduler.filters.retry_filter > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: > []. (host=qh2-rcc20) host_passes > /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39 > 2013-02-25 10:06:37 DEBUG nova.scheduler.host_manager > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function > <bound method CoreFilter.host_passes of > <nova.scheduler.filters.core_filter.CoreFilter object at 0x43f7a50>> failed > for qh2-rcc20 passes_filters > /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175 > 2013-02-25 10:06:37 DEBUG nova.scheduler.filters.retry_filter > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: > []. (host=qh2-rcc77) host_passes > /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39 > 2013-02-25 10:07:14 DEBUG nova.scheduler.filters.compute_filter > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc77': > free_ram_mb:75182 free_disk_mb:3059712 is disabled or has not been heard > from in a while host_passes > /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37 > It looks like the scheduler issues are related to the rabbitmq issues. "host 'qh2-rcc77' ... is disabled or has not been heard from in a while" What does 'nova host-list' say? the clocks must all be synced up? > 2013-02-25 10:07:14 DEBUG nova.scheduler.host_manager > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Host filter function > <bound method ComputeFilter.host_passes of > <nova.scheduler.filters.compute_filter.ComputeFilter object at 0x43f7210>> > failed for qh2-rcc77 passes_filters > /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:175 > 2013-02-25 10:07:14 DEBUG nova.scheduler.filters.retry_filter > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] Previously tried hosts: > []. (host=qh2-rcc73) host_passes > /usr/lib/python2.7/dist-packages/nova/scheduler/filters/retry_filter.py:39 > 2013-02-25 10:07:52 DEBUG nova.scheduler.filters.compute_filter > [req-d7c77ff6-353a-409a-b32c-68627c1d1bb0 25 23] host 'qh2-rcc73': > free_ram_mb:87470 free_disk_mb:3148800 is disabled or has not been heard > from in a while host_passes > /usr/lib/python2.7/dist-packages/nova/scheduler/filters/compute_filter.py:37 > > >
_______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp