FYI ... I tried updating /etc/masakari/masakari.conf with [nova] nova_catalog_admin_info = compute:nova:publicURL and restarting masakari-engine and masakari-api ... but that didn’t work ... not sure why.
So I manually changed the default nova_catalog_admin_info to ‘compute:nova:publicURL’ ... in masakari/masakari/conf/nova.py and restarted masakari-engine ... and YAY !!!! ... my usecase demo of masakari non-intrusive instance monitoring worked !!!!. i.e. masakari-instancemonitor detected and reported failed VM, masakari-engine recovered the VM automatically. See below: stack@devstack-masakari-louie:~$ nova list +--------------------------------------+-------------+--------+------------+-------------+--------------------------------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------------+--------+------------+-------------+--------------------------------------------------------+ | 00f377e6-e21f-431a-aba9-31ce13fd974c | vm-1-cirros | ACTIVE | - | Running | private=fd37:1afb:1393:0:f816:3eff:fec9:7f48, 10.0.0.7 | +--------------------------------------+-------------+--------+------------+-------------+--------------------------------------------------------+ stack@devstack-masakari-louie:~$ masakari notification-list +--------------------------------------+----------------------------+---------+--------------------------------------+------+ | notification_uuid | generated_time | status | source_host_uuid | type | +--------------------------------------+----------------------------+---------+--------------------------------------+------+ | 5b535d99-4e02-44a5-bd21-d131d14aaa36 | 2018-01-30T12:10:43.000000 | running | 51bc8b8b-324f-499a-9166-38c22b3842cd | VM | | b8c6c561-7a93-40a2-8d73-3783024865b4 | 2018-01-26T19:41:29.000000 | running | 51bc8b8b-324f-499a-9166-38c22b3842cd | VM | | ed6433c3-939d-4aa8-bf47-3ce8e8c78d45 | 2018-01-26T17:13:03.000000 | running | 51bc8b8b-324f-499a-9166-38c22b3842cd | VM | +--------------------------------------+----------------------------+---------+--------------------------------------+------+ stack@devstack-masakari-louie:~$ stack@devstack-masakari-louie:~$ !ps ps -ef | fgrep qemu libvirt+ 8113 1 2 12:12 ? 00:00:14 /usr/bin/qemu-system-x86_64 -name guest=instance-00000004,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-4-instance-00000004/master-key.aes -machine pc-i440fx-artful,accel=tcg,usb=off,dump-guest-core=off -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 00f377e6-e21f-431a-aba9-31ce13fd974c -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=17.0.0,serial=dacbc8a8-47c5-4132-86e3-3c902df4cf15,uuid=00f377e6-e21f-431a-aba9-31ce13fd974c,family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-4-instance-00000004/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/opt/stack/data/nova/instances/00f377e6-e21f-431a-aba9-31ce13fd974c/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=80,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:c9:7f:48,bus=pci.0,addr=0x3 -add-fd set=1,fd=83 -chardev pty,id=charserial0,logfile=/dev/fdset/1,logappend=on -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on stack 9195 5888 0 12:23 pts/0 00:00:00 grep -F --color=auto qemu stack@devstack-masakari-louie:~$ stack@devstack-masakari-louie:~$ sudo kill -9 8113 ... masakari detects and restarts VM .... stack@devstack-masakari-louie:~$ nova list +--------------------------------------+-------------+--------+------------+-------------+--------------------------------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------------+--------+------------+-------------+--------------------------------------------------------+ | 00f377e6-e21f-431a-aba9-31ce13fd974c | vm-1-cirros | ACTIVE | - | Running | private=fd37:1afb:1393:0:f816:3eff:fec9:7f48, 10.0.0.7 | +--------------------------------------+-------------+--------+------------+-------------+--------------------------------------------------------+ stack@devstack-masakari-louie:~$ masakari notification-list +--------------------------------------+----------------------------+----------+--------------------------------------+------+ | notification_uuid | generated_time | status | source_host_uuid | type | +--------------------------------------+----------------------------+----------+--------------------------------------+------+ | 89679602-494f-4394-ac32-b24f5e9afd83 | 2018-01-30T12:23:48.000000 | finished | 51bc8b8b-324f-499a-9166-38c22b3842cd | VM | | 5b535d99-4e02-44a5-bd21-d131d14aaa36 | 2018-01-30T12:10:43.000000 | running | 51bc8b8b-324f-499a-9166-38c22b3842cd | VM | | b8c6c561-7a93-40a2-8d73-3783024865b4 | 2018-01-26T19:41:29.000000 | running | 51bc8b8b-324f-499a-9166-38c22b3842cd | VM | | ed6433c3-939d-4aa8-bf47-3ce8e8c78d45 | 2018-01-26T17:13:03.000000 | running | 51bc8b8b-324f-499a-9166-38c22b3842cd | VM | +--------------------------------------+----------------------------+----------+--------------------------------------+------+ stack@devstack-masakari-louie:~$ !ps ps -ef | fgrep qemu libvirt+ 9494 1 10 12:23 ? 00:00:10 /usr/bin/qemu-system-x86_64 -name guest=instance-00000004,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-5-instance-00000004/master-key.aes -machine pc-i440fx-artful,accel=tcg,usb=off,dump-guest-core=off -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 00f377e6-e21f-431a-aba9-31ce13fd974c -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=17.0.0,serial=dacbc8a8-47c5-4132-86e3-3c902df4cf15,uuid=00f377e6-e21f-431a-aba9-31ce13fd974c,family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-5-instance-00000004/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/opt/stack/data/nova/instances/00f377e6-e21f-431a-aba9-31ce13fd974c/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=80,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:c9:7f:48,bus=pci.0,addr=0x3 -add-fd set=1,fd=83 -chardev pty,id=charserial0,logfile=/dev/fdset/1,logappend=on -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on stack 9628 5888 0 12:25 pts/0 00:00:00 grep -F --color=auto qemu stack@devstack-masakari-louie:~$ stack@devstack-masakari-louie:~$ YAY !!! Greg. From: Greg Waines <greg.wai...@windriver.com> Reply-To: "openstack-dev@lists.openstack.org" <openstack-dev@lists.openstack.org> Date: Tuesday, January 30, 2018 at 6:50 AM To: "openstack-dev@lists.openstack.org" <openstack-dev@lists.openstack.org> Subject: Re: [openstack-dev] [masakari] BUG in Masakari Installation and Procedure and/or Documentation Thanks Honjo, I reported bug in masakari Launchpad. https://bugs.launchpad.net/masakari/+bug/1746229 Greg. From: Rikimaru Honjo <honjo.rikim...@po.ntt-tx.co.jp> Reply-To: "openstack-dev@lists.openstack.org" <openstack-dev@lists.openstack.org> Date: Tuesday, January 30, 2018 at 1:49 AM To: "openstack-dev@lists.openstack.org" <openstack-dev@lists.openstack.org> Subject: Re: [openstack-dev] [masakari] BUG in Masakari Installation and Procedure and/or Documentation Hello Greg, Thank you for reporting & researching. On 2018/01/27 5:59, Waines, Greg wrote: Update on this. It turned out that i had incorrectly set the ‘project_name’ and ‘username’ in /etc/masakarimonitors/masakarimonitors.conf Setting both these attributes to ‘admin’ made it such that the instancemonitor’s notification to masakari-engine was successful. e.g. stack@devstack-masakari-louie:~/devstack$ masakari notification-list +--------------------------------------+----------------------------+---------+--------------------------------------+------+ | notification_uuid | generated_time | status | source_host_uuid | type | +--------------------------------------+----------------------------+---------+--------------------------------------+------+ | b8c6c561-7a93-40a2-8d73-3783024865b4 | 2018-01-26T19:41:29.000000 | running | 51bc8b8b-324f-499a-9166-38c22b3842cd | VM | +--------------------------------------+----------------------------+---------+--------------------------------------+------+ stack@devstack-masakari-louie:~/devstack$ However I now get the following error in masakari-engine, when the masakari-engine attempts to do the VM Recovery Jan 26 19:41:28 devstack-masakari-louie masakari-engine[11795]: 2018-01-26 19:41:28.968 TRACE masakari.engine.drivers.taskflow.driver EndpointNotFound: publicURL endpoint for compute service named Compute Service not found Why is masakari-engine looking for a publicURL endpoint for service_type=’compute’ and service_name=’Compute Service’ ? I think there is no reason. This default value was added by the following patch. https://review.openstack.org/#/c/388734/ I think this is a bug. Could you report in Launchpad? See below that the Service Name = ‘nova’ ... NOT ‘Compute Service’ stack@devstack-masakari-louie:~/devstack$ openstack endpoint list +----------------------------------+-----------+--------------+----------------+---------+-----------+----------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+----------------+---------+-----------+----------------------------------------------+ | 0111643ef1584decb523524a3db5ce18 | RegionOne | nova_legacy | compute_legacy | True | public | http://10.10.10.14/compute/v2/$(project_id)s | | 01790448c22f49e69774adf290fba728 | RegionOne | gnocchi | metric | True | internal | http://10.10.10.14/metric | | 0b31693c6650499a981d580721be9e48 | RegionOne | vitrage | rca | True | internal | http://10.10.10.14:8999 | | 40f66ed61b4e4310829aa69e11c75554 | RegionOne | neutron | network | True | public | http://10.10.10.14:9696/ | | 47479cf64af944b996b1fbca42efd945 | RegionOne | nova | compute | True | public | http://10.10.10.14/compute/v2.1 | | 49dccfc61e8246a2a2c0b8d12b3db91a | RegionOne | vitrage | rca | True | admin | http://10.10.10.14:8999 | | 5261ba0327de4c2d92842147636ee770 | RegionOne | masakari | ha | True | internal | http://10.10.10.14:15868/v1/$(tenant_id)s | | 5df28622c6f449ebad12d9b62110cd08 | RegionOne | gnocchi | metric | True | admin | http://10.10.10.14/metric | | 64f8f401431042a0ab1d053ca4f4df02 | RegionOne | glance | image | True | public | http://10.10.10.14/image | | 69ad6b9d0b0b4d0a8da6fa36af8289cb | RegionOne | masakari | ha | True | public | http://10.10.10.14:15868/v1/$(tenant_id)s | | 7dd9d5396e9c49d4a41e2865b841f6a0 | RegionOne | masakari | ha | True | admin | http://10.10.10.14:15868/v1/$(tenant_id)s | | 811fa7f4b3c14612b4aca354dc8ea77e | RegionOne | vitrage | rca | True | public | http://10.10.10.14:8999 | | 8535da724c424363bffe1d033ee033e5 | RegionOne | cinder | volume | True | public | http://10.10.10.14/volume/v1/$(project_id)s | | 853f1783f1014075a03c16f7c3a2568a | RegionOne | keystone | identity | True | admin | http://10.10.10.14/identity | | 9450f5611ca747f2a049f22ff0996dba | RegionOne | cinderv3 | volumev3 | True | public | http://10.10.10.14/volume/v3/$(project_id)s | | 9a73696d88a9438cb0ab75a754a08e9d | RegionOne | gnocchi | metric | True | public | http://10.10.10.14/metric | | b1ff2b4d683c4a58a3b27232699d0058 | RegionOne | cinderv2 | volumev2 | True | public | http://10.10.10.14/volume/v2/$(project_id)s | | d4e66240faff48f2b5e1d0fcfb73a74b | RegionOne | placement | placement | True | public | http://10.10.10.14/placement | | fda917fd368a4a479c9c186df1beb8e9 | RegionOne | keystone | identity | True | public | http://10.10.10.14/identity | +----------------------------------+-----------+--------------+----------------+---------+-----------+----------------------------------------------+ stack@devstack-masakari-louie:~/devstack$ let me know your thoughts, I don’t mind raising the required BUG in Launchpad if required, Greg. p.s. my masakari configurations ... wrt hosts and segments ... are as follows: stack@devstack-masakari-louie:~/devstack$ masakari segment-list +--------------------------------------+-----------+-------------+--------------+-----------------+ | uuid | name | description | service_type | recovery_method | +--------------------------------------+-----------+-------------+--------------+-----------------+ | 9c6e22bd-7fab-40cb-a8e0-3702137f3227 | segment-1 | - | COMPUTE | auto | +--------------------------------------+-----------+-------------+--------------+-----------------+ stack@devstack-masakari-louie:~/devstack$ masakari host-list --segment-id 9c6e22bd-7fab-40cb-a8e0-3702137f3227 +--------------------------------------+-------------------------+---------+--------------------+----------+----------------+--------------------------------------+ | uuid | name | type | control_attributes | reserved | on_maintenance | failover_segment_id | +--------------------------------------+-------------------------+---------+--------------------+----------+----------------+--------------------------------------+ | 51bc8b8b-324f-499a-9166-38c22b3842cd | devstack-masakari-louie | COMPUTE | SSH | False | False | 9c6e22bd-7fab-40cb-a8e0-3702137f3227 | +--------------------------------------+-------------------------+---------+--------------------+----------+----------------+--------------------------------------+ stack@devstack-masakari-louie:~/devstack$ From: Greg Waines <greg.wai...@windriver.com<mailto:greg.wai...@windriver.com>> Reply-To: "openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>" <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Date: Wednesday, January 24, 2018 at 4:13 PM To: "openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>" <openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>> Subject: [openstack-dev] [masakari] BUG in Masakari Installation and Procedure and/or Documentation I am looking for some input before I raise a BUG. I reviewed the following commits which documented the Masakari and MasakariMonitors Installation and Procedures. i.e. https://review.openstack.org/#/c/489570/ https://review.openstack.org/#/c/489095/ I created an AIO devstack with Masakari on current/master ... this morning. I followed the above instructions on configuring and installing Masakari and MasakariMonitors. I created a VM and then ‘sudo kill -9 <qemu-pid-of-vm>’ and I got the following error from instance monitoring trying to send the notification message to masakari-engine. ( The request you have made requires authentication. ) ... see below, Is this a known BUG ? Greg. 2018-01-24 20:29:16.902 12473 INFO masakarimonitors.instancemonitor.libvirt_handler.callback [-] Libvirt Event: type=VM, hostname=devstack-masakari-new, uuid=6884cf13-5797-487b-9cb1-053a2e18b60e, time=2018-01-24 20:29:16.902347, event_id=LIFECYCLE, detail=STOPPED_FAILED) 2018-01-24 20:29:16.903 12473 INFO masakarimonitors.ha.masakari [-] Send a notification. {'notification': {'hostname': 'devstack-masakari-new', 'type': 'VM', 'payload': {'instance_uuid': '6884cf13-5797-487b-9cb1-053a2e18b60e', 'vir_domain_event': 'STOPPED_FAILED', 'event': 'LIFECYCLE'}, 'generated_time': datetime.datetime(2018, 1, 24, 20, 29, 16, 902347)}} 2018-01-24 20:29:16.977 12473 WARNING masakarimonitors.ha.masakari [-] Retry sending a notification. (HttpException: The request you have made requires authentication. (HTTP 401) (Request-ID: req-9c734f56-aca9-40a9-b2dd-3f372de8c34e), The request you have made requires authentication.): HttpException: HttpException: The request you have made requires authentication. (HTTP 401) (Request-ID: req-9c734f56-aca9-40a9-b2dd-3f372de8c34e), The request you have made requires authentication. ... 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari [-] Exception caught: HttpException: The request you have made requires authentication. (HTTP 401) (Request-ID: req-26a5de94-aaad-4f8f-949e-cbfeb5e31b8b), The request you have made requires authentication.: HttpException: HttpException: The request you have made requires authentication. (HTTP 401) (Request-ID: req-26a5de94-aaad-4f8f-949e-cbfeb5e31b8b), The request you have made requires authentication. 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari Traceback (most recent call last): 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/ha/masakari.py", line 91, in send_notification 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari payload=event['notification']['payload']) 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/masakariclient/sdk/ha/v1/_proxy.py", line 65, in create_notification 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari return self._create(_notification.Notification, **attrs) 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/openstack/proxy2.py", line 194, in _create 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari return res.create(self._session) 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/openstack/resource2.py", line 588, in create 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari json=request.body, headers=request.headers) 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari return self.request(url, 'POST', **kwargs) 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 64, in map_exceptions_wrapper 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari return func(*args, **kwargs) 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 352, in request 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari return super(Session, self).request(*args, **kwargs) 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in request 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari auth_headers = self.get_auth_headers(auth) 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in get_auth_headers 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari return auth.get_headers(self, **kwargs) 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 95, in get_headers 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari token = self.get_token(session) 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari return self.get_access(session).auth_token 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari self.auth_ref = self.get_auth_ref(session) 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari return self._plugin.get_auth_ref(session, **kwargs) 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 165, in get_auth_ref 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari authenticated=False, log=False, **rkwargs) 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari return self.request(url, 'POST', **kwargs) 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 66, in map_exceptions_wrapper 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari raise exceptions.from_exception(e) 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari HttpException: HttpException: The request you have made requires authentication. (HTTP 401) (Request-ID: req-26a5de94-aaad-4f8f-949e-cbfeb5e31b8b), The request you have made requires authentication. 2018-01-24 20:31:17.769 12473 ERROR masakarimonitors.ha.masakari __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org<mailto:openstack-dev-requ...@lists.openstack.org>?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/ Rikimaru Honjo E-mail:honjo.rikim...@po.ntt-tx.co.jp<mailto:honjo.rikim...@po.ntt-tx.co.jp> __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org<mailto:openstack-dev-requ...@lists.openstack.org>?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
