Re: [openstack-dev] [ceilometer] about workload partition
On 2017-12-01 05:03 AM, 李田清 wrote: >> Hello, >> we test workload partition, and find it's much slower than not >> using it. >> After some review, we find that, after get samples from >> notifications.sample >> ceilometer unpacks them and sends them one by one to the pipe >> ceilometer.pipe.*, this will make the consumer slow. Right now, >> the rabbit_qos_prefetch_count to 1. If we sent it to 10, the connection >> will be reset > currently, i believe rabbit_qos_prefetch_count will be set to whatever > value you set batch_size to. You mean in the past, it can not be set to whatever? We test newton, and find under 1k vm, if we open workload partition, it will be reset regularly. >> regularly. Under this pos, the consumer will be very slow in >> workload partition. If you do not use workload partition, the messages >> can all be consumer. If you use it, the messages in pipe will be piled >> up more and more。 > what is "pos"? i'm not sure it means the same thing to both of us... or > well i guess it could :) sorry, it is qos. this qos. >> May be right now workload partition is not a good choice? Or any >> suggestion? >> >i'll give a two part answer but first i'll start with a question: what >version of oslo.messaging do you have? newton 5.10.2 >i see a performance drop as well but the reason for it is because of an >oslo.messaging bug introduced into master/pike/ocata releases. more >details can be found here: >https://bugs.launchpad.net/oslo.messaging/+bug/1734788. we're working >on >backporting it. we've also done some work regarding performance/memory >to shrink memory usage of partitioning in master[1]. >with that said, there are only two scenarios where you should have >partitioning enabled. if you have multiple notification agents AND: >1. you have transformations in your pipeline >2. you want to batch efficiently to gnocchi >if you don't have workload partitioning on, your transform metrics will >probably be wrong or missing values. it also won't batch to gnocchi so >you'll see a lot more http requests there. >so yes, you do have a choice to disable it, but the above is your tradeoff. >[1] >https://review.openstack.org/#/q/topic:plugin+ >(status:open+OR+status:merged)+project:openstack/ceilometer >cheers, >-- >gord __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [ceilometer] about workload partition
Hello, we test workload partition, and find it's much slower than not using it. After some review, we find that, after get samples from notifications.sample ceilometer unpacks them and sends them one by one to the pipe ceilometer.pipe.*, this will make the consumer slow. Right now, the rabbit_qos_prefetch_count to 1. If we sent it to 10, the connection will be reset regularly. Under this pos, the consumer will be very slow in workload partition. If you do not use workload partition, the messages can all be consumer. If you use it, the messages in pipe will be piled up more and more。 May be right now workload partition is not a good choice? Or any suggestion?__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [gnocchi]can we put the api on udp?
Hello, right now, ceilometer notification agent can send samples by udp publisher.. But gnocchi can only accept by rest api. Is there a way to use udp to accept notification agent's samples that sending by udp? Thanks a lot__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [ceilometer][oslo_messaging] Random Connection reset by peer
Hello, I test ceilometer agent notification workload partition. I found it is too fragile. The load is 1k cirrors vm. I make processing queue to 4 and workers to 1. I can sure the network is ok. But in the ceilometer agent notification log i see this http://paste.openstack.org/show/624795/ I am sure rabbitmq resource is enough. the status is here http://paste.openstack.org/show/624796/ And after some dig in i found this. https://stackoverflow.com/questions/383738/104-connection-reset-by-peer-socket-error-or-when-does-closing-a-socket-resu Can someone join me to test it? Or is there someone know this why and how to fix this? Thanks a lot a lot.__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] oslo listener heartbeat
Hello, i see this in impl_rabbit.py # NOTE(sileht): if purpose is PURPOSE_LISTEN # the consume code does the heartbeat stuff # we don't need a thread self._heartbeat_thread = None if purpose == rpc_common.PURPOSE_SEND: self._heartbeat_start() When a create a batch listener, i do not find where it start the heartbeat? Can someone help me point it out? Thanks...__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] the workload partition willcauseconsumer disappeared
I use 5.10.2
> I test newton 5.10.2, and in ceilometer agent notification, the log shows
> 2017-10-21 03:33:19.779 225636 ERROR root [-] Unexpected exception
> occurred 60 time(s)... retrying.
> 2017-10-21 03:33:19.779 225636 ERROR root Traceback (most recent call last):
> 2017-10-21 03:33:19.779 225636 ERROR root File
> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 250, in
> wrapper
> 2017-10-21 03:33:19.779 225636 ERROR root return infunc(*args, **kwargs)
> 2017-10-21 03:33:19.779 225636 ERROR root File
> "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/base.py", line
> 203, in _runner
> 2017-10-21 03:33:19.779 225636 ERROR root batch_size=self.batch_size,
> batch_timeout=self.batch_timeout)
> 2017-10-21 03:33:19.779 225636 ERROR root File
> "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/base.py", line
> 52, in wrapper
> 2017-10-21 03:33:19.779 225636 ERROR root msg = func(in_self,
> timeout=timeout_watch.leftover(True))
> 2017-10-21 03:33:19.779 225636 ERROR root File
> "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
> 286, in poll
> 2017-10-21 03:33:19.779 225636 ERROR root
> self._message_operations_handler.process()
> 2017-10-21 03:33:19.779 225636 ERROR root File
> "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
> 89, in process
> 2017-10-21 03:33:19.779 225636 ERROR root task()
> 2017-10-21 03:33:19.779 225636 ERROR root File
> "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py",
> line 251, in acknowledge
> 2017-10-21 03:33:19.779 225636 ERROR root self._raw_message.ack()
> 2017-10-21 03:33:19.779 225636 ERROR root File
> "/usr/lib/python2.7/site-packages/kombu/message.py", line 88, in ack
> 2017-10-21 03:33:19.779 225636 ERROR root
> self.channel.basic_ack(self.delivery_tag)
> 2017-10-21 03:33:19.779 225636 ERROR root File
> "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1583, in basic_ack
> 2017-10-21 03:33:19.779 225636 ERROR root self._send_method((60, 80), args)
> 2017-10-21 03:33:19.779 225636 ERROR root File
> "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 50, in
> _send_method
> 2017-10-21 03:33:19.779 225636 ERROR root raise
> RecoverableConnectionError('connection already closed')
> 2017-10-21 03:33:19.779 225636 ERROR root RecoverableConnectionError:
> connection already closed
what's the oslo.messaging version? i feel like this is something on
oslo.messaging and it does sound familiar to possibly it's fixed.
--
gord
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] the workload partition will causeconsumer disappeared
On 2017-10-23 09:57 PM, 李田清 wrote:
> We test ceilometer workload partition, and find even with one
> rabbitmq server, the ceilometer-pipe
> will lost its consumers. Does anyone know this?
> I configure, batch_size =1, batch_timeout =1,
> and pipeline_processing_queues = 1.
> If anyone know this, please point it out. Thanks
and you see no errors in notification-agent? does it start with a
consumer or is there never a consumer?
The error is
I test newton 5.10.2, and in ceilometer agent notification, the log shows
2017-10-21 03:33:19.779 225636 ERROR root [-] Unexpected exception occurred 60
time(s)... retrying.
2017-10-21 03:33:19.779 225636 ERROR root Traceback (most recent call last):
2017-10-21 03:33:19.779 225636 ERROR root File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 250, in wrapper
2017-10-21 03:33:19.779 225636 ERROR root return infunc(*args, **kwargs)
2017-10-21 03:33:19.779 225636 ERROR root File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/base.py", line 203,
in _runner
2017-10-21 03:33:19.779 225636 ERROR root batch_size=self.batch_size,
batch_timeout=self.batch_timeout)
2017-10-21 03:33:19.779 225636 ERROR root File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/base.py", line 52, in
wrapper
2017-10-21 03:33:19.779 225636 ERROR root msg = func(in_self,
timeout=timeout_watch.leftover(True))
2017-10-21 03:33:19.779 225636 ERROR root File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
286, in poll
2017-10-21 03:33:19.779 225636 ERROR root
self._message_operations_handler.process()
2017-10-21 03:33:19.779 225636 ERROR root File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
89, in process
2017-10-21 03:33:19.779 225636 ERROR root task()
2017-10-21 03:33:19.779 225636 ERROR root File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line
251, in acknowledge
2017-10-21 03:33:19.779 225636 ERROR root self._raw_message.ack()
2017-10-21 03:33:19.779 225636 ERROR root File
"/usr/lib/python2.7/site-packages/kombu/message.py", line 88, in ack
2017-10-21 03:33:19.779 225636 ERROR root
self.channel.basic_ack(self.delivery_tag)
2017-10-21 03:33:19.779 225636 ERROR root File
"/usr/lib/python2.7/site-packages/amqp/channel.py", line 1583, in basic_ack
2017-10-21 03:33:19.779 225636 ERROR root self._send_method((60, 80), args)
2017-10-21 03:33:19.779 225636 ERROR root File
"/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 50, in
_send_method
2017-10-21 03:33:19.779 225636 ERROR root raise
RecoverableConnectionError('connection already closed')
2017-10-21 03:33:19.779 225636 ERROR root RecoverableConnectionError:
connection already closed
are you setting pipeline_processing_queues =1 as a test? because it sort
of defeats purpose.
Yes, i just test the workload partition. If processing queues = 4, there will
be more
partition for each pipeline
cheers,
--
gord
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [ceilometer] the workload partition will cause consumer disappeared
Hello, We test ceilometer workload partition, and find even with one rabbitmq server, the ceilometer-pipe will lost its consumers. Does anyone know this? I configure, batch_size =1, batch_timeout =1, and pipeline_processing_queues = 1. If anyone know this, please point it out. Thanks__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [telemetry] ceilometer-notification restart with no consumers on ceilometer-pipe
Hello, I am testing ceilometer workload partition. And find that if we restart agent notification, the ceilometer-pip* queue will not have consumers any more. Does anyone know this? The pipeline.yaml is here http://paste.openstack.org/show/623909/ And i also find the ceilometer-pipe-meter_source:meter_sink-queue always has many messages in it. Does anyone met this before, or some suggestions can share? Thanks a lot...__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] nova boot from image created volume
Thanks Matt. -- Original -- From: "Matt Riedemann"<mriede...@gmail.com>; Date: Fri, Apr 7, 2017 00:18 AM To: "openstack-dev"<openstack-dev@lists.openstack.org>; Subject: Re: [openstack-dev] [nova] nova boot from image created volume On 4/6/2017 10:05 AM, Jay Pipes wrote: >> jaypipes@serialcoder:~/src/git.openstack.org/openstack/nova/nova$ >> ack-grep --ignore-dir tests --ignore-dir locale "volume_api.create" >> compute/api.py >> 2982:snapshot = self.volume_api.create_snapshot_force( >> >> api/openstack/compute/volumes.py >> 185:new_volume = self.volume_api.create( >> 582:create_func = self.volume_api.create_snapshot_force >> 584:create_func = self.volume_api.create_snapshot >> >> virt/block_device.py >> 63::returns: The availability_zone value to pass to volume_api.create >> 487:vol = volume_api.create(context, self.volume_size, '', >> '', >> 508:vol = volume_api.create(context, self.volume_size, >> 530:vol = volume_api.create(context, self.volume_size, >> vol_name, '', > > Best, > -jay > > On 04/06/2017 03:29 AM, 李田清 wrote: >> Hello, >> If we use nova boot from image and created volume, i think the >> nova will >> use volume/cinder.py:create to create volume. But after insert pdb, >> i do not find >> the specific code of line to call the create. Can someone help me to >> point out the >> code of the line? Thanks a lot. >> >> >> __ >> >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev For the case you're looking for, the volume is created here: https://github.com/openstack/nova/blob/8d9bf947a4c8654a30e016a5d95d9bec55447506/nova/virt/block_device.py#L508 That happens when the compute manager is preparing block devices to attach to the instance while building it on the compute host. After the volume is created and nova gets the volume ID back, we wait for it to be available before we can attach it: https://github.com/openstack/nova/blob/8d9bf947a4c8654a30e016a5d95d9bec55447506/nova/virt/block_device.py#L512 Which uses this function passed in from the compute manager (yes, it's gross tight coupling): https://github.com/openstack/nova/blob/8d9bf947a4c8654a30e016a5d95d9bec55447506/nova/compute/manager.py#L1221 And finally it attaches the volume to the instance via a call to the parent class: https://github.com/openstack/nova/blob/8d9bf947a4c8654a30e016a5d95d9bec55447506/nova/virt/block_device.py#L516 I hope this helps. It only took me 4 years to memorize the flow. :) -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova] nova boot from image created volume
Hello, If we use nova boot from image and created volume, i think the nova will use volume/cinder.py:create to create volume. But after insert pdb, i do not find the specific code of line to call the create. Can someone help me to point out the code of the line? Thanks a lot.__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [opensatck-dev][trove]redis replication
Thanks a lot Mariam -- Original -- From: Mariam Johnmari...@us.ibm.com; Date: Wed, Jun 17, 2015 08:41 PM To: OpenStack Development Mailing List (not for usage questions)openstack-dev@lists.openstack.org; Subject: Re: [openstack-dev] [opensatck-dev][trove]redis replication Have you checked the blueprint for this at: https://review.openstack.org/#/c/189445/. Hope that helps. Regards, Mariam. 李田清 ---06/17/2015 02:06:39 AM---Hello, Right now we can create one replication once, but it is not suitable for redis. What we w From: 李田清 tianq...@unitedstack.com To:openstack-dev openstack-dev@lists.openstack.org Date: 06/17/2015 02:06 AM Subject: [openstack-dev] [opensatck-dev][trove]redis replication Hello, Right now we can create one replication once, but it is not suitable for redis. What we will do for this? And if time permit can the assigner of redis replication tell about the process for redis replication? Thanks a lot.__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [opensatck-dev][trove]redis replication
Hello, Right now we can create one replication once, but it is not suitable for redis. What we will do for this? And if time permit can the assigner of redis replication tell about the process for redis replication? Thanks a lot.__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
