-11-12 17:55 GMT+01:00 Josh Durgin <jdur...@redhat.com>:
> On 11/12/2015 07:41 AM, Saverio Proto wrote:
>> So here is my best guess.
>> Could be that I am missing this patch ?
I am investigating why my cinder is slow deleting volumes.
you might remember my email from few days ago with subject:
"cinder volume_clear=zero makes sense with rbd ?"
so it comes out that volume_clear has nothing to do with the rbd driver.
cinder was not guilty, it was really
2015-11-12 16:25 GMT+01:00 Saverio Proto <ziopr...@gmail.com>:
> Hello there,
> I am investigating why my cinder is slow deleting volumes.
> you might remember my email from few days ago with subject:
> "cinder volume_clear=zero makes sens
I need to quickly find a workaround to be able to use ceph object map
features for cinder volumes with rbd backend.
However, upgrading everything from Juno to Kilo will require a lot of
time for testing and updating all my puppet modules.
Do you think it is feasible to start
I am using cinder with rbd, and most volumes are created from glance
images on rbd as well.
Because of ceph features, these volumes are CoW and only blocks
different from the original parent image are really written.
Today I am debugging why in my production system deleting cinder
I have a cloud user that is trying to implement the following topology
ext_net <|R1|> internal_net <|R2|> dbservers_network
- internal_net: 10.0.2.0/24
- dbservers_net: 10.0.3.0/24
Now according to the documentation:
backend with the object-map.
If anyone is running juno and wants to enable this feature we have
packages published here:
2015-11-26 11:36 GMT+01:00 Saverio Proto <ziopr...@gmail.com>:
> I think it is
you can use r10k
go in a empty folder, create a file called Puppetfile with this content:
> Can you get R10k to NOT install dependencies listed in metadata etc.?
In my experience r10k will not try to install any dependencies.
OpenStack-operators mailing list
the rbd backend.
This means that if glance images do not have the rbd object map
features, the cinder volumes will have flags "object map invalid".
So, we are now trying to backport this feature of the rbd driver in
glance as well.
2015-11-24 13:12 GMT+01:00 Saverio Pr
upgrade one component at a time.
> Cinder was one of the first components that we upgraded to kilo,
> meaning that other components (glance, nova, ...) were running Juno.
> We didn't have any problem with this setup.
> On Tue, Nov 17, 2015 at 6:01 PM, S
what version of Openstack are you running ?
2015-11-23 20:04 GMT+01:00 Xav Paice :
> Over the last few months we've had a few incidents where the process to
> create network namespaces (Neutron, OVS) on the network nodes gets 'stuck'
2015-11-24 9:51 GMT+01:00 Xav Paice <xavpa...@gmail.com>:
> Neutron is Juno, on Trusty boxes with the 3.19 LTS kernel. We're in the
> process of updating to Kilo, and onwards to Liberty.
> On 24 November 2015 at 21:24, Saverio Proto <ziopr...@gmail.co
to test this on his Juno setup I can also share
.deb packages for Ubuntu
2015-11-16 16:21 GMT+01:00 Saverio Proto <ziopr...@gmail.com>:
> I tried to backport this patch to Juno but it is not that trivial for
> me. I have 2 tests failing, about volume cloning a
I made some tests running Hadoop on our Openstack Cloud. The idea is
to do map reduce examples but using the swift storage instead of HDFS.
We have Ceph backend for cinder volumes so HDFS on top of that does
not really fit.
I managed to configure hadoop to access swift with
I will need a visa to come to the US for the Mid-Cycle Ops Meetup.
The process to obtain a Visa can take up to 8 weeks, and I cannot
apply until dates and venue are decided.
please set a date at least 8 weeks ahead, or few people that can't
make it on time to apply for visa will not
first of all I suggest you read this article:
> What is the best way to performe an upgrade from Juno to Mitaka?
I would go for the in place upgrade, but I always upgraded without
reading this documentation page:
I dont get what means having two parallel migration branches.
Do you have to choose one of the branches ? If yes how ?
Or it just means that some operations can be safely
Did we create already a etherpad ?
I hope I did this correctly:
2016-06-22 15:58 GMT+02:00 Mark Voelker :
> Hi Ops,
> FYI for those that may not be aware, that’s also the week of OpenStack East.
> OpenStack East
have a look at Openstack Manila and CephFS
2016-06-21 11:42 GMT+02:00 Michael Stang :
> I think I have asked my question not correctly, it is not for the cinder
> backend, I meant the shared storage for the instances which is
a very widely adopted solution is to use Ceph with rbd volumes.
you find more options here under Volume drivers:
Hello there :)
is anyone from the openstack foundation or from bloomberg that can
help out with this ?
I share this for anyone that needs visa.
for Austin we had something like this:
I think the stacktrace is usefull when debugging, even as an operator.
Instead of changing the logs to satisfy the tools, it is the tools
that should be enable to parse the logs correctly.
2016-01-05 14:08 GMT+01:00 Akihiro Motoki :
> # cross-posting to -dev
your question is very general.
as a general answer I can suggest to use a configuration management
system such as Puppet or Ansible to take care of the servers. It is
easier to keep stuff in different datacenter running the same version
of packages in this way.
I hope this helps.
I need to delete some users and tenants from my public cloud. Before
deleting the users and tenants from keystone, I need to delete all the
resources in the tenants.
I am stucked listing the glance images uploaded in a specific tenant.
I cannot find the way, I always get either all
I am trying to stick to the new openstack client CLI, but sometimes I
get completely lost.
So I used to do with python-novaclient instance snapshots like this:
nova image-create snapshotname
I just cannot understand how to do the same with the new client. Could
> We have an image promotion process that does this for us. The command I use
> to get images from a specific tenant is:
> glance --os-image-api-version 1 image-list --owner=
> I'm sure using the v1 API will make some cringe, but I haven't found
> anything similar in the v2 API.
I used to do this in Juno, and now I upgraded to Kilo and it is not
macsp:~ proto$ openstack server image create --name test
Cannot 'createImage' instance 81da19c6-efbe-4002-b4e8-5ce352ffdd14
while it is in vm_state paused (HTTP
It looks like not all snapshots are listed because I am hitting the
# The maximum number of items that a collection resource
# returns in a single response (integer value)
2016-03-23 9:29 GMT+01:00 Saverio Proto <ziopr...@gmail.com>:
When this kind of very corner case happen I usually hack manually the database.
Go with mysql in the nova database and put the new ID for those two instances.
my 2 cents
2016-03-23 5:17 GMT+01:00 Rahul Sharma :
> Hi All,
> Due to a hostname change, we ended
because I need to tune osapi_max_limit I wrote this puppet patch
but I am very bad at coding :) please review :)
2016-03-23 11:17 GMT+01:00 Saverio Proto <ziopr...@gmail.com>:
> It looks like not all snapshots are liste
Thanks for merging the patch so quickly ! I will try today to build
new ubuntu packages and and for feedback at UCA.
> But yes, thank you for pointing this out. Bug fixes are mostly
> backported to Liberty release, in some rare cases Kilo might get a fix
> as well.
This is an endless discussion
> In our environments, we offer two types of storage. Tenants can either use
> Ceph/RBD and trade speed/latency for reliability and protection against
> physical disk failures, or they can launch instances that are realized as
> LVs on an LVM VG that we create on top of a RAID 0 spanning all but
what kind of snapshot are you doing ?
1) Snapshot of a instance running on a ephimeral volume ?
2) Snapshot of a instance booted from Volume ?
3) Snapshot of a volume ?
in case 1 the ephemeral volume is in the volume pool with the name
when you snapshot, this must be read
t just works. Like so:
> It still needs to go through the review process so you will need to ping
> some horizon developers in IRC.
> Getting that packaged may take longer.
> On Mar 3, 2016 8:43 AM, "Saverio Pr
in Manchester we had this interesting discussion about asking to
backport bugfixes in order to avoid to build own packages.
We run Openstack with Ubuntu and we use the Ubuntu Cloud Archive.
We are upgrading our
with the help of James I was finally able to properly compile ubuntu
packages to carry local patches in my Kilo installation.
I documented the all process here:
I hope this is useful for everyone. The notes I wrote can be
I am not a developer but I tried my best !
I applied the patch in my staging system and it fixes the problem :)
2016-03-22 13:46 GMT+01:00 Saverio Proto <ziopr...@gmail.com>:
> I found the problem. It happens only then the instance h
I upgraded to Kilo and I see the very same bug.
What is the right way to fill a bug to Cinder, is it here ?
Will it be considered if the Bug is in Kilo ?
2016-03-16 13:42 GMT+01:00 Saverio Proto <ziopr...@gmail.com>:
one of the users of our cluster opened a ticket about a snapshot
corner case. It is not possible to snapshot a instance that is booted
from volume when the instance is paused. So I wrote this patch, and
from the discussion you can see that I learnt a lot about snapshots.
I am running Kilo and I found this bug:
because neutron is trying to do operations over and over again on
namespaces that do not exist anymore, this kills the CPU of our
I wrote a patch for kilo that fixes the
> We are in an even worst situation: we have flavors with 256GB of ram
> but only 100GB on the local hard disk, which means that we cannot
> snapshot VMs with this flavor.
> If there is any way to avoid saving the content of the ram to disk (or
> maybe there is a way to snapshot the ram to,
an account by doing swift
> stat -v.
> In this case, the user in step 2 is not able to do anything else in account
> X besides do object listing in the container and get its objects, which is
> what I was aiming for. What does not work for me is if I set the read-acl to
testing this docker setup is in my TODO list from a long time:
Looks very well done but I think is not very well known.
OpenStack-operators mailing list
I did run into the same problem today. Did you find pointers to more
updated documentation ? Were you able to configure the cross tenant
read ACL ?
2016-04-20 13:48 GMT+02:00 Wijngaarden, Pieter van
> Hi all,
LBaaS V1 is deprecated in Liberty.
I am aware of this documentation:
We are now running a public cloud, and some users deployed LBaaS V1
(with heat templates).
How do we migrate to LBaaS V2 without delete users
On ubuntu we just replace these two files:
we make sure with puppet that our version of these two files is in place.
So it is puppetlabs-rabbitmq 5.4.0 that created this problem.
I rolled back to 5.3.1
Here is the commit that introduces the problem:
2016-07-07 13:18 GMT+02:00 Saverio Proto <zi
I have a pet server hosting both mysql and rabbitmq for openstack.
I guess this is common for many people running a Openstack Controller pet.
I just figured out I have a strange problem with my puppet modules dependencies:
we are doing the upgrade Kilo to Liberty pet by pet.
We already upgraded successfully Keystone and Glance.
Now I started the Nova pet upgrade. For the controller node it was ok.
As soon as I upgraded the compute nodes I had a problem with neutron.
I can't lock the
we want to bridge one of our tenants networks with a physical network
where we have some hardware appliances.
we can't easily use provider networks, because our compute-nodes are
connected over a L3 network, so there is not a shared L2 segment where
we can bridge the VMs regardless of
So far I am just learning. I wrote down some notes based on my
experience of this week.
I am looking forward to the Milano Ops Mid-cycle to chat face to face
with more operators interested in Kolla.
The key point is how to grow the Kolla
the ops midcycle is in a about 1 month:
If you need a invitation letter for VISA to Italy please send an email
to egiannu...@enter.eu and Cc: ziopr...@gmail.com .
We will help you will the Italian forms to fill.
ENTER that is hosting
Did you try to restart memcached after chaning the configuration to HA ?
there are two sections where you can configure, memcached_servers
how your config looks like ?
2017-01-24 6:48 GMT+01:00 Chris Apsey :
this feature you describe makes sense only if the tenant VMs are also
aggregated in the same rack. Is this your scenario ?
Given that you could have 3 network nodes running the L3-agent, then
you can make some scripting to stick the specific router (linux
namespace) to the specific
The three new compute nodes that you added are empty, so most likely
the new instances are scheduled to those three (3 attempts) and
something goes wrong.
with admin rights do:
openstack server show uuid
this should give you the info about the compute node where the
instance was scheduled. Check
I have no idea from this log file. Trying to make nova-compute more
verbose if you dont find anything in the logs
2017-02-20 7:50 GMT+01:00 Anwar Durrani <durrani.an...@gmail.com>:
> On Thu, Feb 16, 2017 at 1:44 PM, Saverio Proto <ziopr...@gmail.com> wrote:
I would fill a bug on launchpad for this issue.
114 VMs is not much. Can you identify how to trigger the issue to
reproduce it ? or it just happens randomly ?
When you say rebooting the network node, do you mean the server
running the neutron-server process ?
what version and
What version of Openstack are we talking about ?
2017-03-02 12:11 GMT+01:00 Grant Morley :
> Hi All,
> Not sure if anyone can help, but as of today we are unable to launch any
> instances and I have traced back the error to glance. Whenever I try and
If you pass the uuid of the image does it work ?
2017-03-02 13:49 GMT+01:00 Grant Morley <gr...@absolutedevops.io>:
> Hi Saverio,
> We are running Mitaka - sorry forgot to mention that.
> On 02/03/17 12:45, Saverio Proto wrote:
> What ve
what is the size of the RAM for the flavor of the instance you are trying
to snapshotting ?
2016-08-25 8:48 GMT+02:00 Michael Stang :
> Hi Konstantin, hi Saverio
> thank you for your answers.
> I checked the version, these are
this is merged upstream in Mitaka:
anyone already did a cherry pick in Liberty and is running this in production ?
I plan to test this soon. Any feedback is appreciated !
:00 Logan V. <lo...@protiumit.com>:
> Have been using it since kilo. It was a clean pick into Liberty and worked
> great for me.
> On Friday, September 2, 2016, Saverio Proto <ziopr...@gmail.com> wrote:
Is the connection to the database properly working ?
2016-09-07 5:01 GMT+02:00 :
> Hi Everyone,
> I'm trying to install identify service on Ubuntu 16.04 LTS. The guide I use:
> When I
Sorry you are on Liberty Centos !!
Did you try openstack quota show ?
What do you see ?
Then try to create a Volume from the CLI. Usually you get better errore
Il 08 set 2016 9:05 AM, "Saverio Proto" <ziopr...@gmail.com> ha scritto:
> please tell us a
once I had a very weird problem with quotas. Usually the quota is per
tenant/project. But somehow you can limit the quota per specific user.
Please can you check in your nova database if this table is empty:
mysql> describe project_user_quotas;
please tell us at least the version of openstack :)
Il 07 set 2016 4:31 PM, "William Josefsson" ha
> When I create projects with quotas configured, e.g. number volumes 8, and
> try to provision a VM, I get:
> *The requested instance cannot
Il 08 set 2016 9:07 AM, "Saverio Proto" <ziopr...@gmail.com> ha scritto:
> Sorry you are on Liberty Centos !!
> Did you try openstack quota show ?
> What do you see ?
> Then try to create a Volume from the CLI. Usually you get better
I would like to thank Chris for all the work done to host our Ops
Meetup. I had a great time in NYC, and I came back to Switzerland with
a lot of input for work to do.
Also thanks to all of you that participated. It was great to share
experiences on Openstack operations with so many
I am looking at the same problem today.
I am running Liberty, and I am not sure Puppet is doing the right
thing for Ubuntu here:
Should this patch be backported
if you changed the cinder host name you need also to reflect this
change in the database for the existing volumes.
in mysql you have to do something like:
update volumes set host='newhostname#DEFAULT' where host =
update volumes set host='newhostname' where
everytime I send an email to the list I get back two emails with the
mani...@yahoo-inc.com is no longer with Yahoo! Inc.
sune...@yahoo-inc.com is no longer with Yahoo! Inc.
this happens for every other person subscribed to this list ? Maybe we
should unsubscribe these
this patch fixed my problem:
but it is an ugly hack according to:
anyone knows how to make it better ?
> Most of the topics that were covered in the Ubuntu packaging session are
> summarized in our wiki, to which I've updated based on our discussions:
thanks for updating the wiki so quickly.
I am trying to rebuild stable/liberty cinder adding
n 31 August 2016 at 12:56, Saverio Proto <ziopr...@gmail.com> wrote:
>> Hello ops,
>> this patch fixed my problem:
>> but it is an ugly hack according to:
t to prepend your strings with u in the
> print statements ?
> On 31 August 2016 at 14:03, Saverio Proto <ziopr...@gmail.com> wrote:
>> to stick with the subject of the email you can also call the instance
>> for example
to stick with the subject of the email you can also call the instance
and this will trigger the bug anyway :)
2016-08-31 14:54 GMT+02:00 Saverio Proto <ziopr...@gmail.com>:
> Hello Matt,
> I am sorry, I realize now I sent a very dumb emai
In NYC we had this session:
It came out that most of us configure Nagios to be less noisy, and
there are Warning strings that most people just ignore, because these
warnings are harmless and you dont want an email for each
If your setup has a single user database, so all users are under the
same administrative domain, what you describe is like having different
Openstack Regions, or different Nova Cells.
I would suggest to look into Multi Region that is the easier to implement.
> Alternatively, you could drop the 'external' attribute and attach your
> instances directly to the provider network (no routers or private networks).
I can't. Because in my network design I do not have all the compute
nodes on a common L2 segment.
I have a l3 fabric between the compute nodes.
Is the '@' character allowed in the tenant/project names ?
2016-10-05 23:36 GMT+02:00 Steve Martinelli :
> There are some restrictions.
> 1. The project name cannot be longer than 64 characters.
> 2. Within a domain, the project name is unique. So you can have
>  http://beyondtheclouds.github.io
> - Mail original -
>> De: "Saverio Proto" <ziopr...@gmail.com>
>> À: "stig openstack" <stig.openst...@telf
sorry I will not make it today
Il 04 ott 2016 4:42 PM, "Melvin Hillsman" ha scritto:
> Melvin Hillsman
> Ops Technical Lead
> OpenStack Innovation Center
> phone: (210)
you might want to read also this old thread:
to debug this issue, get a shell on the compute node where the
instance is running. When requesting the snapshot check these two
Sorry I missed the Mailing List in the Cc:
2016-10-03 9:15 GMT+02:00 Saverio Proto <ziopr...@gmail.com>:
> Hello Kevin,
> thanks for your answer.
> so far I managed to make the network not shared just by making it not
> external. Because I dont need NAT and flo
> How are you creating the provider (external) network?
> On Thu, Sep 29, 2016 at 6:01 AM, Saverio Proto <ziopr...@gmail.com> wrote:
>> - openstack liberty
>> - ubuntu trusty
>> - neutron net
At GARR in Rome there will be an event shortly before Barcelona about
Openstack and Identity Federation.
This is a use case that is very important for NREN running public
cloud for Universities, where a Identity Federation is already
- openstack liberty
- ubuntu trusty
- neutron networking with vxlan tunnels
we have been running Openstack with a single external network so far.
Now we have a specific VLAN in our datacenter with some hardware boxes
that need a connection to a specific tenant network.
we just had the Ops Meetup Team IRC meeting.
There are two possible venues for the next OpenStack Operators
Mid-Cycle meetup: Milano and Tokyo.
To select the venue we created
t time? It is normally 10AM my time which will be 13:00 UTC on the
> By the way, the Barcelona hosting offer has been updated with possible week
> day slots, see
snapshot are meant to be backups.
I would not install software on a machine, and then use the snapshot
as a glance image to start many new instances.
you could just create machines with a clean glance image and install
the necessary software with ansible/puppet
if the installation time
look at the log files !
2016-11-09 17:54 GMT+01:00 suresh kumar :
> Hi All,
> I am unable to delete a security group in my kilo environment, this security
> group is not associated to any instances or LB
> nova secgroup-delete Test
if the instance is booted from Volume, when you "nova snapshot" in the
glance repo you find a image of 0 bytes, that just contains metadata.
you need to make cinder volume snapshots.
This is a recurring FAQ for the users of our cloud, I wrote something
about it here:
in the upgrade did the version of ovs change ?
what Openstack distribution are you using ?
2017-01-10 16:30 GMT+01:00 Telmo Morais :
> Hi All,
> We are currently on the process of upgrading from Mitaka to Newton, and on
> the upgraded compute
> Many thanks for the reply, I'll answer your queries below;
> On 01/12/16 12:49, Saverio Proto wrote:
>> while the problem is in place, you should share the output of
>> ip rule show
>> ip route show table 1
while the problem is in place, you should share the output of
ip rule show
ip route show table 1
It could be just a problem in your ruleset
and, which one is your webserver ? can you tcpdump to make sure reply
packets get out on the NIC with src address 10.0.16.11 ?
in the osops-tools-contrib repo so far I proposed always python
scripts that are contained in a single file.
Now I have this file:
that I reuse in many scripts, look at this:
but maybe is not the best idea to commit this generic
floating IPs is the real issue.
When using horizon it is very easy for users to allocate floating ips
but it is also very difficult to release them.
In our production cloud we had to change the default from 50 to 2. We
have to be very conservative with floatingips quota because our
we use rclone a lot, and we are happy with it.
the real problem I would say is that a lot of these tools use the
latest AWS4 signature.
AFAIK the radosgw with Ceph Jewel and Openstack keystone integration
supports only AWS2 signature because of this bug:
what version of openstack ?
is the instance booting from ephemeral disk or booting from cinder volume ?
When you boot from volume, that will be the root disk of your
instance. The user could have clicked on "Delete Volume on Instance
Delete". It can be selected when creating a new
1 - 100 of 160 matches
Mail list logo