Re: [openstack-dev] [glance] Why no DB index on sort parameters
Now that raises a question: do we really need sorting based on arbitrary keys in our API (e.g. listing images, volumes, instances)? If we have this feature in our API, we're bound to run into problems by creating or not creating indexes, at large volumes -- hurts our motive to be easily-implementable for clouds of all sizes. -Rushi On 23 April 2015 at 20:40, Nikhil Komawar nikhil.koma...@rackspace.com wrote: Messing with indices is not a good idea to do iteratively. Indexing large data sets is a really expensive operation and should be done carefully and consistently. Changing around indices is only going to make things unstable. Thanks, -Nikhil From: Flavio Percoco fla...@redhat.com Sent: Thursday, April 23, 2015 7:52 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [glance] Why no DB index on sort parameters On 21/04/15 14:55 +, Nikhil Komawar wrote: Rally is great overall however, we need good EXPLAIN examples on real world data. Smaller deployments might benefit from a simple sample performance analysis however, larger data sets can have impacts on areas that you never expect. A spec means that we document the indices proposed in the code base, based on all of the use cases. The way I look at it, a patch is needed anyways and it (rally gate job) would get attention from reviewers when the patch is proposed. Yes, I believe we need both. However, I'd probably just start with something smaller and see how it behaves before going with big data sets. I'm not saying we don't need tests with proper data sets, I'm saying that I'd probably start with smaller ones. As Mike already mentioned in his email, there's an impact in writes and we can see that from Rally tests, AFAICT. The spec can come later, IMHO. Cheers, Flavio From: Flavio Percoco fla...@redhat.com Sent: Tuesday, April 21, 2015 10:48 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [glance] Why no DB index on sort parameters On 21/04/15 14:39 +, Nikhil Komawar wrote: This is a good idea. We recently removed a unique constraint that may result into some queries being very slow especially those that involve name property. I would recommend sketching out a spec that identifies potential full table scans especially for queries that join over image_properties table. We should discuss there what other use cases look like rather than smaller feedback on the ML. More thatn a spec, I'd be interested in seeing the patch with the change up and the results reported in Rally. I guess we'll need a spec anyway, although I'd probably be ok with a good bug report here. /me *shrugs* Flavio Thanks, -Nikhil ━━━ From: Mike Bayer mba...@redhat.com Sent: Tuesday, April 21, 2015 9:45 AM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [glance] Why no DB index on sort parameters On 4/21/15 2:47 AM, Ajaya Agrawal wrote: Hi All, I see that glance supports arbitrary sort parameters and the default is created_at while listing images. Is there any reason why we don't have index over these fields? If we have an index over these fields then we would avoid a full table scan to do sorting. IMO at least the created_at field should have an index on it. just keep in mind that more indexes will place a performance penalty on INSERT statements, particularly at larger volumes. I have no idea if that is important here but something to keep in mind. Cheers, Ajaya __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
Re: [openstack-dev] [OpenStack Foundation] Finding people to work on the EC2 API in Nova
There seems to be an agreement that people are fine if we improve the in-tree Nova EC2 API more robust by adding proper Tempest tests to it, regardless of the way forward (in-Nova-tree vs out-of-tree repo). But there are also concerns that Tempest is not the right place for these EC2 API tests. While I am not mature enough with testing methodologies to comment on what is good vs bad, I am seeing a problem if we start blocking new EC2 Tempest tests, and ask to move them out-of-Tempest first. This will particularly hurt the EC2-code-in-Nova camp (which includes me) who have seemingly been given a lifeline until the next summit to prove they care about in-tree EC2 code. So I just wanted to know what does concerned people think about this problem. On solution I can see is allow tests to be added to Tempest for now, and then make the switch post-summit. I am hoping moving tests out of Tempest at-once wouldn't be a tough job (mostly tidying import statements?). Regards, Rushi Agrawal Cloud Engineer, Reliance Jio Infocomm On 5 February 2015 at 19:41, Sean Dague s...@dague.net wrote: On 02/05/2015 07:01 AM, Alexandre Levine wrote: Davanum, We've added the devstack support. It's in our stackforge repository. https://github.com/stackforge/ec2-api/tree/master/contrib/devstack Best regards, Alex Levine I've converted it to a devstack external plugin structure in this review - https://review.openstack.org/#/c/153206/ so that will make using this as simple as enable_plugin ec2-api https://github.com/stackforge/ec2-api Once that merges. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][oslo.db][nova] TL; DR Things everybody should know about Galera
On 5 February 2015 at 23:07, Clint Byrum cl...@fewbar.com wrote: Excerpts from Avishay Traeger's message of 2015-02-04 22:19:53 -0800: On Wed, Feb 4, 2015 at 11:00 PM, Robert Collins robe...@robertcollins.net wrote: On 5 February 2015 at 10:24, Joshua Harlow harlo...@outlook.com wrote: How interesting, Why are people using galera if it behaves like this? :-/ Because its actually fairly normal. In fact its an instance of point 7 on https://wiki.openstack.org/wiki/BasicDesignTenets - one of our oldest wiki pages :). When I hear MySQL I don't exactly think of eventual consistency (#7), scalability (#1), horizontal scalability (#4), etc. For the past few months I have been advocating implementing an alternative to db/sqlalchemy, but of course it's a huge undertaking. NoSQL (or even distributed key-value stores) should be considered IMO. Just some food for thought :) I know it is popular to think that MySQL* == old slow and low-scale, but that is only popular with those who have not actually tried to scale MySQL. You may want to have a chat with the people running MySQL at Google, Facebook, and a long tail of not quite as big sites but still massively bigger than most clouds. Note that many of the people who helped those companies scale up are involved directly with OpenStack. Just an aside: Youtube relies completely on MySQL for all of it's database traffic, but uses a layer on top of it called Vitess [1] to allow it to scale. [1]: https://github.com/youtube/vitess The NoSQL bits that are popular out there make the easy part easy. There is no magic bullet for the hard part, which is when you need to do both synchronous and asynchronous. Factor in its maturity and the breadth of talent available, and I'll choose MySQL for this task every time. * Please let's also give a nod to our friends working on MariaDB, a MySQL-compatible fork that many find preferrable and for the purposes of this discussion, equivalent. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][ec2-api] Tagging functionality in nova's EC2 API
whatever functionality is currently present in nova for EC2 and used by clients. EC2 API already shares database with Nova's, so the tight coupling between EC2 API and Nova's database is not going to go away till the time EC2 API server/controller is present in Nova. Nova instance metadata is being used as EC2 instance tags, and what the above-referenced spec is doing is is very similar: Cinder volume metadata is being used as EC2 volume tags, and similarly for volume snapshots. I don't see a difference between instances and volumes and volume snapshots in the sense that they all are non-share-able (yet). I completely understand that these patches look like feature additions. I started working on them first in January 2014 ( https://review.openstack.org/#/c/64690/ ), and at that time it was just a sincere effort to improve EC2 API using the first possible way I could find out. Since we have not deprecated the in-Nova EC2 support yet, and we are yet to reach a concrete plan to move forward, I am tempted to ask for allowing this patch to be considered for review.. I am fine if people think these patches shouldn't be allowed to go in. I can only wish that the patches got more attention when it was possible to get them merged :) Regards, Rushi Agrawal Best regards, Alex Levine __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance
Thanks for bringing this up. We at Reliance are building a team completely dedicated to make the EC2 API support in OpenStack rock-solid and complete. Coming from block-storage background, I submitted a few blueprints[1][2][3] for the volumes stuff, along with the code for them[4][5][6] in the Icehouse release. Unfortunately, the reviews didn't get the attention they required and they were pushed to Juno (probably, partly due to it being residing in Nova but actually being related to Cinder). I was the only person in our team for a few months to take care of efforts in this direction. The process of adding more resources has already been started, although I am expecting that it will take a month of time before they start actively contributing to upstream code. The point I am trying to make is that there are people interested in keeping this layer functional and in a healthy state. Now that we have promised in the writing that we will dedicate efforts, I think waiting till end of Icehouse is a good idea. Let this cycle be our last chance to show that we care for this piece of code :) I will make sure I make it clear to my team that making the EC2 API 'robust' is more important than making it 'feature complete'. As you can see from my contributions, I have been guilty of not following this myself in the past. Links: 1. https://blueprints.launchpad.net/nova/+spec/ec2-volume-and-snapshot-tags 2. https://blueprints.launchpad.net/nova/+spec/ec2-volume-type 3. https://blueprints.launchpad.net/nova/+spec/ec2-volume-filtering 4. [DescribeVolumes all filters](https://review.openstack.org/#/c/70085/) 5. [EC2 Volume type](https://review.openstack.org/#/c/61041/) 6. [EC2 volume tags(metadata)](https://review.openstack.org/#/c/64690/) Regards, Rushi Agrawal Cloud engineer, Reliance Jio Infocomm On 23 April 2014 17:17, Sean Dague s...@dague.net wrote: I've spent a couple of days getting to the bottom of: Bug 1302774 - Failed to detach volume because of volume not found error prevents vm teardown This is an ec2 specific failure path, which mostly looks like a combination of a not very good test case and the EC2 code in nova collapsing the volume states in a way that seems completely incorrect based on what I can read on what's expected from this call. However, these are symptoms of a bigger issue. The EC2 paths in Nova are old, fragile, and error prone. The test coverage for these paths is minimal, and largely hasn't evolved in the last year. The last substantial addition to the EC2 tests in Tempest was by Burt Holtzman in July 2013, Burt has also been contributing to the Nova side, but beyond Burt, there basically aren't contributors right now. I really don't like shipping code in Nova that we know isn't good. With very few contributions in this code though, it's defacto, if not officially, deprecated. I'd like to see if there are any more people interested in keeping these interfaces functional (by contributing both on the nova and tempest sides). If so, great! If we get to the end of Juno in the current state, I think we need to consider actually deprecating the EC2 support in Nova. Because I'm pretty sure what we have today actually only works if you are using boto on the client side, and doesn't really look like EC2 at any real level of inspection. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Regards, Rushi Agrawal Cloud engineer, Reliance Jio Infocomm, India Ph: (+91) 99 4518 4519 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][EC2][Cinder] Asking for time to review my patches
Thanks, Ben. Point noted. Sorry for the noise :) Regards, Rushi Agrawal Cloud engineer, Reliance Jio Infocomm, India Ph: (+91) 99 4518 4519 On 17 February 2014 22:13, Ben Nemec openst...@nemebean.com wrote: Rushi, please see http://lists.openstack.org/pipermail/openstack-dev/2013-September/015264.html Thanks. -Ben On 2014-02-15 07:18, Rushi Agrawal wrote: Over the last two months, I have submitted few patches which increases support of block storage volumes in OpenStack's EC2 API. The blueprints for them have been approved, and the code is ready for review. * Expose volume type https://review.openstack.org/#/c/61041/ * Expose volume tags https://review.openstack.org/#/c/64690/ * Expose volume snapshot tags https://review.openstack.org/#/c/66291/ * Expose filtering of volumes https://review.openstack.org/#/c/62350/ https://review.openstack.org/#/c/70085/ I would like to solicit some feedback from interested/affected folks, so that I can incorporate them sooner so as not to bother you people near the end of milestone :) Any help would be greatly appreciated. Thanks and regards, Rushi Agrawal Cloud engineer, Reliance Jio Infocomm, India Ph: (+91) 99 4518 4519 ___ OpenStack-dev mailing listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][EC2] attach and detach volume response status
I remember seeing the same while attaching -- return value is 'detached'. So I can confirm this is a bug. I couldn't locate a bug report for it, so I created one: https://bugs.launchpad.net/nova/+bug/1280572 Please mark it as a dup if you already have a bug report. Regards, Rushi Agrawal Ph: (+91) 99 4518 4519 On Sat, Feb 15, 2014 at 11:56 AM, wu jiang win...@gmail.com wrote: Hi, I checked the AttachVolume in AWS EC2: http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiReference-query-AttachVolume.html The status returned is 'attaching': AttachVolumeResponse xmlns=http://ec2.amazonaws.com/doc/2013-10-15/; requestId59dbff89-35bd-4eac-99ed-be587EXAMPLE/requestId volumeIdvol-1a2b3c4d/volumeId instanceIdi-1a2b3c4d/instanceId device/dev/sdh/device statusattaching/status attachTime-MM-DDTHH:MM:SS.000Z/attachTime /AttachVolumeResponse So I think it's a bug IMO.Thanks~ wingwj On Sat, Feb 15, 2014 at 11:35 AM, Rui Chen chenrui.m...@gmail.com wrote: Hi Stackers; I use Nova EC2 interface to attach a volume, attach success, but volume status is detached in message response. # euca-attach-volume -i i-000d -d /dev/vdb vol-0001 ATTACHMENT vol-0001i-000d detached This make me confusion, I think the status should be attaching or in-use. I find attach and detach volume interfaces return volume['attach_status'], but describe volume interface return volume['status'] Is it a bug? or for other considerations I do not know. Thanks ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Nova][EC2][Cinder] Asking for time to review my patches
Over the last two months, I have submitted few patches which increases support of block storage volumes in OpenStack's EC2 API. The blueprints for them have been approved, and the code is ready for review. * Expose volume type https://review.openstack.org/#/c/61041/ * Expose volume tags https://review.openstack.org/#/c/64690/ * Expose volume snapshot tags https://review.openstack.org/#/c/66291/ * Expose filtering of volumes https://review.openstack.org/#/c/62350/ https://review.openstack.org/#/c/70085/ I would like to solicit some feedback from interested/affected folks, so that I can incorporate them sooner so as not to bother you people near the end of milestone :) Any help would be greatly appreciated. Thanks and regards, Rushi Agrawal Cloud engineer, Reliance Jio Infocomm, India Ph: (+91) 99 4518 4519 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack][Cinder] No IRC channel #openstack-cinder !!!
You missed the pound sign. It is '/join #openstack-cinder' and not '/join openstack-cinder' :) Regards, Rushi Agrawal Ph: (+91) 99 4518 4519 On Thu, Jan 2, 2014 at 1:15 PM, Swapnil Kulkarni swapnilkulkarni2...@gmail.com wrote: Just got connected!!! Strange :| Thanks :) Best Regards, Swapnil On Thu, Jan 2, 2014 at 1:13 PM, Swapnil Kulkarni swapnilkulkarni2...@gmail.com wrote: Vijay, yes I'm on irc.freenode.net, the channel list is not showing #openstack-cinder, also /join openstack-cinder returns openstack-cinder :No such channel Best Regards, Swapnil On Thu, Jan 2, 2014 at 12:58 PM, atul jha stackera...@gmail.com wrote: Swapnil, This has it all https://wiki.openstack.org/wiki/IRC cheers!! On Thu, Jan 2, 2014 at 12:47 PM, Vijay Bellur vbel...@redhat.comwrote: On 01/02/2014 12:25 PM, Swapnil Kulkarni wrote: Have you tried joining this channel on irc.freenode.net? Regards, Vijay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Atul Jha http://atuljha.com (irc.freenode.net:koolhead17) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Devstack Ceph
Hi Sebastien, +1 from my side if Ceph can be installed in a single-node. I am interested in making a contribution towards this effort, but my understanding towards Ceph is only elementary at present. Regards, Rushi Agrawal, OpenStack storage engineer, Reliance Jio Infocomm Ph: (+91) 99 4518 4519 On Tue, Dec 24, 2013 at 5:19 PM, Sebastien Han sebastien@enovance.comwrote: Hello everyone, I’ve been working on a new feature for Devstack that includes a native support for Ceph. The patch includes the following: * Ceph installation (using the ceph.com repo) * Glance integration * Cinder integration (+ nova virsh secret) * Cinder backup integration * Partial Nova integration since master is currently broken. Lines are already there, the plan is to un-comment those lines later. * Everything works with Cephx (Ceph authentification system). Does anyone will be interested to see this going into Devstack mainstream? Cheers. Sébastien Han Cloud Engineer Always give 100%. Unless you're giving blood.” Phone: +33 (0)1 49 70 99 72 Mail: sebastien@enovance.com Address : 10, rue de la Victoire - 75009 Paris Web : www.enovance.com - Twitter : @enovance ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
