[openstack-dev] [nova] NLS support for database collators
Hello, We are trying to understand how the various GET REST APIs handle sorting/filtering in different NLS environments. For example, when retrieving sorted String data (ie, display name), the order of the results should vary based on the NLS of the caller (as opposed to having everything sorted in English). For Nova, the instances database has a "vm_state" column and the value is an English string (ie, "active", "error"). Is this value an NLS-key or the actual text that would be exposed to the caller? Is there any existing collator support for sorting based on locale? Links to any documentation or previous discussions would be appreciated. Thanks, Steven Kaufer___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Cinder] Get volumes REST API with filters and limit
I am investigating some pagination enhancements in nova and cinder (see nova blueprint https://blueprints.launchpad.net/nova/+spec/nova-pagination). In cinder, it appears that all filtering is done after the volumes are retrieved from the database (see the API.get_all function in https://github.com/openstack/cinder/blob/master/cinder/volume/api.py). Therefore, the usage combination of filters and limit will only work if all volumes matching the filters are in the page of data being retrieved from the database. For example, assume that all of the volumes with a name of "foo" would be retrieved from the database starting at index 100 and that you query for all volumes with a name of "foo" while specifying a limit of 50. In this case, the query would yield 0 results since the filter did not match any of the first 50 entries retrieved from the database. Is this a known problem? Is this considered a bug? How should this get resolved? As a blueprint for juno? I am new to the community and am trying to determine how this should be addressed. Thanks, Steven Kaufer___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Cinder] Get volumes REST API with filters and limit
Duncan Thomas wrote on 03/04/2014 07:53:49 AM: > From: Duncan Thomas > To: "OpenStack Development Mailing List (not for usage questions)" > , > Date: 03/04/2014 08:06 AM > Subject: Re: [openstack-dev] [Cinder] Get volumes REST API with > filters and limit > > Definitely file a bug... a script to reproduce would be fantastic. > Needs to be fixed... I don't think you need a blueprint if the fix is > simple, but if you're making deep changes then a blueprint always > helps.# Bug created: https://bugs.launchpad.net/cinder/+bug/1287813 > > Thanks for pointing this out > > On 28 February 2014 20:52, Steven Kaufer wrote: > > I am investigating some pagination enhancements in nova and cinder(see nova > > blueprint https://blueprints.launchpad.net/nova/+spec/nova-pagination). > > > > In cinder, it appears that all filtering is done after the volumes are > > retrieved from the database (see the API.get_all function in > > https://github.com/openstack/cinder/blob/master/cinder/volume/api.py). > > Therefore, the usage combination of filters and limit will only work if all > > volumes matching the filters are in the page of data being retrieved from > > the database. > > > > For example, assume that all of the volumes with a name of "foo" would be > > retrieved from the database starting at index 100 and that you query for all > > volumes with a name of "foo" while specifying a limit of 50. In this case, > > the query would yield 0 results since the filter did not match any of the > > first 50 entries retrieved from the database. > > > > Is this a known problem? > > Is this considered a bug? > > How should this get resolved? As a blueprint for juno? > > > > I am new to the community and am trying to determine how this should be > > addressed. > > > > Thanks, > > > > Steven Kaufer > > > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > -- > Duncan Thomas > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > Thanks, Steven Kaufer___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [api] Counting resources
The neturon spec appears to be a copy/paste of the nova spec that I wrote.
Based on the conversation below, I agree that this is not the best
approach: GET ///count
I'll get started on updating the nova spec to include the total count value
in some new attribute based on the existence of a query parameter (ie,
include_count=1).
The details will have to be in limbo a bit until this gets resolved:
https://review.openstack.org/#/c/133660/
Thanks,
Steven Kaufer
Sean Dague wrote on 11/20/2014 10:48:05 AM:
> From: Sean Dague
> To: openstack-dev@lists.openstack.org
> Date: 11/20/2014 10:57 AM
> Subject: Re: [openstack-dev] [api] Counting resources
>
> I'm looking at the Nova spec, and it seems very taylored to a specific
> GUI. I'm also not sure that 17128 errors is more useful than 500+ errors
> when presenting to the user (the following in my twitter stream made me
> think about that this morning -
> https://twitter.com/NINK/status/535299029383380992)
>
> 500+ also better describes the significant figures we're talking about
here.
>
>-Sean
>
> On 11/20/2014 11:28 AM, Morgan Fainberg wrote:
> > The only thing I want to caution against is making a SQL-specific
> > choice. In the case of some other backends, it may not be possible (for
> > an extremely large dataset) to get a full count, where SQL does this
> > fairly elegantly. For example, LDAP (in some cases) may have an
> > administrative limit that will say that no more than 10,000 entries
> > would be returned; likely you’re going to have an issue, since you need
> > to issue the query and see how many things match, if you hit the
overall
> > limit you’ll get the same count every time (but possibly a different
> > dataset).
> >
> > I want to be very careful that we’re not recommending functionality as
a
> > baseline that should be used as a pattern across all similar APIs,
> > especially since we have some backends/storage systems that can’t
> > elegantly always support it.
> >
> > Personally, I like Gerrit’s model (as Sean described) - with the above
> > caveat that not all backends support this type of count.
> >
> > Cheers,
> > Morgan
> >
> >> On Nov 20, 2014, at 8:04 AM, Salvatore Orlando >> <mailto:sorla...@nicira.com>> wrote:
> >>
> >> The Nova proposal appears to be identical to neutron's, at least from
> >> a consumer perspective.
> >>
> >> If I were to pick a winner, I'd follow Sean's advice regarding the
> >> 'more' attribute in responses, and put the total number of resources
> >> there; I would also take Jay's advice of including the total only if
> >> requested with a query param. In this way a user can retrieve the
> >> total number of items regardless of the current pagination index (in
> >> my first post I suggested the total number should be returned only on
> >> the first page of results).
> >>
> >> Therefore one could ask for a total number of resources with something
> >> like the following:
> >>
> >> GET /some_resources?include_total=1
> >>
> >> and obtain a response like the following:
> >>
> >> {'resources': [{meh}, {meh}, {meh_again}],
> >> 'something': {
> >>'_links': {'prev': ..., 'next': ...},
> >>'total': agazillion}
> >> }
> >>
> >> where the exact structure and naming of 'something' depends on the
> >> outcome of the discussion at [1]
> >>
> >> Salvatore
> >>
> >> [1] https://review.openstack.org/#/c/133660/7/guidelines/
> representation_structure.rst
> >>
> >> On 20 November 2014 15:24, Christopher Yeoh >> <mailto:cbky...@gmail.com>> wrote:
> >>
> >> On Thu, 20 Nov 2014 14:47:16 +0100
> >> Salvatore Orlando >> <mailto:sorla...@nicira.com>> wrote:
> >>
> >> > Aloha guardians of the API!
> >> >
> >> > I haven recently* reviewed a spec for neutron [1] proposing a
> >> > distinct URI for returning resource count on list operations.
> >> > This proposal is for selected neutron resources, but I believe
the
> >> > topic is general enough to require a guideline for the API
working
> >> > group. Your advice is therefore extremely valuable.
> >> >
> >> > In a nutshell the proposal is to retrieve resource count in the
> >> > following way:
> >> &
Re: [openstack-dev] [api] Counting resources
This is a follow up to this thread from a few weeks ago: https://www.mail-archive.com/openstack-dev@lists.openstack.org/msg40287.html I've updated the nova spec in this area to include the total server count in the server_links based on the existence of an "include_count" query parameter (eg: GET /servers?include_count=1). The spec no longer references a GET /servers/count API. Nova spec: https://review.openstack.org/#/c/134279/ Thanks, Steven Kaufer___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [python-cinderclient] Supported client-side sort keys
The cinder client supports passing a sort key via the --sort_key argument. The client restricts the sort keys that the user can supply to the following: https://github.com/openstack/python-cinderclient/blob/master/cinderclient/v2/volumes.py#L28-L29 This list of sort keys is not complete. As far I know, all attributes on this class are valid: https://github.com/openstack/cinder/blob/master/cinder/db/sqlalchemy/models.py#L104 I noticed that the 'name' key is incorrect and it should instead be 'display_name'. Before I create a bug/fix to address this, I have the following questions: Does anyone know the rational behind the client restricting the possible sort keys? Why not allow the user to supply any sort key (assuming that invalid keys are gracefully handled)? Note, if you try this out at home, you'll notice that the client table is not actually sorted, fixed under: https://review.openstack.org/#/c/141964/ Thanks, Steven Kaufer___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [cinder] [nova] [glance] Consistency in client side sorting
The nova, cinder, and glance REST APIs support listing instances, volumes, and images in a specific order. In general, the REST API supports something like: ?sort_key=key1&sort_dir=asc&sort_key=key2&sort_dir=desc This sorts the results using 'key1' as the primary key (in ascending order), 'key2' as the secondary key (in descending order), etc. Note that this behavior is not consistent across the projects. Nova supports multiple sort keys and multiple sort directions, glance supports multiple sort keys but a single direction, and cinder only supports a single sort key and a single sort direction (approved kilo BP to support multiple sort keys and directions is here: https://blueprints.launchpad.net/cinder/+spec/cinder-pagination). The purpose of this thread is to discuss how the sort information should be inputted to the client. In nova, (committed in kilo https://review.openstack.org/#/c/117591/) the syntax is: --sort key1:asc,key2:desc In cinder, the syntax is: --sort_key key1 --sort_dir desc In glance, the proposed syntax (from https://review.openstack.org/#/c/120777/) is: --sort-key key1 --sort-key key2 --sort-dir desc Note that the keys are different for cinder and glance (--sort_key vs. --sort-key). Also, client side sorting does not actually work in cinder (fix under review at https://review.openstack.org/#/c/141964/). Giving that each of these 3 clients will be supporting client-side sorting in kilo, it seems that we should get this implemented in a consistent manner. It seems that the 2 options are either: --sort-key key1 --sort-dir desc --sort-key key2 --sort-dir asc --sort key1:asc,key2:desc Personally, I favor option 2 but IMO it is more important that these are made consistent. Thoughts on getting consistency across all 3 projects (and possibly others)? Thanks, Steven Kaufer___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [api] [sdk] Proposal to achieve consistency in client side sorting
This is a follow up thread to [1] In order to have consistency across clients, I am proposing that the client side sorting has the following syntax: --sort [:] Where the --sort parameter is comma-separated and is used to specify one or more sort keys and directions. The direction defaults to 'desc' for each sort key and the user can supply 'asc' to override. For example: nova list --sort display_name nova list --sort display_name,vm_state nova list --sort display_name:asc,vm_state:asc If approved, then the following changes are needed for glance and cinder (note that nova already uses this syntax): Cinder: Deprecate --sort_key and --sort_dir and add support for --sort Glance: Modify [2] to this new syntax I have not verified how all other projects handle sorting, there may be other projects that would also need to be changed. Thoughts? Objections? Also, if there is a more formal way to propose/review this standard please let me know. [1] http://www.mail-archive.com/openstack-dev@lists.openstack.org/msg42854.html [2] https://review.openstack.org/#/c/120777/ Thanks, Steven Kaufer___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [api] [sdk] Proposal to achieve consistency in client side sorting
Sean Dague wrote on 01/07/2015 06:21:52 AM: > From: Sean Dague > To: openstack-dev@lists.openstack.org > Date: 01/07/2015 06:22 AM > Subject: Re: [openstack-dev] [api] [sdk] Proposal to achieve > consistency in client side sorting > > On 01/06/2015 09:37 PM, Rochelle Grober wrote: > > Steven, > > > > > > > > This sounds like a perfect place for a cross project spec. It wouldn’t > > have to be a big one, but all the projects would have a chance to review > > and the TC would oversee to ensure it gets proper review. > > > > > > > > TCms, am I on point here? > > Yes, this sounds reasonable. It would be a general CLI guidelines spec > which we could expand over time to include common patterns that we > prefer CLIs use when interfacing with their users. > >-Sean Thanks for the feedback. Spec up for review at: https://review.openstack.org/#/c/145544/ Thanks, Steven Kaufer > > -- > Sean Dague > http://dague.net > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Globalization] [horizon] REST API sorting by status severity vs. alphabetical status key
Adding [horizon] to the title line to get feedback from the Horizon team.
Thanks,
Steven Kaufer
Hi Steven, thx for the detailed email. Some comments inline...
On Wed, 2014-04-23 at 13:59 -0500, Steven Kaufer wrote:
> I am trying to address the following use case:
>
> - Assume that the REST APIs support returning data based on a
> user-defined sort key (assuming that this get approved:
> https://review.openstack.org/#/c/84451/)
> - UI contains a table showing items (servers, volumes, etc.) and their
> status (as a sortable column) and uses pagination to get only a page
> of data
> - UI is translated into a non-English language
> - User wants to sort the table by status
>
> In this case, the sorting by status is done against the English key
> values in the database (active, error, etc.). The UI will then
> translate the status values into the user's locale and (from the
> user's perspective) the data will not be in sorted order -- thus
> confusing and frustrating the user. Note that UI cannot do the sort
> client-side since pagination is used so it the client only has a
> sub-set of the total data.
Indeed, this is a problem.
> I have prototyped a "sort by case" solution that would allow status to
> be sorted by severity. In SQL the "case" statement can be used to map
> a string to an int and then sort the rows based on the int value (ie,
> error=0, building=1, active=2, etc.). Using this approach, sorting by
> status would result in an enum-like sort (based on severity) instead
> of an alphabetical sort based on the English key values. This
> solution allows enum-like data to be sorted in a consistent way across
> all locales -- the solution is generic and can be applied to any
> column where the values are a known set.
>
> The case processing would need to be done in the common paginate_query
> function:
>
https://github.com/openstack/oslo-incubator/blob/master/openstack/common/db/sqlalchemy/utils.py#L62
>
> This type of sort would not be the default behavior for a status
> column (or any enum-like column) and the caller would need to specify
> a unique sort direction key for it (ie, 'asc_case' or 'desc_case').
> In theory, this type of sorting support could also be globally
> enabled/disabled by a deployer (default would be disabled) to further
> reduce impact.
>
> Lastly, I have some performance data and sorting the status by case
> (vs. alphabetical) has a minimal impact on performance.
>
> Before I create a proposal for juno I wanted to get some early
> feedback on the high-level approach. Please reply with feedback on
> this solution.
So, I feel that the above solution (while innovative certainly! :) ) is
not actually addressing the underlying source of the problem here, and
that is that statuses are stored in the database as English-language
strings instead of integer code values in a lookup table.
By addressing the underlying source of the problem -- by changing the
instances.{vm,task,power}_state columns to an integer value and using a
lookup table in the cases when translation from code to display is
needed -- we both solve the problem of i18n sorting and increase the
database performance, since queries on these state columns will use an
index on a datatype with a much slimmer width.
Best,
-jay
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Globalization] REST API sorting by status severity vs. alphabetical status key
Jay, Thanks again for the reply. If this migration is implemented using the object "versioning", then the new "status as int" column cannot be utilized (ie, sorted on) until the existing "status as string" column is eventually dropped. Is this correct? If so, then this approach will not actually solve the globalization sort problem until more release cycles have completed -- this does not seem like a viable solution. Until we know that the new "status as int" column is populated then we cannot use it as a sortable column. In theory, a deployer could conditionally choose to migrate to the new column if they needed that function and were willing to take the hit during the migration. However, this just complicates the sorting logic since we would then need to know which column to use during the sort (the new int column if the migration has completed or the old string column if the migration has not completed). Thanks, Steven Kaufer Jay Pipes wrote on 04/28/2014 09:05:51 AM: > From: Jay Pipes > To: openstack-dev@lists.openstack.org, > Date: 04/28/2014 09:07 AM > Subject: Re: [openstack-dev] [Globalization] REST API sorting by > status severity vs. alphabetical status key > > On Wed, 2014-04-23 at 22:07 -0500, Steven Kaufer wrote: > > > yeah, we're talking about thousands and thousands of rows that have > > to > > > be updated before the API can be restarted… > > > > > > > There's also a possibility of adding support for the status codes, > > but > > > > keeping the string columns in the database, and then using the > > nova > > > > object versioning to "migrate" the object schema over time to the > > point > > > > where the migration is a simple DROP COLUMN. > > > > > > I like that idea better, TBH, but we're probably talking about a > > > long-time deprecation here, like on the order of a couple of > > releases; > > > that would give plenty of time for the majority of the records to be > > > revisited and make the final migration run for a lot shorter time. > > > -- > > > > Thanks for the discussion. > > No prob, sorry for the delayed response... > > > So how would this new flow work? > > In Juno would there be an additional status_int column that would be > > populated and (eventually) replace the existing status (as string) > > column? > > That would be the cleanest way, yes. > > > How would the object versioning populate the new column for the > > existing records? > > Within the nova.objects.instance.Instance object itself, we can put a > small check-and-transform function in the object to do the translation > in-line. > > > Any examples or details that would help explain how this could work > > would be appreciated. > > Probably worth putting a blueprint up about it. I can work with you on > it, if you'd like, though it will likely be after the summit until I > have time to work on it. > > > Lastly, is there agreement that this is an issue that needs to be > > addressed? Note that this seems to be a pervasive problem, I've > > investigated the status column in cinder and nova but I suspect that > > the same issue exists in other components. > > Yes, the same issue unfortunately exists in lots of the other > components, and they don't have the benefit of the nova objects work in > them, which makes it a lot more of a nuisance to migrate the database > schema. Though, personally I'm not entirely sure going through the > effort of doing a long-time in-object translation is worth it. A change > to the database schema, even on tens of millions of records wouldn't > take more than a couple minutes. But it all depends on the operator's > tolerance for downtime, since the instances table would certainly be > locked for the duration of the migration. > > Best, > -jay > > > Thanks, > > Steven Kaufer > > > > > > > Kevin L. Mitchell > > > Rackspace > > > > > > > > > ___ > > > OpenStack-dev mailing list > > > OpenStack-dev@lists.openstack.org > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Cinder] cinder not support query volume/snapshot with regular expression
Jay Pipes wrote on 04/29/2014 02:26:42 PM: > From: Jay Pipes > To: openstack-dev@lists.openstack.org, > Date: 04/29/2014 02:27 PM > Subject: Re: [openstack-dev] [Cinder] cinder not support query > volume/snapshot with regular expression > > On 04/29/2014 02:16 AM, Zhangleiqiang (Trump) wrote: > > Currently, Nova API achieve this feature based on the database’s REGEX > > support. Do you have advice on alternative way to achieve it? > > Hi Trump, > > Unfortunately, REGEXP support in databases is almost always ridiculously > slow compared to prefix searches (WHERE col LIKE 'foo%'). > > Lately, I've been considering that a true tagging system for Nova would > allow for better-performing and more user-friendly search/winnow > functions in the Nova API. I'll post a blueprint specification for this > and hopefully have some time to implement in Juno... Jay, I am interested in this design, please add me as a reviewer when the blueprint is created. Thanks! Steven Kaufer > > Best, > -jay > > > zhangleiqiang (Trump) > > > > Best Regards > > > > *From:*laserjetyang [mailto:laserjety...@gmail.com] > > *Sent:* Tuesday, April 29, 2014 1:49 PM > > *To:* OpenStack Development Mailing List (not for usage questions) > > *Subject:* Re: [openstack-dev] [Cinder] cinder not support query > > volume/snapshot with regular expression > > > > It looks to me the Nova API will be dangerous source of DoS attacks due > > to the regexp? > > > > On Mon, Apr 28, 2014 at 7:04 PM, Duncan Thomas > <mailto:duncan.tho...@gmail.com>> wrote: > > > > Regex matching in APIs can be a dangerous source of DoS attacks - see > > http://en.wikipedia.org/wiki/ReDoS. Unless this is mitigated sensibly, > > I will continue to resist any cinder patch that adds them. > > > > Glob matches might be safer? > > > > > > On 26 April 2014 05:02, Zhangleiqiang (Trump) > <mailto:zhangleiqi...@huawei.com>> wrote: > >> Hi, all: > >> > >> I see Nova allows search instances by name, ip and ip6 > fields which can be normal string and regular expression: > >> > >> [stack@leiqzhang-stack cinder]$ nova help list > >> > >> List active servers. > >> > >> Optional arguments: > >> --ip Search with regular > expression match by IP address > >> (Admin only). > >> --ip6 Search with regular > expression match by IPv6 address > >> (Admin only). > >> --name Search with regular > expression match by name > >> --instance-name Search with regular > expression match by server name > >> (Admin only). > >> > >> I think it is also needed for Cinder when query the > volume/snapshot/backup by name. Any advice? > >> > >> -- > >> zhangleiqiang (Trump) > >> > >> Best Regards > >> > >> > >> ___ > >> OpenStack-dev mailing list > >>OpenStack-dev@lists.openstack.org <mailto:OpenStack-dev@lists.openstack.org> > >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > -- > > Duncan Thomas > > > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org <mailto:OpenStack-dev@lists.openstack.org> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Duplicate code for processing REST APIs
While investigating some REST API updates I've discovered that there is a lot of duplicated code across the various OpenStack components. For example, the paginate_query function exists in all these locations and there are a few slight differences between most of them: https://github.com/openstack/ceilometer/blob/master/ceilometer/openstack/common/db/sqlalchemy/utils.py#L61 https://github.com/openstack/cinder/blob/master/cinder/openstack/common/db/sqlalchemy/utils.py#L37 https://github.com/openstack/glance/blob/master/glance/openstack/common/db/sqlalchemy/utils.py#L64 https://github.com/openstack/heat/blob/master/heat/openstack/common/db/sqlalchemy/utils.py#L62 https://github.com/openstack/keystone/blob/master/keystone/openstack/common/db/sqlalchemy/utils.py#L64 https://github.com/openstack/neutron/blob/master/neutron/openstack/common/db/sqlalchemy/utils.py#L61 https://github.com/openstack/nova/blob/master/nova/openstack/common/db/sqlalchemy/utils.py#L64 Does anyone know if there is any work going on to move stuff like this into oslo and then deprecate these functions? There are also many functions that process the REST API request parameters (getting the limit, marker, sort data, etc.) that are also replicated across many components. If no existing work is done in this area, how should this be tackled? As a blueprint for Juno? Thanks, Steven Kaufer Cloud Systems Software kau...@us.ibm.com 507-253-5104 Dept HMYS / Bld 015-2 / G119 / Rochester, MN 55901___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Offset support in REST API pagination
First, here is some background on this topic: http://www.gossamer-threads.com/lists/openstack/dev/2777 Does anyone have any insight as to why offset is not supported in the REST API calls that support pagination? I realize that there are tradeoffs when using a offset (vs. marker) but I believe that there is value in supporting both. For example, if you want to jump to the n-th page of data without having to traverse all of the previous pages. Is there a reason why the APIs do not support either a marker or an offset (obviously not both) on the API request? It appears that sqlalchemy has offset support. Also, it seems that cinder at least looks for the offset parameter (but ignores it). Does this mean that it was supported at one time but later the support was removed? https://github.com/openstack/cinder/blob/master/cinder/api/v2/volumes.py#L214 Thanks for the information. Steven Kaufer___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Offset support in REST API pagination
Jay Pipes wrote on 03/18/2014 12:02:50 PM: > From: Jay Pipes > To: openstack-dev@lists.openstack.org, > Date: 03/18/2014 12:15 PM > Subject: Re: [openstack-dev] Offset support in REST API pagination > > On Tue, 2014-03-18 at 11:31 -0500, Steven Kaufer wrote: > > First, here is some background on this topic: > > http://www.gossamer-threads.com/lists/openstack/dev/2777 > > > > Does anyone have any insight as to why offset is not supported in the > > REST API calls that support pagination? I realize that there are > > tradeoffs when using a offset (vs. marker) but I believe that there is > > value in supporting both. For example, if you want to jump to the > > n-th page of data without having to traverse all of the previous > > pages. > > > > Is there a reason why the APIs do not support either a marker or an > > offset (obviously not both) on the API request? It appears that > > sqlalchemy has offset support. > > > > Also, it seems that cinder at least looks for the offset parameter > > (but ignores it). Does this mean that it was supported at one time > > but later the support was removed? > > https://github.com/openstack/cinder/blob/master/cinder/api/v2/ > volumes.py#L214 > > > > Thanks for the information. > > Hail to thee, stranger! Thou hast apparently not passed into the cave of > marker/offset before! > > I humbly direct you to buried mailing list treasures which shall > enlighten you! > > This lengthy thread shall show you how yours truly was defeated in > written combat by the great Justin Santa Barbara, who doth exposed the > perils of the offset: > > http://www.gossamer-threads.com/lists/openstack/dev/2803 > > A most recent incantation of the marker/offset wars is giveth here: > > http://lists.openstack.org/pipermail/openstack-dev/2013-November/018861.html > > Best of days to you, > -jay Jay: Thanks for the feedback and the history on this topic. I understand that the limit/marker approach is superior when simply traversing all of the pages. However, consider the following: - User knows that there are 1000 items (VMs, volumes, images, really doesn't matter) - User knows that the item that they want is in roughly the middle of the data set (assume everything is sorted by display name) - User cannot remember the exact name so a filter will not help and changing the sort direction will not help (since the item they want it is in the middle of the dataset) - User supplies an offset of 500 to jump into the middle of the data set - User then uses the marker approach to traverse the pages from this point to find the item that they want In this case the offset approach is not used to traverse pages so there are no issues with missing an item or seeing a duplicate. Why couldn't the APIs support either marker or offset on a given request? Also, to encourage the use of marker instead of offset, the next/previous links on any request with an offset supplied should contain the appropriate marker key values -- this should help discourage simply increasing the offset when traversing the pages. I realize that if only one solution had to be chosen, then limit/marker would always win this war. But why can't both be supported? Thanks, Steven Kaufer > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Cinder] [Nova] Blueprint review request for pagination sorting enhancements
Please review the following blueprints, both are scoped to supporting multiple sort key and sort directions on the API request when retrieving volumes and servers. https://blueprints.launchpad.net/cinder/+spec/cinder-pagination https://blueprints.launchpad.net/nova/+spec/nova-pagination Note that I will be updating the nova blueprint to adhere to the new nova-specs process once that is ready. In the mean time, the specification wiki page has been updated with the information from the template. Any comments, questions, and recommendations would be appreciated. Thanks, Steven Kaufer___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Support for multiple sort keys and sort directions in REST GET APIs
I have proposed blueprints in both nova and cinder for supporting multiple sort keys and sort directions for the GET APIs (servers and volumes). I am trying to get feedback from other projects in order to have a more uniform API across services. Problem description from nova proposal: "There is no support for retrieving server data in a specific order, it is defaulted to descending sort order by the "created date" and "id" keys. In order to retrieve data in any sort order and direction, the REST APIs need to accept multiple sort keys and directions. Use Case: A UI that displays a table with only the page of data that it has retrieved from the server. The items in this table need to be sorted by status first and by display name second. In order to retrieve data in this order, the APIs must accept multiple sort keys/directions." See nova proposal .rst file (cinder is basically the same) for more information: https://review.openstack.org/#/c/84451/ Most projects have similar GET requests and I am trying to get some consensus on this approach across the various projects; the goal is to have this type of functionality common across projects (not just nova and cinder). Note that some projects (ie, cinder) already support a single sort key and sort direction, see https://github.com/openstack/cinder/blob/master/cinder/api/v2/volumes.py#L212-L213 Note that the DB layer already accepts multiple sort keys and sort directions (see https://github.com/openstack/oslo-incubator/blob/master/openstack/common/db/sqlalchemy/utils.py#L62), the work I am describing here only exposes the sorting options at the REST API layer. Please provide feedback on this direction. Specifically, do you see any issues (and, if so, why) with allowing the caller to specify sort orders and directions on the GET APIs? Feel free to leave your feedback in the Gerrit review for the nova blueprint or reply to this thread. Thanks, Steven Kaufer___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Support for multiple sort keys and sort directions in REST GET APIs
Duncan, Thanks for the reply. The sorting is done in the common sqlalchemy.utils.paginate_query function, which takes an ORM model class as an argument (https://github.com/openstack/oslo-incubator/blob/master/openstack/common/db/sqlalchemy/utils.py#L82). The only valid sort columns are attributes on the given model class. For example, In cinder that class is "models.Volume": https://github.com/openstack/cinder/blob/master/cinder/db/sqlalchemy/api.py#L1331 In nova that class is "models.Instance": https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L1946 In glance that class is "models.Image"; https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/api.py#L540 This limits the scope of what can be sorted on (ie, it cannot be any attribute that exists on an item returned from a detailed query). The blueprint is lacking this level of detail and I will update it accordingly. Does this address your concern? Thanks, Steven Kaufer Duncan Thomas wrote on 04/03/2014 05:18:47 AM: > From: Duncan Thomas > To: "OpenStack Development Mailing List (not for usage questions)" > , > Date: 04/03/2014 05:25 AM > Subject: Re: [openstack-dev] Support for multiple sort keys and sort > directions in REST GET APIs > > Some of the cinder APIs do weird database joins and double lookups and > things, making every field sortable might have some serious database > performance impact and open up a DoS attack. Will need more > investigation to be sure. > > On 2 April 2014 19:42, Steven Kaufer wrote: > > I have proposed blueprints in both nova and cinder for supporting multiple > > sort keys and sort directions for the GET APIs (servers and volumes). I am > > trying to get feedback from other projects in order to have a more uniform > > API across services. > > > > Problem description from nova proposal: > > > > "There is no support for retrieving server data in a specific order, it is > > defaulted to descending sort order by the "created date" and "id" keys. In > > order to retrieve data in any sort order and direction, the REST APIs need > > to accept multiple sort keys and directions. > > > > Use Case: A UI that displays a table with only the page of data that it has > > retrieved from the server. The items in this table need to be sorted by > > status first and by display name second. In order to retrieve data in this > > order, the APIs must accept multiple sort keys/directions." > > > > See nova proposal .rst file (cinder is basically the same) for more > > information: https://review.openstack.org/#/c/84451/ > > > > Most projects have similar GET requests and I am trying to get some > > consensus on this approach across the various projects; the goal is to have > > this type of functionality common across projects (not just nova and > > cinder). Note that some projects (ie, cinder) already support a single sort > > key and sort direction, see > > https://github.com/openstack/cinder/blob/master/cinder/api/v2/ > volumes.py#L212-L213 > > > > Note that the DB layer already accepts multiple sort keys and sort > > directions (see > > https://github.com/openstack/oslo-incubator/blob/master/openstack/ > common/db/sqlalchemy/utils.py#L62), > > the work I am describing here only exposes the sorting options at the REST > > API layer. > > > > Please provide feedback on this direction. Specifically, do you see any > > issues (and, if so, why) with allowing the caller to specify sort orders and > > directions on the GET APIs? > > > > Feel free to leave your feedback in the Gerrit review for the novablueprint > > or reply to this thread. > > > > Thanks, > > > > Steven Kaufer > > > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > -- > Duncan Thomas > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Support for multiple sort keys and sort directions in REST GET APIs
I have submitted a session for the Juno summit for this work: http://summit.openstack.org/cfp/details/265 Thanks, Steven Kaufer Duncan Thomas wrote on 04/06/2014 01:21:57 AM: > From: Duncan Thomas > To: "OpenStack Development Mailing List (not for usage questions)" > , > Date: 04/06/2014 01:28 AM > Subject: Re: [openstack-dev] Support for multiple sort keys and sort > directions in REST GET APIs > > Stephen > > Mike is right, it is mostly (possibly only?) extensions that do double > lookups. Your plan looks sensible, and definitely useful. I guess I'll > see if I can actually break it once the review is up :-) I mostly > wanted to give a heads-up - there are people who are way better at > reviewing this than me. > > > > On 3 April 2014 19:15, Mike Perez wrote: > > Duncan, I think the point you raise could happen even without thischange. In > > the example of listing volumes, you would first query for the list in some > > multi-key sort. The API extensions for example that add additional response > > keys will do another lookup on that resource for the appropriate column it's > > retrieving. There are some extensions that still do this unfortunately, but > > quite a few got taken care of in Havana in using cache instead of > doing these > > wasteful lookups. > > > > Overall Steven, I think this change is useful, especially from one of the > > Horizon sessions I heard in Hong Kong for filtering/sorting. > > > > -- > > Mike Perez > > > > On 11:18 Thu 03 Apr , Duncan Thomas wrote: > >> Some of the cinder APIs do weird database joins and double lookups and > >> things, making every field sortable might have some serious database > >> performance impact and open up a DoS attack. Will need more > >> investigation to be sure. > >> > >> On 2 April 2014 19:42, Steven Kaufer wrote: > >> > I have proposed blueprints in both nova and cinder for > supporting multiple > >> > sort keys and sort directions for the GET APIs (servers and > volumes). I am > >> > trying to get feedback from other projects in order to have a > more uniform > >> > API across services. > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > -- > Duncan Thomas > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Globalization] REST API sorting by status severity vs. alphabetical status key
I am trying to address the following use case: - Assume that the REST APIs support returning data based on a user-defined sort key (assuming that this get approved: https://review.openstack.org/#/c/84451/) - UI contains a table showing items (servers, volumes, etc.) and their status (as a sortable column) and uses pagination to get only a page of data - UI is translated into a non-English language - User wants to sort the table by status In this case, the sorting by status is done against the English key values in the database (active, error, etc.). The UI will then translate the status values into the user's locale and (from the user's perspective) the data will not be in sorted order -- thus confusing and frustrating the user. Note that UI cannot do the sort client-side since pagination is used so it the client only has a sub-set of the total data. I have prototyped a "sort by case" solution that would allow status to be sorted by severity. In SQL the "case" statement can be used to map a string to an int and then sort the rows based on the int value (ie, error=0, building=1, active=2, etc.). Using this approach, sorting by status would result in an enum-like sort (based on severity) instead of an alphabetical sort based on the English key values. This solution allows enum-like data to be sorted in a consistent way across all locales -- the solution is generic and can be applied to any column where the values are a known set. The case processing would need to be done in the common paginate_query function: https://github.com/openstack/oslo-incubator/blob/master/openstack/common/db/sqlalchemy/utils.py#L62 This type of sort would not be the default behavior for a status column (or any enum-like column) and the caller would need to specify a unique sort direction key for it (ie, 'asc_case' or 'desc_case'). In theory, this type of sorting support could also be globally enabled/disabled by a deployer (default would be disabled) to further reduce impact. Lastly, I have some performance data and sorting the status by case (vs. alphabetical) has a minimal impact on performance. Before I create a proposal for juno I wanted to get some early feedback on the high-level approach. Please reply with feedback on this solution. Thanks, Steven Kaufer___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Globalization] REST API sorting by status severity vs. alphabetical status key
Jay,
Thanks for the reply. I agree that changing the datamodel would be the
ideal solution. But, to be honest, the scope of that change frightens me.
How would you recommend that a change like this would be handled (in
addition to the DB migration work)? We obviously cannot break existing
codepaths that assume that the existing English key values would be
returned from the DB.
Is there an existing layer that would perform the mapping between the enum
values in the DB and the String keys?
Thanks,
Steven Kaufer
Jay Pipes wrote on 04/23/2014 02:56:14 PM:
> From: Jay Pipes
> To: openstack-dev@lists.openstack.org,
> Date: 04/23/2014 02:56 PM
> Subject: Re: [openstack-dev] [Globalization] REST API sorting by
> status severity vs. alphabetical status key
>
> Hi Steven, thx for the detailed email. Some comments inline...
>
> On Wed, 2014-04-23 at 13:59 -0500, Steven Kaufer wrote:
> > I am trying to address the following use case:
> >
> > - Assume that the REST APIs support returning data based on a
> > user-defined sort key (assuming that this get approved:
> > https://review.openstack.org/#/c/84451/)
> > - UI contains a table showing items (servers, volumes, etc.) and their
> > status (as a sortable column) and uses pagination to get only a page
> > of data
> > - UI is translated into a non-English language
> > - User wants to sort the table by status
> >
> > In this case, the sorting by status is done against the English key
> > values in the database (active, error, etc.). The UI will then
> > translate the status values into the user's locale and (from the
> > user's perspective) the data will not be in sorted order -- thus
> > confusing and frustrating the user. Note that UI cannot do the sort
> > client-side since pagination is used so it the client only has a
> > sub-set of the total data.
>
> Indeed, this is a problem.
>
> > I have prototyped a "sort by case" solution that would allow status to
> > be sorted by severity. In SQL the "case" statement can be used to map
> > a string to an int and then sort the rows based on the int value (ie,
> > error=0, building=1, active=2, etc.). Using this approach, sorting by
> > status would result in an enum-like sort (based on severity) instead
> > of an alphabetical sort based on the English key values. This
> > solution allows enum-like data to be sorted in a consistent way across
> > all locales -- the solution is generic and can be applied to any
> > column where the values are a known set.
> >
> > The case processing would need to be done in the common paginate_query
> > function:
> > https://github.com/openstack/oslo-incubator/blob/master/
> openstack/common/db/sqlalchemy/utils.py#L62
> >
> > This type of sort would not be the default behavior for a status
> > column (or any enum-like column) and the caller would need to specify
> > a unique sort direction key for it (ie, 'asc_case' or 'desc_case').
> > In theory, this type of sorting support could also be globally
> > enabled/disabled by a deployer (default would be disabled) to further
> > reduce impact.
> >
> > Lastly, I have some performance data and sorting the status by case
> > (vs. alphabetical) has a minimal impact on performance.
> >
> > Before I create a proposal for juno I wanted to get some early
> > feedback on the high-level approach. Please reply with feedback on
> > this solution.
>
> So, I feel that the above solution (while innovative certainly! :) ) is
> not actually addressing the underlying source of the problem here, and
> that is that statuses are stored in the database as English-language
> strings instead of integer code values in a lookup table.
>
> By addressing the underlying source of the problem -- by changing the
> instances.{vm,task,power}_state columns to an integer value and using a
> lookup table in the cases when translation from code to display is
> needed -- we both solve the problem of i18n sorting and increase the
> database performance, since queries on these state columns will use an
> index on a datatype with a much slimmer width.
>
> Best,
> -jay
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Globalization] REST API sorting by status severity vs. alphabetical status key
> yeah, we're talking about thousands and thousands of rows that have to > be updated before the API can be restarted… > > > There's also a possibility of adding support for the status codes, but > > keeping the string columns in the database, and then using the nova > > object versioning to "migrate" the object schema over time to the point > > where the migration is a simple DROP COLUMN. > > I like that idea better, TBH, but we're probably talking about a > long-time deprecation here, like on the order of a couple of releases; > that would give plenty of time for the majority of the records to be > revisited and make the final migration run for a lot shorter time. > -- Thanks for the discussion. So how would this new flow work? In Juno would there be an additional status_int column that would be populated and (eventually) replace the existing status (as string) column? How would the object versioning populate the new column for the existing records? Any examples or details that would help explain how this could work would be appreciated. Lastly, is there agreement that this is an issue that needs to be addressed? Note that this seems to be a pervasive problem, I've investigated the status column in cinder and nova but I suspect that the same issue exists in other components. Thanks, Steven Kaufer > Kevin L. Mitchell > Rackspace > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] Marker based paging
Here are a few ML threads on this topic: http://lists.openstack.org/pipermail/openstack-dev/2014-March/030322.html http://www.gossamer-threads.com/lists/openstack/dev/2777 http://lists.openstack.org/pipermail/openstack-dev/2013-November/018861.html Thanks, Steven Kaufer Roman Podoliaka wrote on 11/03/2014 10:35:21 AM: > From: Roman Podoliaka > To: "OpenStack Development Mailing List (not for usage questions)" > > Date: 11/03/2014 10:43 AM > Subject: Re: [openstack-dev] [oslo.db] Marker based paging > > Hi Mike, > > I think that code was taken from Nova (or maybe some other project) as > is and we haven't touched it since then. > > Please speak up - we want to know about all possible problems with > current approach. > > Thanks, > Roman > > On Fri, Oct 31, 2014 at 2:58 PM, Heald, Mike wrote: > > Hi all, > > > > I'm implementing paging on storyboard, and I wanted to ask why we > decided to use marker based paging. I have some opinions on this, > but I want to keep my mouth shut until I find out what problem it > was solving :) > > > > Thanks, > > Mike > > > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova] [api] Get servers with limit and IP address filter
Hello, When applying an IP address filter to a paginated servers query (eg, supplying servers/detail?ip=192.168&limit=100), the IP address filtering is only being applied against the non-filtered page of servers that were retrieved from the DB; see [1]. I believe that the IP address filtering should be done before the limit is applied, returning up to servers that match the IP address filter. Currently, if the servers in the page of data returned from the DB do not happen to match the IP address filter (applied in the compute API), then no servers will be returned by the REST API (even if there are servers that match the IP address filter). This seems like a bug to me, shouldn't all filtering be done at the DB layer? [1]: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2037-L2042 Thanks, Steven Kaufer__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] [api] Get servers with limit and IP address filter
Vishvananda Ishaya wrote on 01/27/2015 04:29:50 PM: > From: Vishvananda Ishaya > To: "OpenStack Development Mailing List (not for usage questions)" > > Date: 01/27/2015 04:32 PM > Subject: Re: [openstack-dev] [nova] [api] Get servers with limit and > IP address filter > > The network info for an instance is cached as a blob of data > (neutron has the canonical version in most installs), so it isn’t > particularly easy to do at the database layer. You would likely need > a pretty complex stored procedure to do it accurately. > > Vish Vish, Thanks for the reply. I agree with your point about the difficultly in accurately querying the blob of data; however, IMHO, the complexity this fix does not preclude the current behavior as being classified as a bug. With that in mind, I was wondering if anyone in the community has any thoughts on if the current behavior is considered a bug? If so, how should it be resolved? A couple options that I could think of: 1. Disallow the combination of using both a limit and an IP address filter by raising an error. 2. Workaround the problem by removing the limit from the DB query and then manually limiting the list of servers (after manually applying the IP address filter). 3. Break up the query so that the server UUIDs that match the IP filter are retrieved first and then used as a UUID DB filter. As far as I can tell, this type of solution was originally implemented but the network query was deemed to expensive [1]. Is there a less expensive method to determine the UUIDs (possibly querying the cached 'network_info' in the ' instance_info_caches' table)? 4. Figure out how to accurately query the blob of network info that is cached in the nova DB and apply the IP filter at the DB layer. [1]: https://review.openstack.org/#/c/131460/ Thanks, Steven Kaufer > > On Jan 27, 2015, at 2:00 PM, Steven Kaufer wrote: > > Hello, > > When applying an IP address filter to a paginated servers query (eg, > supplying servers/detail?ip=192.168&limit=100), the IP address > filtering is only being applied against the non-filtered page of > servers that were retrieved from the DB; see [1]. > > I believe that the IP address filtering should be done before the > limit is applied, returning up to servers that match the IP > address filter. Currently, if the servers in the page of data > returned from the DB do not happen to match the IP address filter > (applied in the compute API), then no servers will be returned by > the REST API (even if there are servers that match the IP address filter). > > This seems like a bug to me, shouldn't all filtering be done at the DB layer? > > [1]: https://github.com/openstack/nova/blob/master/nova/compute/ > api.py#L2037-L2042 > > Thanks, > Steven Kaufer > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] [api] Get servers with limit and IP address filter
Vishvananda Ishaya wrote on 01/28/2015 11:32:16 AM: > From: Vishvananda Ishaya > To: "OpenStack Development Mailing List (not for usage questions)" > > Date: 01/28/2015 11:50 AM > Subject: Re: [openstack-dev] [nova] [api] Get servers with limit and > IP address filter > > On Jan 28, 2015, at 7:05 AM, Steven Kaufer wrote: > > Vishvananda Ishaya wrote on 01/27/2015 04:29:50 PM: > > > From: Vishvananda Ishaya > > To: "OpenStack Development Mailing List (not for usage questions)" > > > > Date: 01/27/2015 04:32 PM > > Subject: Re: [openstack-dev] [nova] [api] Get servers with limit and > > IP address filter > > > > The network info for an instance is cached as a blob of data > > (neutron has the canonical version in most installs), so it isn’t > > particularly easy to do at the database layer. You would likely need > > a pretty complex stored procedure to do it accurately. > > > > Vish > > Vish, > > Thanks for the reply. > > I agree with your point about the difficultly in accurately querying > the blob of data; however, IMHO, the complexity this fix does not > preclude the current behavior as being classified as a bug. > > With that in mind, I was wondering if anyone in the community has > any thoughts on if the current behavior is considered a bug? > > Yes it should be classified as a bug. Bug filed: https://bugs.launchpad.net/nova/+bug/1417649 > > If so, how should it be resolved? A couple options that I could think of: > > 1. Disallow the combination of using both a limit and an IP address > filter by raising an error. > > I think this is the simplest solution. > > Vish > > 2. Workaround the problem by removing the limit from the DB query > and then manually limiting the list of servers (after manually > applying the IP address filter). I have proposed a fix that implements this solution: https://review.openstack.org/#/c/152614 Thanks, Steven Kaufer > 3. Break up the query so that the server UUIDs that match the IP > filter are retrieved first and then used as a UUID DB filter. As far > as I can tell, this type of solution was originally implemented but > the network query was deemed to expensive [1]. Is there a less > expensive method to determine the UUIDs (possibly querying the > cached 'network_info' in the 'instance_info_caches' table)? > 4. Figure out how to accurately query the blob of network info that > is cached in the nova DB and apply the IP filter at the DB layer. > > [1]: https://review.openstack.org/#/c/131460/ > > Thanks, > Steven Kaufer__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Etherpad for volume replication created ...
Erlon Cruz wrote on 02/13/2015 07:51:34 AM: > From: Erlon Cruz > To: Danny Al-Gaaf , "OpenStack Development > Mailing List (not for usage questions)" > Date: 02/13/2015 07:53 AM > Subject: Re: [openstack-dev] [cinder] Etherpad for volume > replication created ... > > Do you have the log of the discussion as well? Erlon, The discussion is logged at [1], starting at 2015-02-12T18:32:40 and ending around 2015-02-12T19:53:20. [1]: http://eavesdrop.openstack.org/irclogs/%23openstack-cinder/%23openstack-cinder.2015-02-12.log Thanks, Steven Kaufer > > On Fri, Feb 13, 2015 at 7:09 AM, Danny Al-Gaaf > wrote: > Hi Jay, > > do you have a link to the etherpad? > > Danny > > Am 13.02.2015 um 05:54 schrieb Jay S. Bryant: > > All, > > > > Several members of the Cinder team and I were discussing the > > current state of volume replication while trying to figure out the > > best way to resolve bug 1383524 [1]. The outcome of the discussion > > was a decision to hold off on integrating volume replication > > support for additional drivers. > > > > I took notes from the discussion and have put them in the etherpad. > > We can use that, first thing in L, as a starting point to rework > > and fix replication support. > > > > Please let me know if you have any questions and feel free to > > update the etherpad with addition thoughts. > > > > Thanks! Jay > > > > > > [1] https://bugs.launchpad.net/cinder/+bug/1383524-- Periodic > > update replication status causing issues > > > > __ > > > > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [api] Retrieving the previous page of data
I am trying to understand which projects support a previous link for pagination and how that support is implemented. I believe that supporting a next link is quite common and is implemented as: 1. Caller includes a "marker" query string parameter which represents the ID of the last item in the previous page of data 2. Corresponding marker object is used to set the filtering information on the ORM query object in the common paginate_query function [1] As a convenience, when the GET reply has been limited, many projects also include a "next" href link that includes the next marker (allowing the caller to simply invoke that link to get the next page of data). I cannot find any utility functions to help generate the ORM query to retrieve the previous page of data, does anyone know how projects implement this? I have a few ideas on how to implement this by reversing the sort directions but am first trying to understand what currently exists. Does anyone know what support currently exists for a retrieving a previous page of data? [1]: https://github.com/openstack/oslo.db/blob/master/oslo_db/sqlalchemy/utils.py#L174-L177 Steven Kaufer__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
