Re: [openstack-dev] [oslo.db] Marker based paging
Unfortunately, ordering on anything but the timestamp results in problems with usability and results consistency. Example: Ordering on priority so I can see all my high priority tasks first, then medium, then low. Works ok, as long as the underlying data is geared so that the priorities are ordered that way. But I have no way of skipping pages to find them if they're not, and if the task that that we're using as the marker changes priority as I page, I'm jumped to somewhere else in the list and I have no way of knowing where I am in the list. I'd either see a load of repeated data (with no way of skipping it) or miss out on a lot of data with no indication that this has happened. So that's quite specific to Storyboard, but think of a situation where you've ordered your instances in horizon by status so you can see all that are spawning, running, deleting, etc. grouped together. As you page, the marker's instance goes from spawning to running, or running to deleting. These aren't edge cases. The nature of the data we deal with forces us to handle results that are going to change beneath us. Now, I'm in no way suggesting that offset/limit would be a better solution! But we do need to document these use cases somewhere and make sure our paging solution can fulfil them, because Storyboard cannot provide the user experience it needs to with the current pagination method in oslo.db. Cheers, Mike From: Jay Pipes [jaypi...@gmail.com] Sent: 04 November 2014 07:07 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [oslo.db] Marker based paging On 11/04/2014 01:08 AM, Heald, Mike wrote: > Thanks for that, Steven :) > > So just to clarify, results are ordered by the relevant timestamps to > ensure consistent order and so that new records would never show on > previous "pages" and be missed, and we're limited to just a "next > page" navigation, and we cannot order the entire result set on any > column but the timestamps, as this would break the paging because we > can't do the comparisons we need to if the results aren't in that > order. Have I got that correct? No, that's not correct. There's nothing limiting one from ordering on other columns than timestamp. We always ensure that there is a secondary order on a column with unique values (like the primary key), in order to ensure that pages of results are strictly ordered even when the sort field is non-unique (like timestamp). We're limited to next-previous pagination by choice because of the scalability and performance limitations of a limit-offset pagination strategy. Best, jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] Marker based paging
On 11/04/2014 01:08 AM, Heald, Mike wrote: Thanks for that, Steven :) So just to clarify, results are ordered by the relevant timestamps to ensure consistent order and so that new records would never show on previous "pages" and be missed, and we're limited to just a "next page" navigation, and we cannot order the entire result set on any column but the timestamps, as this would break the paging because we can't do the comparisons we need to if the results aren't in that order. Have I got that correct? No, that's not correct. There's nothing limiting one from ordering on other columns than timestamp. We always ensure that there is a secondary order on a column with unique values (like the primary key), in order to ensure that pages of results are strictly ordered even when the sort field is non-unique (like timestamp). We're limited to next-previous pagination by choice because of the scalability and performance limitations of a limit-offset pagination strategy. Best, jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] Marker based paging
Thanks for that, Steven :) So just to clarify, results are ordered by the relevant timestamps to ensure consistent order and so that new records would never show on previous "pages" and be missed, and we're limited to just a "next page" navigation, and we cannot order the entire result set on any column but the timestamps, as this would break the paging because we can't do the comparisons we need to if the results aren't in that order. Have I got that correct? Thanks, Mike From: Steven Kaufer [kau...@us.ibm.com] Sent: 03 November 2014 22:39 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [oslo.db] Marker based paging Here are a few ML threads on this topic: http://lists.openstack.org/pipermail/openstack-dev/2014-March/030322.html http://www.gossamer-threads.com/lists/openstack/dev/2777 http://lists.openstack.org/pipermail/openstack-dev/2013-November/018861.html Thanks, Steven Kaufer Roman Podoliaka wrote on 11/03/2014 10:35:21 AM: > From: Roman Podoliaka > To: "OpenStack Development Mailing List (not for usage questions)" > > Date: 11/03/2014 10:43 AM > Subject: Re: [openstack-dev] [oslo.db] Marker based paging > > Hi Mike, > > I think that code was taken from Nova (or maybe some other project) as > is and we haven't touched it since then. > > Please speak up - we want to know about all possible problems with > current approach. > > Thanks, > Roman > > On Fri, Oct 31, 2014 at 2:58 PM, Heald, Mike wrote: > > Hi all, > > > > I'm implementing paging on storyboard, and I wanted to ask why we > decided to use marker based paging. I have some opinions on this, > but I want to keep my mouth shut until I find out what problem it > was solving :) > > > > Thanks, > > Mike > > > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] Marker based paging
Here are a few ML threads on this topic: http://lists.openstack.org/pipermail/openstack-dev/2014-March/030322.html http://www.gossamer-threads.com/lists/openstack/dev/2777 http://lists.openstack.org/pipermail/openstack-dev/2013-November/018861.html Thanks, Steven Kaufer Roman Podoliaka wrote on 11/03/2014 10:35:21 AM: > From: Roman Podoliaka > To: "OpenStack Development Mailing List (not for usage questions)" > > Date: 11/03/2014 10:43 AM > Subject: Re: [openstack-dev] [oslo.db] Marker based paging > > Hi Mike, > > I think that code was taken from Nova (or maybe some other project) as > is and we haven't touched it since then. > > Please speak up - we want to know about all possible problems with > current approach. > > Thanks, > Roman > > On Fri, Oct 31, 2014 at 2:58 PM, Heald, Mike wrote: > > Hi all, > > > > I'm implementing paging on storyboard, and I wanted to ask why we > decided to use marker based paging. I have some opinions on this, > but I want to keep my mouth shut until I find out what problem it > was solving :) > > > > Thanks, > > Mike > > > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] Marker based paging
Hi Mike, I think that code was taken from Nova (or maybe some other project) as is and we haven't touched it since then. Please speak up - we want to know about all possible problems with current approach. Thanks, Roman On Fri, Oct 31, 2014 at 2:58 PM, Heald, Mike wrote: > Hi all, > > I'm implementing paging on storyboard, and I wanted to ask why we decided to > use marker based paging. I have some opinions on this, but I want to keep my > mouth shut until I find out what problem it was solving :) > > Thanks, > Mike > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo.db] Marker based paging
Hi all, I'm implementing paging on storyboard, and I wanted to ask why we decided to use marker based paging. I have some opinions on this, but I want to keep my mouth shut until I find out what problem it was solving :) Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
