Re: [openstack-dev] [Ceilometer][Horizon] The future or pagination

2013-11-13 Thread John Dickinson
Swift uses marker+limit for pagination when listing containers or objects (with 
additional support for prefix, delimiters, and end markers). This is done 
because the total size of the listing may be rather large, and going to a 
correct page based on an offset gets expensive and doesn't allow for 
repeatable queries.

Pagination implies some sort of ordering, and I'm guessing (assuming+hoping) 
that your listings are based around something more meaningful that an 
incrementing id. By itself, metric number 32592 doesn't mean anything, and 
listings like go to metric 4200 and give me the next 768 items doesn't 
tell the consumer anything and probably isn't even a very repeatable query. 
Therefore, using a marker+prefix+limit style pagination system is very useful 
(eg give me up to 1000 metrics that start with 'nova/instance_id/42/'). Also, 
end_marker queries are very nice (half-closed ranges).

One thing I would suggest (and I hope we change in Swift whenever we update the 
API version) is that you don't promise to return the full page in a response. 
Instead, you should return a no matches or end of listing token. This 
allows you the flexibility to return responses quickly without consuming too 
many resources on the server side. Clients can then continue to iterate over 
subsequent pages as they are needed.

Something else that I'd like to see in Swift (it was almost added once) is the 
ability to reverse the order of the listings so you can iterate backwards over 
pages.

--John




On Nov 13, 2013, at 2:58 AM, Julien Danjou jul...@danjou.info wrote:

 Hi,
 
 We've been discussing and working for a while on support for pagination
 on our API v2 in Ceilometer. There's a large amount that already been
 done, but that is now stalled because we are not sure about the
 consensus.
 
 There's mainly two approaches around pagination as far as I know, one
 being using limit/offset and the other one being marker based. As of
 today, we have no clue of which one we should pick, in the case we would
 have a technical choice doable between these two.
 
 I've added the Horizon tag in the subject because I think it may concern
 Horizon, since it shall be someday in the future one of the main
 consumer of the Ceilometer API.
 
 I'd be also happy to learn what other projects do in this regard, and
 what has been said and discussed during the summit.
 
 To a certain extend, we Ceilometer would also be happy to find common
 technical ground on this to some extend so _maybe_ we can generalise
 this into WSME itself for consumption from other projects.
 
 Cheers,
 -- 
 Julien Danjou
 ;; Free Software hacker ; independent consultant
 ;; http://julien.danjou.info
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceilometer][Horizon] The future or pagination

2013-11-13 Thread Lyle, David
From a purely UI perspective, the limit/offset is a lot more useful.  Then we 
can show links to previous page, next page and display the current page number.

Past mailing list conversations have indicated that limit/offset can be less 
efficient on the server side.  The marker/limit approach works for paginating 
UI side, just in a more primitive way.  With that approach, we are generally 
limited to a next page link only.

David 

 -Original Message-
 From: John Dickinson [mailto:m...@not.mn]
 Sent: Wednesday, November 13, 2013 10:09 AM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Ceilometer][Horizon] The future or
 pagination
 
 Swift uses marker+limit for pagination when listing containers or objects
 (with additional support for prefix, delimiters, and end markers). This is
 done because the total size of the listing may be rather large, and going to a
 correct page based on an offset gets expensive and doesn't allow for
 repeatable queries.
 
 Pagination implies some sort of ordering, and I'm guessing
 (assuming+hoping) that your listings are based around something more
 meaningful that an incrementing id. By itself, metric number 32592
 doesn't mean anything, and listings like go to metric 4200 and give me
 the next 768 items doesn't tell the consumer anything and probably isn't
 even a very repeatable query. Therefore, using a marker+prefix+limit style
 pagination system is very useful (eg give me up to 1000 metrics that start
 with 'nova/instance_id/42/'). Also, end_marker queries are very nice (half-
 closed ranges).
 
 One thing I would suggest (and I hope we change in Swift whenever we
 update the API version) is that you don't promise to return the full page in a
 response. Instead, you should return a no matches or end of listing
 token. This allows you the flexibility to return responses quickly without
 consuming too many resources on the server side. Clients can then continue
 to iterate over subsequent pages as they are needed.
 
 Something else that I'd like to see in Swift (it was almost added once) is the
 ability to reverse the order of the listings so you can iterate backwards over
 pages.
 
 --John
 
 
 
 
 On Nov 13, 2013, at 2:58 AM, Julien Danjou jul...@danjou.info wrote:
 
  Hi,
 
  We've been discussing and working for a while on support for
  pagination on our API v2 in Ceilometer. There's a large amount that
  already been done, but that is now stalled because we are not sure
  about the consensus.
 
  There's mainly two approaches around pagination as far as I know, one
  being using limit/offset and the other one being marker based. As of
  today, we have no clue of which one we should pick, in the case we
  would have a technical choice doable between these two.
 
  I've added the Horizon tag in the subject because I think it may
  concern Horizon, since it shall be someday in the future one of the
  main consumer of the Ceilometer API.
 
  I'd be also happy to learn what other projects do in this regard, and
  what has been said and discussed during the summit.
 
  To a certain extend, we Ceilometer would also be happy to find common
  technical ground on this to some extend so _maybe_ we can generalise
  this into WSME itself for consumption from other projects.
 
  Cheers,
  --
  Julien Danjou
  ;; Free Software hacker ; independent consultant ;;
  http://julien.danjou.info
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev