GitHub user vdloo opened a pull request:
https://github.com/apache/libcloud/pull/1263
Pagination in various OpenStack_2_NodeDriver methods
The default max_limits for OpenStack is 1000. If you have more than 1000
resources (i.e. snapshots) then everything but the newest 1000 will not be
listed. If you set the max_limits lower even less will not be returned, etc.
This change implements pagination for ex_list_snapshots, ex_list_ports, and
list_volumes in the OpenStack_2_NodeDriver. This PR builds on top of (and
contains) https://github.com/apache/libcloud/pull/1242.
Inspired by `_paginated_request` in the [DigitalOcean
driver](https://github.com/apache/libcloud/blob/bc097ffe5b04990e0ca302231388ee7d5d6fe911/libcloud/common/digitalocean.py#L176)
before:
```
In [3]: out = conn.list_snapshots()
In [4]: len(out)
Out[4]: 1000
```
after:
```
In [3]: out = conn.list_snapshots()
In [4]: len(out)
Out[4]: 53098
```
before:
```
In [6]: out = conn.list_volumes()
In [7]: len(out)
Out[7]: 1000
```
after:
```
In [3]: out = conn.list_volumes()
In [4]: len(out)
Out[4]: 4876
```
before:
```
In [8]: out = conn.ex_list_ports()
In [9]: len(out)
Out[9]: 123
```
after:
```
In [5]: out = conn.ex_list_ports()
In [6]: len(out)
Out[6]: 123
```
Note that this does not affect the OpenStack_1_1_NodeDriver and those still
won't be paginated and will not return anything except for the 1000 most recent
resources.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vdloo/libcloud
perform-pagination-for-ex-list-snapshots
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/libcloud/pull/1263.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1263
----
commit b3d8b15b81a934d9364df168ae5433549a5a880e
Author: micafer <micafer1@...>
Date: 2018-09-24T10:54:47Z
Add cinder support LIBCLOUD-874
commit 1761c77c836faa0af53f390bc9ae012ab56e653b
Author: micafer <micafer1@...>
Date: 2018-09-24T11:03:38Z
Add cinder support LIBCLOUD-874
commit 78e96a51eaca71c0156553de599fbf6b9e1179f6
Author: micafer <micafer1@...>
Date: 2018-09-24T13:19:43Z
Add cinder support LIBCLOUD-874
commit 82b3dd034eb23e2ce7df01f11305a08b52c4773c
Author: micafer <micafer1@...>
Date: 2018-09-24T14:10:39Z
Add cinder support LIBCLOUD-874
commit 557e1e5fa661eb2dedc914913606ea5cdb7fcb90
Author: micafer <micafer1@...>
Date: 2018-09-24T14:20:20Z
Add cinder support LIBCLOUD-874
commit a65f4a6cc4439355ae3d39dec6f606e0470a992e
Author: micafer <micafer1@...>
Date: 2018-09-24T14:32:58Z
Add cinder support LIBCLOUD-874
commit b6103c7be4f22960b45cb5d24ee50a18fe16cc28
Author: micafer <micafer1@...>
Date: 2018-09-24T15:13:29Z
Move sec groups to neutron api LIBCLOUD-874
commit bd8318b53676f84e91ed775dc174545f00ac834b
Author: micafer <micafer1@...>
Date: 2018-09-24T15:16:19Z
Add cinder support LIBCLOUD-874
commit 860bfad85d108d77923acb8422bf561e58b68d16
Author: micafer <micafer1@...>
Date: 2018-09-24T15:21:20Z
Merge branch 'trunk' into sec_groups
commit c5add2fc8436088e07f20e9da3a6c7a491257101
Author: micafer <micafer1@...>
Date: 2018-09-25T06:57:05Z
Move sec groups to neutron api LIBCLOUD-874
commit 9cc717f2643e3375d729f04816e8df65ed5a52db
Author: micafer <micafer1@...>
Date: 2018-09-25T13:35:28Z
Move floatingips to neutron api LIBCLOUD-874
commit ef30fe7f9ec8f6fbe09e384485f933a1ec7904c9
Author: micafer <micafer1@...>
Date: 2018-09-25T14:24:14Z
Fix _to_port to make it work with old versions
commit bcbd0ca33c4ce4649e12d8289091a850bdc38ffd
Author: micafer <micafer1@...>
Date: 2018-09-25T14:38:45Z
Remove print
commit fbe0e2613b1f8e6a40164d7513b357602b73de8e
Author: micafer <micafer1@...>
Date: 2018-09-25T14:54:01Z
Update comment
commit 7e1ccd5e6eeb8dac3060bc4fb10e45ed781738ed
Author: Miguel Caballer <micafer1@...>
Date: 2018-09-28T06:44:55Z
add direction in security_group_rule creation
commit 46e00c62be47cd1061f547d7f17a937b04567262
Author: micafer <micafer1@...>
Date: 2018-10-23T10:19:50Z
Add create and delete subnets
commit 57c674a1d45d9c9840c050cf02f807b3233037cd
Author: micafer <micafer1@...>
Date: 2018-10-23T10:20:16Z
Remove prints
commit a3944fe9292ef84c87a6321afe29919f1a6c33d5
Author: micafer <micafer1@...>
Date: 2018-10-23T11:05:07Z
Solve PR issues
commit 7f4d709dcd3755eb1bf46ad60d65e28697965289
Author: micafer <micafer1@...>
Date: 2018-10-23T11:05:33Z
Merge branch 'sec_groups' of https://github.com/micafer/libcloud into
sec_groups
commit d0be0552614764cfc24597154077a3abc118b870
Author: micafer <micafer1@...>
Date: 2018-10-23T14:34:43Z
Fix conn issues
commit 7ff6a15db791ea4064cd320f53d05003cb763561
Author: micafer <micafer1@...>
Date: 2018-11-28T11:10:50Z
Change order in gets
commit 5c5d45eecae5cc654816e2e25e03ce706ee91beb
Author: Rick van de Loo <rickvandeloo@...>
Date: 2018-11-28T16:29:45Z
list calls paginate in OpenStackNodeDriver
the default max_limits for OpenStack is 1000. If you have more than 1000
resources (i.e. snapshots) then everything but the newest 1000 will not be
listed. If you set the max_limits lower even less will not be returned, etc.
This change implements pagination for ex_list_snapshots, ex_list_ports, and
list_volumes in the OpenStack_2_NodeDriver.
----
---
