[jira] [Commented] (LIBCLOUD-991) Can not retrieve more than 500VM on Google cloud
[ https://issues.apache.org/jira/browse/LIBCLOUD-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439876#comment-16439876 ] charles walker commented on LIBCLOUD-991: - I manage to fix my issue with "driver.ex_list(alisteNodeFun).page()" so i will close this ticket. I don t have time to pursue the possible handling of >500 VM directly inside libcloud method but I hope this ticket will help future users that have more than 500 entities on GCE. Thx [~quentinp] for the active support and help. > Can not retrieve more than 500VM on Google cloud > > > Key: LIBCLOUD-991 > URL: https://issues.apache.org/jira/browse/LIBCLOUD-991 > Project: Libcloud > Issue Type: Bug > Components: Compute > Environment: Google cloud project with more than 500VMs (python 3 and > last libcloud version AVL on pip) >Reporter: charles walker >Priority: Minor > > Today I notice that some of our VMs were not retrieve by our internal > toolings. After some investigation it seems that libcloud "list_nodes()" > method on google can only retrieve max 500VMs: > > [root@stackviewer-5cbcd8695-x964t /]# grep -o Vm response.log | wc -l > 500 > > when we have in reality around 750 VMs. So i had a quick look and my first > guess is that it comes from the call made by liblcoud using this REST API: > [https://cloud.google.com/compute/docs/reference/rest/v1/instances/aggregatedList] > with an optional argument "maxResults" that has a default value of 500 (which > seems also to be the max value BTW). I think i m on the good lead due to this > 500 limit mentioned in the doc and also observe on my side. > I will have a deeper look but just quickly open this ticket in case other > people already saw this limit > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIBCLOUD-991) Can not retrieve more than 500VM on Google cloud
[ https://issues.apache.org/jira/browse/LIBCLOUD-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436112#comment-16436112 ] Quentin Pradet commented on LIBCLOUD-991: - You're right: page() is needed. I have to agree that it's surprising, and I probably would not have done it this way: libcloud should abstract those things. At the same time GCEList and ex_list are quite powerful, and they should continue to work in the next versions. I don't know if it's possible to extend list_nodes to retrieve all nodes while not breaking any existing code? I won't be able to help here. Sorry! You could take it to the d...@libcloud.apache.org mailing list to see what the more experienced libcloud contributors think. > Can not retrieve more than 500VM on Google cloud > > > Key: LIBCLOUD-991 > URL: https://issues.apache.org/jira/browse/LIBCLOUD-991 > Project: Libcloud > Issue Type: Bug > Components: Compute > Environment: Google cloud project with more than 500VMs (python 3 and > last libcloud version AVL on pip) >Reporter: charles walker >Priority: Minor > > Today I notice that some of our VMs were not retrieve by our internal > toolings. After some investigation it seems that libcloud "list_nodes()" > method on google can only retrieve max 500VMs: > > [root@stackviewer-5cbcd8695-x964t /]# grep -o Vm response.log | wc -l > 500 > > when we have in reality around 750 VMs. So i had a quick look and my first > guess is that it comes from the call made by liblcoud using this REST API: > [https://cloud.google.com/compute/docs/reference/rest/v1/instances/aggregatedList] > with an optional argument "maxResults" that has a default value of 500 (which > seems also to be the max value BTW). I think i m on the good lead due to this > 500 limit mentioned in the doc and also observe on my side. > I will have a deeper look but just quickly open this ticket in case other > people already saw this limit > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIBCLOUD-991) Can not retrieve more than 500VM on Google cloud
[ https://issues.apache.org/jira/browse/LIBCLOUD-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432591#comment-16432591 ] charles walker commented on LIBCLOUD-991: - Hi, Thx for the feedbacks. I m not sure about "no need to call page()". I made a quick test with the following code: [https://github.com/charly37/testlibcloud/blob/master/test.py] and get the following output: [XX@XX test]# python3.6 test.py Starting Test1: Not calling page GCEList init Page Test 1 nbvmTest1: 500 Test2: calling page GCEList init Page Test2 Page Test2 nbvmTest2: 632 Starting Test 3 GCEList init Page test 3 Page test 3 nbvmTest3: 632 Either I miss something or made an error on my code but I think it is require to call page() when you have more than 500 results. I also come accross this PR which tend to confirms that: [https://github.com/apache/libcloud/pull/1095/files] Nevertheless I m able to get my 635 VMs and thus confirms that the 500 limitation can be handle by using the existing code (with the help of GCEList class) but still wonder if it would be better to have this hiden in libcloud (especialy since you said it yourself "were about to add it in a third place!". It is strange that all users need to add this logic on their side (I would understand that if i was using a specific google lib but libcloud should abstract this). What do you think ? Thx again for your feedback and support in resolving/understanding this issue. Charles. > Can not retrieve more than 500VM on Google cloud > > > Key: LIBCLOUD-991 > URL: https://issues.apache.org/jira/browse/LIBCLOUD-991 > Project: Libcloud > Issue Type: Bug > Components: Compute > Environment: Google cloud project with more than 500VMs (python 3 and > last libcloud version AVL on pip) >Reporter: charles walker >Priority: Minor > > Today I notice that some of our VMs were not retrieve by our internal > toolings. After some investigation it seems that libcloud "list_nodes()" > method on google can only retrieve max 500VMs: > > [root@stackviewer-5cbcd8695-x964t /]# grep -o Vm response.log | wc -l > 500 > > when we have in reality around 750 VMs. So i had a quick look and my first > guess is that it comes from the call made by liblcoud using this REST API: > [https://cloud.google.com/compute/docs/reference/rest/v1/instances/aggregatedList] > with an optional argument "maxResults" that has a default value of 500 (which > seems also to be the max value BTW). I think i m on the good lead due to this > 500 limit mentioned in the doc and also observe on my side. > I will have a deeper look but just quickly open this ticket in case other > people already saw this limit > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIBCLOUD-991) Can not retrieve more than 500VM on Google cloud
[ https://issues.apache.org/jira/browse/LIBCLOUD-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431708#comment-16431708 ] Quentin Pradet commented on LIBCLOUD-991: - Ah, you're right! So this functionality exists in two places: Ansible and Libcloud itself, and we were about to add it in a third place! I think you need to use driver.ex_list(driver.list_nodes) instead of calling driver.list_nodes() directly. Then you get an iterator that you can use (no need to call page()). See https://github.com/apache/libcloud/blob/9702f72b57f9b1372c289146a20cf201a0d6ea5d/libcloud/compute/drivers/gce.py#L241-L247. I think Ansible should have done that too, but they probably did not see the GCEList feature, just like me. Sorry, even if I can approve pull requests, I'm still quite new to libcloud. Feel free to close the issue if your problem is solved. > Can not retrieve more than 500VM on Google cloud > > > Key: LIBCLOUD-991 > URL: https://issues.apache.org/jira/browse/LIBCLOUD-991 > Project: Libcloud > Issue Type: Bug > Components: Compute > Environment: Google cloud project with more than 500VMs (python 3 and > last libcloud version AVL on pip) >Reporter: charles walker >Priority: Minor > > Today I notice that some of our VMs were not retrieve by our internal > toolings. After some investigation it seems that libcloud "list_nodes()" > method on google can only retrieve max 500VMs: > > [root@stackviewer-5cbcd8695-x964t /]# grep -o Vm response.log | wc -l > 500 > > when we have in reality around 750 VMs. So i had a quick look and my first > guess is that it comes from the call made by liblcoud using this REST API: > [https://cloud.google.com/compute/docs/reference/rest/v1/instances/aggregatedList] > with an optional argument "maxResults" that has a default value of 500 (which > seems also to be the max value BTW). I think i m on the good lead due to this > 500 limit mentioned in the doc and also observe on my side. > I will have a deeper look but just quickly open this ticket in case other > people already saw this limit > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIBCLOUD-991) Can not retrieve more than 500VM on Google cloud
[ https://issues.apache.org/jira/browse/LIBCLOUD-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430435#comment-16430435 ] charles walker commented on LIBCLOUD-991: - I think first i need to spend some time understanding the logic because it seems I miss something. Talking with another DEV in my company about that he was surprise and told me it was already fix and send me a commit: [https://github.com/ansible/ansible/pull/18554/files] I realize we were not talking about the same thing since it is a commit in Ansible but seems like to the same topic. After some more reading on the code it seems the 500 limitation is well know and in fact should maybe taken care on customer side (with pagination support) like Ansible done but i m still wondering if it would be nicer to have it directly integrated on libcloud... Maybe with a new method using GCE_LIST ([https://github.com/apache/libcloud/blob/trunk/libcloud/compute/drivers/gce.py#L209)] ? I still thinking about it and wondering if this Jira ticket is the best place to gather feedback on this topic > Can not retrieve more than 500VM on Google cloud > > > Key: LIBCLOUD-991 > URL: https://issues.apache.org/jira/browse/LIBCLOUD-991 > Project: Libcloud > Issue Type: Bug > Components: Compute > Environment: Google cloud project with more than 500VMs (python 3 and > last libcloud version AVL on pip) >Reporter: charles walker >Priority: Minor > > Today I notice that some of our VMs were not retrieve by our internal > toolings. After some investigation it seems that libcloud "list_nodes()" > method on google can only retrieve max 500VMs: > > [root@stackviewer-5cbcd8695-x964t /]# grep -o Vm response.log | wc -l > 500 > > when we have in reality around 750 VMs. So i had a quick look and my first > guess is that it comes from the call made by liblcoud using this REST API: > [https://cloud.google.com/compute/docs/reference/rest/v1/instances/aggregatedList] > with an optional argument "maxResults" that has a default value of 500 (which > seems also to be the max value BTW). I think i m on the good lead due to this > 500 limit mentioned in the doc and also observe on my side. > I will have a deeper look but just quickly open this ticket in case other > people already saw this limit > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIBCLOUD-991) Can not retrieve more than 500VM on Google cloud
[ https://issues.apache.org/jira/browse/LIBCLOUD-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427235#comment-16427235 ] Quentin Pradet commented on LIBCLOUD-991: - Great! So the first step would be to run the tests locally: http://libcloud.readthedocs.io/en/latest/testing.html. I like using `pytest` and `pytest -k gce` myself instead of messing with PYTHONPATH and setup.py test as documented. Then I guess you can try implementing the pagination support itself, which should not be too hard? I don't know. What I know is that adding tests can take some time (you need to understand how they work and record example responses from GCE), but we'll see that later. Please tell me if I can help with anything. > Can not retrieve more than 500VM on Google cloud > > > Key: LIBCLOUD-991 > URL: https://issues.apache.org/jira/browse/LIBCLOUD-991 > Project: Libcloud > Issue Type: Bug > Components: Compute > Environment: Google cloud project with more than 500VMs (python 3 and > last libcloud version AVL on pip) >Reporter: charles walker >Priority: Minor > > Today I notice that some of our VMs were not retrieve by our internal > toolings. After some investigation it seems that libcloud "list_nodes()" > method on google can only retrieve max 500VMs: > > [root@stackviewer-5cbcd8695-x964t /]# grep -o Vm response.log | wc -l > 500 > > when we have in reality around 750 VMs. So i had a quick look and my first > guess is that it comes from the call made by liblcoud using this REST API: > [https://cloud.google.com/compute/docs/reference/rest/v1/instances/aggregatedList] > with an optional argument "maxResults" that has a default value of 500 (which > seems also to be the max value BTW). I think i m on the good lead due to this > 500 limit mentioned in the doc and also observe on my side. > I will have a deeper look but just quickly open this ticket in case other > people already saw this limit > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIBCLOUD-991) Can not retrieve more than 500VM on Google cloud
[ https://issues.apache.org/jira/browse/LIBCLOUD-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422250#comment-16422250 ] charles walker commented on LIBCLOUD-991: - I m checking with my manager if I could spend some time on this topic. Will update this ticket ASAP. > Can not retrieve more than 500VM on Google cloud > > > Key: LIBCLOUD-991 > URL: https://issues.apache.org/jira/browse/LIBCLOUD-991 > Project: Libcloud > Issue Type: Bug > Components: Compute > Environment: Google cloud project with more than 500VMs (python 3 and > last libcloud version AVL on pip) >Reporter: charles walker >Priority: Minor > > Today I notice that some of our VMs were not retrieve by our internal > toolings. After some investigation it seems that libcloud "list_nodes()" > method on google can only retrieve max 500VMs: > > [root@stackviewer-5cbcd8695-x964t /]# grep -o Vm response.log | wc -l > 500 > > when we have in reality around 750 VMs. So i had a quick look and my first > guess is that it comes from the call made by liblcoud using this REST API: > [https://cloud.google.com/compute/docs/reference/rest/v1/instances/aggregatedList] > with an optional argument "maxResults" that has a default value of 500 (which > seems also to be the max value BTW). I think i m on the good lead due to this > 500 limit mentioned in the doc and also observe on my side. > I will have a deeper look but just quickly open this ticket in case other > people already saw this limit > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIBCLOUD-991) Can not retrieve more than 500VM on Google cloud
[ https://issues.apache.org/jira/browse/LIBCLOUD-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420151#comment-16420151 ] Quentin Pradet commented on LIBCLOUD-991: - Thank you for the report! This makes a lot of sense. Would you be interested in implementing pagination support? > Can not retrieve more than 500VM on Google cloud > > > Key: LIBCLOUD-991 > URL: https://issues.apache.org/jira/browse/LIBCLOUD-991 > Project: Libcloud > Issue Type: Bug > Components: Compute > Environment: Google cloud project with more than 500VMs (python 3 and > last libcloud version AVL on pip) >Reporter: charles walker >Priority: Minor > > Today I notice that some of our VMs were not retrieve by our internal > toolings. After some investigation it seems that libcloud "list_nodes()" > method on google can only retrieve max 500VMs: > > [root@stackviewer-5cbcd8695-x964t /]# grep -o Vm response.log | wc -l > 500 > > when we have in reality around 750 VMs. So i had a quick look and my first > guess is that it comes from the call made by liblcoud using this REST API: > [https://cloud.google.com/compute/docs/reference/rest/v1/instances/aggregatedList] > with an optional argument "maxResults" that has a default value of 500 (which > seems also to be the max value BTW). I think i m on the good lead due to this > 500 limit mentioned in the doc and also observe on my side. > I will have a deeper look but just quickly open this ticket in case other > people already saw this limit > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIBCLOUD-991) Can not retrieve more than 500VM on Google cloud
[ https://issues.apache.org/jira/browse/LIBCLOUD-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419650#comment-16419650 ] charles walker commented on LIBCLOUD-991: - [root@stackviewer-5cbcd8695-x964t /]# python3.6 --version Python 3.6.4 [root@stackviewer-5cbcd8695-x964t /]# pip3.6 list DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning. apache-libcloud (2.3.0) ... requests (2.18.4) setuptools (36.6.0) urllib3 (1.22) > Can not retrieve more than 500VM on Google cloud > > > Key: LIBCLOUD-991 > URL: https://issues.apache.org/jira/browse/LIBCLOUD-991 > Project: Libcloud > Issue Type: Bug > Components: Compute > Environment: Google cloud project with more than 500VMs (python 3 and > last libcloud version AVL on pip) >Reporter: charles walker >Priority: Minor > > Today I notice that some of our VMs were not retrieve by our internal > toolings. After some investigation it seems that libcloud "list_nodes()" > method on google can only retrieve max 500VMs: > > [root@stackviewer-5cbcd8695-x964t /]# grep -o Vm response.log | wc -l > 500 > > when we have in reality around 750 VMs. So i had a quick look and my first > guess is that it comes from the call made by liblcoud using this REST API: > [https://cloud.google.com/compute/docs/reference/rest/v1/instances/aggregatedList] > with an optional argument "maxResults" that has a default value of 500 (which > seems also to be the max value BTW). I think i m on the good lead due to this > 500 limit mentioned in the doc and also observe on my side. > I will have a deeper look but just quickly open this ticket in case other > people already saw this limit > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
