[GitHub] libcloud pull request: [LIBCLOUD-707] Fixed exception when parsing...
GitHub user chrisob opened a pull request: https://github.com/apache/libcloud/pull/514 [LIBCLOUD-707] Fixed exception when parsing CloudSigma static IPs Fixes issue with CloudSigma compute driver: https://issues.apache.org/jira/browse/LIBCLOUD-707 You can merge this pull request into a Git repository by running: $ git pull https://github.com/chrisob/libcloud LIBCLOUD-707_cloudsigma_response_parsing_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/libcloud/pull/514.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #514 commit 2a48a10f262f18c9e107b050dee464a9bae73886 Author: Chris O'Brien chriso...@gmail.com Date: 2015-05-13T11:14:07Z [LIBCLOUD-707] Fixed exception when parsing CloudSigma static IPs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (LIBCLOUD-707) CloudSigma driver throws exception when parsing response in list_nodes
[ https://issues.apache.org/jira/browse/LIBCLOUD-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541741#comment-14541741 ] ASF GitHub Bot commented on LIBCLOUD-707: - GitHub user chrisob opened a pull request: https://github.com/apache/libcloud/pull/514 [LIBCLOUD-707] Fixed exception when parsing CloudSigma static IPs Fixes issue with CloudSigma compute driver: https://issues.apache.org/jira/browse/LIBCLOUD-707 You can merge this pull request into a Git repository by running: $ git pull https://github.com/chrisob/libcloud LIBCLOUD-707_cloudsigma_response_parsing_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/libcloud/pull/514.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #514 commit 2a48a10f262f18c9e107b050dee464a9bae73886 Author: Chris O'Brien chriso...@gmail.com Date: 2015-05-13T11:14:07Z [LIBCLOUD-707] Fixed exception when parsing CloudSigma static IPs CloudSigma driver throws exception when parsing response in list_nodes -- Key: LIBCLOUD-707 URL: https://issues.apache.org/jira/browse/LIBCLOUD-707 Project: Libcloud Issue Type: Bug Components: Compute Affects Versions: 0.14.0-beta3, 0.13.3, 0.15.0, 0.15.1 Reporter: Chris O'Brien While attempting to parse the IPs from a node's NICs in CloudSigma's API response, libcloud throws an exception when the node's IP is static: {noformat} TypeError: unhashable type: 'dict' File /home/dbs_support/dev/libcloudsigma/test.py, line 55, in module instances = deployer.getInstances() File /home/dbs_support/dev/libcloudsigma/test.py, line 44, in getInstances return self.driver.list_nodes() File /home/dbs_support/lib/libcloud/compute/drivers/cloudsigma.py, line 1026, in list_nodes nodes = [self._to_node(data=item) for item in response['objects']] File /home/dbs_support/lib/libcloud/compute/drivers/cloudsigma.py, line 1910, in _to_node _public_ips, _private_ips = self._parse_ips_from_nic(nic=nic) File /home/dbs_support/lib/libcloud/compute/drivers/cloudsigma.py, line 1888, in _parse_ips_from_nic ips = set(ips) {noformat} It seems that libcloud doesn't account for CloudSigma's response for static IPs containing the 'uuid' as a subobject of 'ip' in the 'ip_v4_conf' for a NIC. For example, a NIC with a static IP belonging to a server: {noformat} { firewall_policy:null, boot_order:null, vlan:null, ip_v4_conf:{ ip:{ uuid:178.22.68.44, resource_uri:/api/2.0/ips/178.22.68.44/ }, conf:static }, mac:22:9d:42:9d:30:2f, model:virtio, runtime:{ interface_type:public, ip_v4:{ uuid:178.22.68.44, resource_uri:/api/2.0/ips/178.22.68.44/ }, io:{ bytes_sent:403849752, packets_sent:6089897, bytes_recv:1348149166, packets_recv:115 }, ip_v6:null }, ip_v6_conf:null } {noformat} The NIC's {{\['ip_v4_conf'\]\['ip'\]}} object (which is a dict) gets put into a list (ips) with other strings (representing IPs). It chokes when attempting to convert this list containing both dicts and strings into a set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
libcloud git commit: [google|compute] Add support for GCE list paging and filtering
Repository: libcloud Updated Branches: refs/heads/trunk 582633929 - 1ed502045 [google|compute] Add support for GCE list paging and filtering GCE will return a maximum of 500 resources in a single list. This change adds an iterator that allows filtering and/or paging of list results. Closes #491 Signed-off-by: Eric Johnson erjoh...@google.com Project: http://git-wip-us.apache.org/repos/asf/libcloud/repo Commit: http://git-wip-us.apache.org/repos/asf/libcloud/commit/1ed50204 Tree: http://git-wip-us.apache.org/repos/asf/libcloud/tree/1ed50204 Diff: http://git-wip-us.apache.org/repos/asf/libcloud/diff/1ed50204 Branch: refs/heads/trunk Commit: 1ed5020450e6e8deb0361239bfb91ba795447e87 Parents: 5826339 Author: Lee Verberne v...@google.com Authored: Tue Jan 6 09:54:05 2015 -0800 Committer: Eric Johnson erjoh...@google.com Committed: Wed May 13 13:52:46 2015 + -- CHANGES.rst | 4 + libcloud/compute/drivers/gce.py | 174 ++- .../compute/fixtures/gce/regions-paged-1.json | 97 +++ .../compute/fixtures/gce/regions-paged-2.json | 52 ++ libcloud/test/compute/fixtures/gce/zones.json | 12 +- libcloud/test/compute/test_gce.py | 63 ++- 6 files changed, 393 insertions(+), 9 deletions(-) -- http://git-wip-us.apache.org/repos/asf/libcloud/blob/1ed50204/CHANGES.rst -- diff --git a/CHANGES.rst b/CHANGES.rst index 98ba7a4..5b7ef6f 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -25,6 +25,10 @@ General Compute ~~~ +- Google Compute now supports paginated lists including filtering. + (GITHUB-491) + [Lee Verberne] + - OpenStackNodeSize objects now support optional, additional fields that are supported in OpenStack 2.1: `ephemeral_disk`, `swap`, `extra`. (GITHUB-488, LIBCLOUD-682) http://git-wip-us.apache.org/repos/asf/libcloud/blob/1ed50204/libcloud/compute/drivers/gce.py -- diff --git a/libcloud/compute/drivers/gce.py b/libcloud/compute/drivers/gce.py index 666f1c1..e1e7549 100644 --- a/libcloud/compute/drivers/gce.py +++ b/libcloud/compute/drivers/gce.py @@ -64,7 +64,28 @@ class GCEResponse(GoogleResponse): class GCEConnection(GoogleBaseConnection): -Connection class for the GCE driver. + +Connection class for the GCE driver. + +GCEConnection extends :class:`google.GoogleBaseConnection` for 2 reasons: + 1. modify request_path for GCE URI. + 2. Implement gce_params functionality described below. + +If the parameter gce_params is set to a dict prior to calling request(), +the URL parameters will be updated to include those key/values FOR A +SINGLE REQUEST. If the response contains a nextPageToken, +gce_params['pageToken'] will be set to its value. This can be used to +implement paging in list: + + params, more_results = {'maxResults': 2}, True + while more_results: +... driver.connection.gce_params=params +... driver.ex_list_urlmaps() +... more_results = 'pageToken' in params +... +[GCEUrlMap id=... name=cli-map, GCEUrlMap id=... name=lc-map] +[GCEUrlMap id=... name=web-map] + host = 'www.googleapis.com' responseCls = GCEResponse @@ -76,6 +97,138 @@ class GCEConnection(GoogleBaseConnection): **kwargs) self.request_path = '/compute/%s/projects/%s' % (API_VERSION, project) +self.gce_params = None + +def pre_connect_hook(self, params, headers): + +Update URL parameters with values from self.gce_params. + +@inherits: :class:`GoogleBaseConnection.pre_connect_hook` + +params, headers = super(GCEConnection, self).pre_connect_hook(params, + headers) +if self.gce_params: +params.update(self.gce_params) +return params, headers + +def request(self, *args, **kwargs): + +Perform request then do GCE-specific processing of URL params. + +@inherits: :class:`GoogleBaseConnection.request` + +response = super(GCEConnection, self).request(*args, **kwargs) + +# If gce_params has been set, then update the pageToken with the +# nextPageToken so it can be used in the next request. +if self.gce_params: +if 'nextPageToken' in response.object: +self.gce_params['pageToken'] = response.object['nextPageToken'] +elif 'pageToken' in self.gce_params: +del self.gce_params['pageToken'] +self.gce_params = None + +return response + + +class
[GitHub] libcloud pull request: Add support for GCE list paging and filteri...
Github user asfgit closed the pull request at: https://github.com/apache/libcloud/pull/491 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (LIBCLOUD-657) Update Google auth to support new JSON key format
[ https://issues.apache.org/jira/browse/LIBCLOUD-657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Johnson resolved LIBCLOUD-657. --- Resolution: Implemented Update Google auth to support new JSON key format - Key: LIBCLOUD-657 URL: https://issues.apache.org/jira/browse/LIBCLOUD-657 Project: Libcloud Issue Type: New Feature Components: Compute Reporter: Eric Johnson Assignee: Eric Johnson Google introduced a new Service Account key format (JSON) and over time this will likely become the preferred format. We should begin by supporting this new format soon. If Google does make this the default, we may want to follow-up and issue a (debug | info) warning to users relying on P12 format. AFAIK, there is no clear documentation on this format. However, if you open up the Developers Console and look under APIs Auth - Credentials, you'll see the Generate New JSON Key button next to any existing Service Accounts (or be given the option to create one with a new SA). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (LIBCLOUD-627) service account auth fails with gce driver on python3
[ https://issues.apache.org/jira/browse/LIBCLOUD-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Johnson resolved LIBCLOUD-627. --- Resolution: Implemented service account auth fails with gce driver on python3 - Key: LIBCLOUD-627 URL: https://issues.apache.org/jira/browse/LIBCLOUD-627 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Siim Põder Assignee: Eric Johnson Priority: Minor Labels: Google Fails due to string/binary confusion in GoogleServiceAcctAuthConnection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (LIBCLOUD-709) Implement retry on rate limit for nova api
Arpith Prakash created LIBCLOUD-709: --- Summary: Implement retry on rate limit for nova api Key: LIBCLOUD-709 URL: https://issues.apache.org/jira/browse/LIBCLOUD-709 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Arpith Prakash Priority: Trivial Nova client raises exception when there are too many request, and retry-after time is being sent within the header. We should be catching that exception and retry automatically which could resolve most of downstream additional logic of implement it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (LIBCLOUD-699) GCE Compute driver cannot accept ex_nic_gce_struct for create_node
[ https://issues.apache.org/jira/browse/LIBCLOUD-699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Johnson reassigned LIBCLOUD-699: - Assignee: Eric Johnson GCE Compute driver cannot accept ex_nic_gce_struct for create_node -- Key: LIBCLOUD-699 URL: https://issues.apache.org/jira/browse/LIBCLOUD-699 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Jason DeTiberus Assignee: Eric Johnson If you specify the {{ex_nic_gce_struct}} param for the {{create_node}} or {{ex_create_multiple_nodes}} Any value other than {{None}} and the test for {{if network and ex_nic_gce_struct:}} in {{_create_node_req}} fails due to network having a default value of 'default'. As a workaround I tried to set network to '', since that would have caused it to pass the comparison test, however that leads to an error from the following snippet: {code} if not hasattr(ex_network, 'name'): ex_network = self.ex_get_network(ex_network) {code} Which returns with the following error: {code} error: {'locationType': 'parameter', 'domain': 'global', 'message': Invalid value 'None'. Values must match the following regular expression: '[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?', 'reason': 'invalidParameter', 'location': 'network'} {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (LIBCLOUD-697) GCE Compute driver create_node does not accept str for ex_boot_disk
[ https://issues.apache.org/jira/browse/LIBCLOUD-697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Johnson reassigned LIBCLOUD-697: - Assignee: Eric Johnson GCE Compute driver create_node does not accept str for ex_boot_disk --- Key: LIBCLOUD-697 URL: https://issues.apache.org/jira/browse/LIBCLOUD-697 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Jason DeTiberus Assignee: Eric Johnson The documentation for the GCE compute driver states that the ex_boot_disk param for create_node takes either a StorageVolume or str. If you pass a string, it results in the following traceback: {code:none} failed: [localhost] = {failed: true, parsed: false} Traceback (most recent call last): File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 2278, in module main() File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 577, in main module.exit_json(**(node_mgr.execute())) File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 352, in execute metadata) File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 461, in create_instances_by_names lc_network, lc_zone, metadata) File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 433, in create_instance_by_name ex_nic_gce_struct=nics File /usr/lib/python2.7/site-packages/libcloud/compute/drivers/gce.py, line 2202, in create_node ex_automatic_restart) File /usr/lib/python2.7/site-packages/libcloud/compute/drivers/gce.py, line 4507, in _create_node_req 'deviceName': boot_disk.name, AttributeError: 'str' object has no attribute 'name' {code} The issue appears to be that create_node is passing the ex_boot_disk argument directly to _create_node_req (where the boot_disk param expects a StorageVolume or None). Either the docs need to be updated or create_node should instantiate a StorageVolume from ex_boot_disk if it is a str. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (LIBCLOUD-697) GCE Compute driver create_node does not accept str for ex_boot_disk
[ https://issues.apache.org/jira/browse/LIBCLOUD-697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Johnson resolved LIBCLOUD-697. --- Resolution: Implemented GCE Compute driver create_node does not accept str for ex_boot_disk --- Key: LIBCLOUD-697 URL: https://issues.apache.org/jira/browse/LIBCLOUD-697 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Jason DeTiberus Assignee: Eric Johnson The documentation for the GCE compute driver states that the ex_boot_disk param for create_node takes either a StorageVolume or str. If you pass a string, it results in the following traceback: {code:none} failed: [localhost] = {failed: true, parsed: false} Traceback (most recent call last): File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 2278, in module main() File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 577, in main module.exit_json(**(node_mgr.execute())) File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 352, in execute metadata) File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 461, in create_instances_by_names lc_network, lc_zone, metadata) File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 433, in create_instance_by_name ex_nic_gce_struct=nics File /usr/lib/python2.7/site-packages/libcloud/compute/drivers/gce.py, line 2202, in create_node ex_automatic_restart) File /usr/lib/python2.7/site-packages/libcloud/compute/drivers/gce.py, line 4507, in _create_node_req 'deviceName': boot_disk.name, AttributeError: 'str' object has no attribute 'name' {code} The issue appears to be that create_node is passing the ex_boot_disk argument directly to _create_node_req (where the boot_disk param expects a StorageVolume or None). Either the docs need to be updated or create_node should instantiate a StorageVolume from ex_boot_disk if it is a str. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (LIBCLOUD-687) Unable to set GCE metadata if no metadata already exists
[ https://issues.apache.org/jira/browse/LIBCLOUD-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542681#comment-14542681 ] ASF subversion and git services commented on LIBCLOUD-687: -- Commit aaa8e9e3a3a73d44aa6632089d60e8806f0f2de4 in libcloud's branch refs/heads/trunk from [~erjohnso] [ https://git-wip-us.apache.org/repos/asf?p=libcloud.git;h=aaa8e9e ] [google compute] Fixes LIBCLOUD-687: handle empty project metadata Closes #521 Signed-off-by: Eric Johnson erjoh...@google.com Unable to set GCE metadata if no metadata already exists Key: LIBCLOUD-687 URL: https://issues.apache.org/jira/browse/LIBCLOUD-687 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Eli Courtwright Assignee: Eric Johnson The {{libcloud.compute.drivers.gce.GCENodeDriver}} class has a nonstandard method {{ex_set_common_metadata}} which we can use to (among other things) set SSH keys. When calling this method on the latest libcloud 0.17.0 (which is not currently a listed version in Jira), I get the following error: {code} conn.ex_set_common_instance_metadata({}) Traceback (most recent call last): File stdin, line 1, in module File /opt/sideboard/lib/python2.7/site-packages/libcloud/compute/drivers/gce.py, line 1129, in ex_set_common_instance_metadata for md in current_metadata['items']: KeyError: 'items' {code} I get this error regardless of what data I pass to the {{ex_set_common_instance_metadata}} method. From delving into the libcloud code, I find the following block of code: {code} project = self.ex_get_project() current_metadata = project.extra['commonInstanceMetadata'] fingerprint = current_metadata['fingerprint'] # grab copy of current 'sshKeys' in case we want to retain them current_keys = for md in current_metadata['items']: {code} So it looks as if the method assumes that {{self.ex_get_project()}} will return: - an object with a {{.extra}} dictionary (which it does) - that dictionary will have a commonInstanceMetadata key which maps to another dictionary (which is correct) - that sub-dictionary will have an items key whose value is iterable (which is wrong) That last assumption is only wrong if there is no project metadata already associated with the project, e.g. for a new project. I'm working around this problem currently by monkeypatching libcloud: {code} def patch_gce_ex_get_project(): orig_get_project = GCENodeDriver.ex_get_project def ex_get_project(self): project = orig_get_project(self) project.extra['commonInstanceMetadata'].setdefault('items', []) return project GCENodeDriver.ex_get_project = ex_get_project {code} but having a real fix would be great for the next version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
libcloud git commit: [google compute] Fixes LIBCLOUD-687: handle empty project metadata
Repository: libcloud Updated Branches: refs/heads/trunk 8664118c8 - aaa8e9e3a [google compute] Fixes LIBCLOUD-687: handle empty project metadata Closes #521 Signed-off-by: Eric Johnson erjoh...@google.com Project: http://git-wip-us.apache.org/repos/asf/libcloud/repo Commit: http://git-wip-us.apache.org/repos/asf/libcloud/commit/aaa8e9e3 Tree: http://git-wip-us.apache.org/repos/asf/libcloud/tree/aaa8e9e3 Diff: http://git-wip-us.apache.org/repos/asf/libcloud/diff/aaa8e9e3 Branch: refs/heads/trunk Commit: aaa8e9e3a3a73d44aa6632089d60e8806f0f2de4 Parents: 8664118 Author: Eric Johnson erjoh...@google.com Authored: Wed May 13 20:31:45 2015 + Committer: Eric Johnson erjoh...@google.com Committed: Wed May 13 20:45:23 2015 + -- libcloud/compute/drivers/gce.py | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/libcloud/blob/aaa8e9e3/libcloud/compute/drivers/gce.py -- diff --git a/libcloud/compute/drivers/gce.py b/libcloud/compute/drivers/gce.py index 4f15715..90c1e05 100644 --- a/libcloud/compute/drivers/gce.py +++ b/libcloud/compute/drivers/gce.py @@ -1300,10 +1300,13 @@ class GCENodeDriver(NodeDriver): project = self.ex_get_project() current_metadata = project.extra['commonInstanceMetadata'] fingerprint = current_metadata['fingerprint'] +md_items = [] +if 'items' in current_metadata: +md_items = current_metadata['items'] # grab copy of current 'sshKeys' in case we want to retain them current_keys = -for md in current_metadata['items']: +for md in md_items: if md['key'] == 'sshKeys': current_keys = md['value']
[GitHub] libcloud pull request: [google compute] handle empty project metad...
Github user asfgit closed the pull request at: https://github.com/apache/libcloud/pull/521 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (LIBCLOUD-699) GCE Compute driver cannot accept ex_nic_gce_struct for create_node
[ https://issues.apache.org/jira/browse/LIBCLOUD-699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542523#comment-14542523 ] ASF GitHub Bot commented on LIBCLOUD-699: - Github user erjohnso closed the pull request at: https://github.com/apache/libcloud/pull/518 GCE Compute driver cannot accept ex_nic_gce_struct for create_node -- Key: LIBCLOUD-699 URL: https://issues.apache.org/jira/browse/LIBCLOUD-699 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Jason DeTiberus Assignee: Eric Johnson If you specify the {{ex_nic_gce_struct}} param for the {{create_node}} or {{ex_create_multiple_nodes}} Any value other than {{None}} and the test for {{if network and ex_nic_gce_struct:}} in {{_create_node_req}} fails due to network having a default value of 'default'. As a workaround I tried to set network to '', since that would have caused it to pass the comparison test, however that leads to an error from the following snippet: {code} if not hasattr(ex_network, 'name'): ex_network = self.ex_get_network(ex_network) {code} Which returns with the following error: {code} error: {'locationType': 'parameter', 'domain': 'global', 'message': Invalid value 'None'. Values must match the following regular expression: '[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?', 'reason': 'invalidParameter', 'location': 'network'} {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (LIBCLOUD-687) Unable to set GCE metadata if no metadata already exists
[ https://issues.apache.org/jira/browse/LIBCLOUD-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542682#comment-14542682 ] ASF GitHub Bot commented on LIBCLOUD-687: - Github user asfgit closed the pull request at: https://github.com/apache/libcloud/pull/521 Unable to set GCE metadata if no metadata already exists Key: LIBCLOUD-687 URL: https://issues.apache.org/jira/browse/LIBCLOUD-687 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Eli Courtwright Assignee: Eric Johnson The {{libcloud.compute.drivers.gce.GCENodeDriver}} class has a nonstandard method {{ex_set_common_metadata}} which we can use to (among other things) set SSH keys. When calling this method on the latest libcloud 0.17.0 (which is not currently a listed version in Jira), I get the following error: {code} conn.ex_set_common_instance_metadata({}) Traceback (most recent call last): File stdin, line 1, in module File /opt/sideboard/lib/python2.7/site-packages/libcloud/compute/drivers/gce.py, line 1129, in ex_set_common_instance_metadata for md in current_metadata['items']: KeyError: 'items' {code} I get this error regardless of what data I pass to the {{ex_set_common_instance_metadata}} method. From delving into the libcloud code, I find the following block of code: {code} project = self.ex_get_project() current_metadata = project.extra['commonInstanceMetadata'] fingerprint = current_metadata['fingerprint'] # grab copy of current 'sshKeys' in case we want to retain them current_keys = for md in current_metadata['items']: {code} So it looks as if the method assumes that {{self.ex_get_project()}} will return: - an object with a {{.extra}} dictionary (which it does) - that dictionary will have a commonInstanceMetadata key which maps to another dictionary (which is correct) - that sub-dictionary will have an items key whose value is iterable (which is wrong) That last assumption is only wrong if there is no project metadata already associated with the project, e.g. for a new project. I'm working around this problem currently by monkeypatching libcloud: {code} def patch_gce_ex_get_project(): orig_get_project = GCENodeDriver.ex_get_project def ex_get_project(self): project = orig_get_project(self) project.extra['commonInstanceMetadata'].setdefault('items', []) return project GCENodeDriver.ex_get_project = ex_get_project {code} but having a real fix would be great for the next version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (LIBCLOUD-621) Google Compute Engine Driver: ex_set_volume_auto_delete method don't work with volume parameter
[ https://issues.apache.org/jira/browse/LIBCLOUD-621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542608#comment-14542608 ] Eric Johnson commented on LIBCLOUD-621: --- This appears to be fixed. Please re-open if not the case. Google Compute Engine Driver: ex_set_volume_auto_delete method don't work with volume parameter --- Key: LIBCLOUD-621 URL: https://issues.apache.org/jira/browse/LIBCLOUD-621 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Matthias Wiesner Assignee: Eric Johnson Labels: Google On passing a StorageVolume as parameter one get the error: libcloud.common.google.InvalidRequestError: {u'locationType': u'parameter', u'domain': u'global', u'message': uInvalid value 'StorageVolume id=17316053985365710060 size=200 driver=Google Compute Engine'. Values must match the following regular expression: '\\w[\\w.-]{0,254}', u'reason': u'invalidParameter', u'location': u'deviceName'} Solution: rewrite: {{'deviceName': volume,}} to: {{'deviceName': volume.name,}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (LIBCLOUD-621) Google Compute Engine Driver: ex_set_volume_auto_delete method don't work with volume parameter
[ https://issues.apache.org/jira/browse/LIBCLOUD-621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Johnson resolved LIBCLOUD-621. --- Resolution: Fixed Google Compute Engine Driver: ex_set_volume_auto_delete method don't work with volume parameter --- Key: LIBCLOUD-621 URL: https://issues.apache.org/jira/browse/LIBCLOUD-621 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Matthias Wiesner Assignee: Eric Johnson Labels: Google On passing a StorageVolume as parameter one get the error: libcloud.common.google.InvalidRequestError: {u'locationType': u'parameter', u'domain': u'global', u'message': uInvalid value 'StorageVolume id=17316053985365710060 size=200 driver=Google Compute Engine'. Values must match the following regular expression: '\\w[\\w.-]{0,254}', u'reason': u'invalidParameter', u'location': u'deviceName'} Solution: rewrite: {{'deviceName': volume,}} to: {{'deviceName': volume.name,}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (LIBCLOUD-697) GCE Compute driver create_node does not accept str for ex_boot_disk
[ https://issues.apache.org/jira/browse/LIBCLOUD-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542590#comment-14542590 ] ASF subversion and git services commented on LIBCLOUD-697: -- Commit 8664118c877a43f1434a576bd46223707589b078 in libcloud's branch refs/heads/trunk from [~erjohnso] [ https://git-wip-us.apache.org/repos/asf?p=libcloud.git;h=8664118 ] [google compute] fixes LIBCLOUD-697: allow string for ex_boot_disk on create_node Closes #520 Signed-off-by: Eric Johnson erjoh...@google.com GCE Compute driver create_node does not accept str for ex_boot_disk --- Key: LIBCLOUD-697 URL: https://issues.apache.org/jira/browse/LIBCLOUD-697 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Jason DeTiberus Assignee: Eric Johnson The documentation for the GCE compute driver states that the ex_boot_disk param for create_node takes either a StorageVolume or str. If you pass a string, it results in the following traceback: {code:none} failed: [localhost] = {failed: true, parsed: false} Traceback (most recent call last): File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 2278, in module main() File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 577, in main module.exit_json(**(node_mgr.execute())) File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 352, in execute metadata) File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 461, in create_instances_by_names lc_network, lc_zone, metadata) File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 433, in create_instance_by_name ex_nic_gce_struct=nics File /usr/lib/python2.7/site-packages/libcloud/compute/drivers/gce.py, line 2202, in create_node ex_automatic_restart) File /usr/lib/python2.7/site-packages/libcloud/compute/drivers/gce.py, line 4507, in _create_node_req 'deviceName': boot_disk.name, AttributeError: 'str' object has no attribute 'name' {code} The issue appears to be that create_node is passing the ex_boot_disk argument directly to _create_node_req (where the boot_disk param expects a StorageVolume or None). Either the docs need to be updated or create_node should instantiate a StorageVolume from ex_boot_disk if it is a str. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] libcloud pull request: [LIBCLOUD-711] Fixed occasional CRC check f...
GitHub user chrisob opened a pull request: https://github.com/apache/libcloud/pull/519 [LIBCLOUD-711] Fixed occasional CRC check failure when decompressing ⦠â¦large responses fixes issue: https://issues.apache.org/jira/browse/LIBCLOUD-711 You can merge this pull request into a Git repository by running: $ git pull https://github.com/chrisob/libcloud LIBCLOUD-711_gzip_crc_check_fail_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/libcloud/pull/519.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #519 commit e931f2b9fd19a1d2b16e25ee44c96acb768371dc Author: Chris O'Brien chriso...@gmail.com Date: 2015-05-13T19:07:50Z [LIBCLOUD-711] Fixed occasional CRC check failure when decompressing large responses --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (LIBCLOUD-699) GCE Compute driver cannot accept ex_nic_gce_struct for create_node
[ https://issues.apache.org/jira/browse/LIBCLOUD-699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542524#comment-14542524 ] Eric Johnson commented on LIBCLOUD-699: --- Forgot to close this via commit message. Committed upstream https://git-wip-us.apache.org/repos/asf?p=libcloud.git;a=commit;h=5c1a95bf199094008c682d3cffab560095668ceb GCE Compute driver cannot accept ex_nic_gce_struct for create_node -- Key: LIBCLOUD-699 URL: https://issues.apache.org/jira/browse/LIBCLOUD-699 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Jason DeTiberus Assignee: Eric Johnson If you specify the {{ex_nic_gce_struct}} param for the {{create_node}} or {{ex_create_multiple_nodes}} Any value other than {{None}} and the test for {{if network and ex_nic_gce_struct:}} in {{_create_node_req}} fails due to network having a default value of 'default'. As a workaround I tried to set network to '', since that would have caused it to pass the comparison test, however that leads to an error from the following snippet: {code} if not hasattr(ex_network, 'name'): ex_network = self.ex_get_network(ex_network) {code} Which returns with the following error: {code} error: {'locationType': 'parameter', 'domain': 'global', 'message': Invalid value 'None'. Values must match the following regular expression: '[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?', 'reason': 'invalidParameter', 'location': 'network'} {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] libcloud pull request: [google compute] fix network options for cr...
Github user erjohnso closed the pull request at: https://github.com/apache/libcloud/pull/518 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (LIBCLOUD-711) Periodic GZIP CRC check failure
[ https://issues.apache.org/jira/browse/LIBCLOUD-711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542472#comment-14542472 ] ASF GitHub Bot commented on LIBCLOUD-711: - GitHub user chrisob opened a pull request: https://github.com/apache/libcloud/pull/519 [LIBCLOUD-711] Fixed occasional CRC check failure when decompressing … …large responses fixes issue: https://issues.apache.org/jira/browse/LIBCLOUD-711 You can merge this pull request into a Git repository by running: $ git pull https://github.com/chrisob/libcloud LIBCLOUD-711_gzip_crc_check_fail_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/libcloud/pull/519.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #519 commit e931f2b9fd19a1d2b16e25ee44c96acb768371dc Author: Chris O'Brien chriso...@gmail.com Date: 2015-05-13T19:07:50Z [LIBCLOUD-711] Fixed occasional CRC check failure when decompressing large responses Periodic GZIP CRC check failure --- Key: LIBCLOUD-711 URL: https://issues.apache.org/jira/browse/LIBCLOUD-711 Project: Libcloud Issue Type: Bug Environment: Python 2.6.6 Reporter: Chris O'Brien When attempting to parse a gzipped server response, occasionally a CRC check fails while decompressing the response body. Most of the time the response is correctly decompressed and parsed. Although the compressed data is complete (verified by writing data to file and gunzipping it), this issue only seems to happen with chunked responses. I believe this is due to the fact that the response body is unusually large (~43K uncompressed). I'm using the CloudSigma driver, and this specifically happens with the list_nodes method (the one which responds with the largest amount of data): {noformat} IOError: CRC check failed 0xdebd5ac != 0x42c31c02L ... File /home/dbs_support/dev/libcloudsigma/test.py, line 298, in _get_node nodes = self.cloud_driver.list_nodes() File /home/dbs_support/lib/libcloud/compute/drivers/cloudsigma.py, line 1025, in list_nodes response = self.connection.request(action=action, method='GET').object File /home/dbs_support/lib/libcloud/compute/drivers/cloudsigma.py, line 965, in request raw=raw) File /home/dbs_support/lib/libcloud/common/base.py, line 750, in request 'response': self.connection.getresponse()} File /home/dbs_support/lib/libcloud/common/base.py, line 404, in getresponse r, rv = self._log_response(r) File /home/dbs_support/lib/libcloud/common/base.py, line 311, in _log_response body = decompress_data('gzip', body) File /home/dbs_support/lib/libcloud/utils/compression.py, line 39, in decompress_data return gzip.GzipFile(fileobj=cls(data)).read() File /usr/lib64/python2.6/gzip.py, line 212, in read self._read(readsize) File /usr/lib64/python2.6/gzip.py, line 267, in _read self._read_eof() File /usr/lib64/python2.6/gzip.py, line 304, in _read_eof hex(self.crc))) {noformat} In utils/compression.py, would it be wise to replace line #39 ({{return gzip.GzipFile(fileobj=cls(data)).read()}}) to use zlib? This seems to fix the problem for me, but I'm unaware if there are any negative impacts. For example: {noformat} decomp = zlib.decompressobj(16+zlib.MAX_WBITS) return decomp.decompress(data) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (LIBCLOUD-697) GCE Compute driver create_node does not accept str for ex_boot_disk
[ https://issues.apache.org/jira/browse/LIBCLOUD-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542591#comment-14542591 ] ASF GitHub Bot commented on LIBCLOUD-697: - Github user asfgit closed the pull request at: https://github.com/apache/libcloud/pull/520 GCE Compute driver create_node does not accept str for ex_boot_disk --- Key: LIBCLOUD-697 URL: https://issues.apache.org/jira/browse/LIBCLOUD-697 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Jason DeTiberus Assignee: Eric Johnson The documentation for the GCE compute driver states that the ex_boot_disk param for create_node takes either a StorageVolume or str. If you pass a string, it results in the following traceback: {code:none} failed: [localhost] = {failed: true, parsed: false} Traceback (most recent call last): File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 2278, in module main() File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 577, in main module.exit_json(**(node_mgr.execute())) File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 352, in execute metadata) File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 461, in create_instances_by_names lc_network, lc_zone, metadata) File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 433, in create_instance_by_name ex_nic_gce_struct=nics File /usr/lib/python2.7/site-packages/libcloud/compute/drivers/gce.py, line 2202, in create_node ex_automatic_restart) File /usr/lib/python2.7/site-packages/libcloud/compute/drivers/gce.py, line 4507, in _create_node_req 'deviceName': boot_disk.name, AttributeError: 'str' object has no attribute 'name' {code} The issue appears to be that create_node is passing the ex_boot_disk argument directly to _create_node_req (where the boot_disk param expects a StorageVolume or None). Either the docs need to be updated or create_node should instantiate a StorageVolume from ex_boot_disk if it is a str. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (LIBCLOUD-696) GCE Compute driver returns null for image.
[ https://issues.apache.org/jira/browse/LIBCLOUD-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542826#comment-14542826 ] ASF GitHub Bot commented on LIBCLOUD-696: - Github user asfgit closed the pull request at: https://github.com/apache/libcloud/pull/522 GCE Compute driver returns null for image. -- Key: LIBCLOUD-696 URL: https://issues.apache.org/jira/browse/LIBCLOUD-696 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Jason DeTiberus Assignee: Eric Johnson The GCE Compute driver is currently returning null for image. The issue appears to be related to a change in the data returned from the api. In libcloud/compute/drivers/gce.py: {code:none|title=_to_node} extra['image'] = node.get('image') ... for disk in extra['disks']: if disk.get('boot') and disk.get('type') == 'PERSISTENT': bd = self._get_components_from_path(disk['source']) extra['boot_disk'] = self.ex_get_volume(bd['name'], bd['zone']) extra['image'] = bd['name'] ... if extra['image']: image = self._get_components_from_path(extra['image'])['name'] else: image = None {code} This looks like it should be: {code:none} for disk in extra['disks']: if disk.get('boot') and disk.get('type') == 'PERSISTENT': bd = self._get_components_from_path(disk['source']) extra['boot_disk'] = self.ex_get_volume(bd['name'], bd['zone']) extra['image'] = bd['name'] ... if extra['image']: image = self._get_components_from_path(extra['image'])['name'] else: image = None {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
libcloud git commit: [google compute] LIBCLOUD-696: improve node image detection
Repository: libcloud Updated Branches: refs/heads/trunk aaa8e9e3a - 1e3d3cbc6 [google compute] LIBCLOUD-696: improve node image detection Closes #522 Signed-off-by: Eric Johnson erjoh...@google.com Project: http://git-wip-us.apache.org/repos/asf/libcloud/repo Commit: http://git-wip-us.apache.org/repos/asf/libcloud/commit/1e3d3cbc Tree: http://git-wip-us.apache.org/repos/asf/libcloud/tree/1e3d3cbc Diff: http://git-wip-us.apache.org/repos/asf/libcloud/diff/1e3d3cbc Branch: refs/heads/trunk Commit: 1e3d3cbc6fcc6f3cc862da31b9e17ca5450b95f6 Parents: aaa8e9e Author: Eric Johnson erjoh...@google.com Authored: Wed May 13 21:52:39 2015 + Committer: Eric Johnson erjoh...@google.com Committed: Wed May 13 22:06:31 2015 + -- libcloud/compute/drivers/gce.py | 19 ++- 1 file changed, 18 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/libcloud/blob/1e3d3cbc/libcloud/compute/drivers/gce.py -- diff --git a/libcloud/compute/drivers/gce.py b/libcloud/compute/drivers/gce.py index 90c1e05..ec5b6f4 100644 --- a/libcloud/compute/drivers/gce.py +++ b/libcloud/compute/drivers/gce.py @@ -5262,6 +5262,7 @@ n extra['canIpForward'] = node.get('canIpForward') extra['serviceAccounts'] = node.get('serviceAccounts', []) extra['scheduling'] = node.get('scheduling', {}) +extra['boot_disk'] = None for disk in extra['disks']: if disk.get('boot') and disk.get('type') == 'PERSISTENT': @@ -5281,10 +5282,17 @@ n # For the node attributes, use just machine and image names, not full # paths. Full paths are available in the extra dict. +image = None if extra['image']: image = self._get_components_from_path(extra['image'])['name'] else: -image = None +if extra['boot_disk'] and \ +hasattr(extra['boot_disk'], 'extra') and \ +'sourceImage' in extra['boot_disk'].extra and \ +extra['boot_disk'].extra['sourceImage'] is not None: +src_image = extra['boot_disk'].extra['sourceImage'] +image = self._get_components_from_path(src_image)['name'] +extra['image'] = image size = self._get_components_from_path(node['machineType'])['name'] return Node(id=node['id'], name=node['name'], @@ -5422,6 +5430,15 @@ n extra['status'] = volume.get('status') extra['creationTimestamp'] = volume.get('creationTimestamp') extra['description'] = volume.get('description') +extra['sourceImage'] = volume.get('sourceImage') +extra['sourceImageId'] = volume.get('sourceImageId') +extra['sourceSnapshot'] = volume.get('sourceSnapshot') +extra['sourceSnapshotId'] = volume.get('sourceSnapshotId') +extra['options'] = volume.get('options') +if 'licenses' in volume: +lic_objs = self._licenses_from_urls(licenses=volume['licenses']) +extra['licenses'] = lic_objs + extra['type'] = volume.get('type', 'pd-standard').split('/')[-1] return StorageVolume(id=volume['id'], name=volume['name'],
[GitHub] libcloud pull request: [google compute] LIBCLOUD-696: improve node...
Github user asfgit closed the pull request at: https://github.com/apache/libcloud/pull/522 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (LIBCLOUD-696) GCE Compute driver returns null for image.
[ https://issues.apache.org/jira/browse/LIBCLOUD-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542824#comment-14542824 ] ASF subversion and git services commented on LIBCLOUD-696: -- Commit 1e3d3cbc6fcc6f3cc862da31b9e17ca5450b95f6 in libcloud's branch refs/heads/trunk from [~erjohnso] [ https://git-wip-us.apache.org/repos/asf?p=libcloud.git;h=1e3d3cb ] [google compute] LIBCLOUD-696: improve node image detection Closes #522 Signed-off-by: Eric Johnson erjoh...@google.com GCE Compute driver returns null for image. -- Key: LIBCLOUD-696 URL: https://issues.apache.org/jira/browse/LIBCLOUD-696 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Jason DeTiberus Assignee: Eric Johnson The GCE Compute driver is currently returning null for image. The issue appears to be related to a change in the data returned from the api. In libcloud/compute/drivers/gce.py: {code:none|title=_to_node} extra['image'] = node.get('image') ... for disk in extra['disks']: if disk.get('boot') and disk.get('type') == 'PERSISTENT': bd = self._get_components_from_path(disk['source']) extra['boot_disk'] = self.ex_get_volume(bd['name'], bd['zone']) extra['image'] = bd['name'] ... if extra['image']: image = self._get_components_from_path(extra['image'])['name'] else: image = None {code} This looks like it should be: {code:none} for disk in extra['disks']: if disk.get('boot') and disk.get('type') == 'PERSISTENT': bd = self._get_components_from_path(disk['source']) extra['boot_disk'] = self.ex_get_volume(bd['name'], bd['zone']) extra['image'] = bd['name'] ... if extra['image']: image = self._get_components_from_path(extra['image'])['name'] else: image = None {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (LIBCLOUD-696) GCE Compute driver returns null for image.
[ https://issues.apache.org/jira/browse/LIBCLOUD-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Johnson resolved LIBCLOUD-696. --- Resolution: Implemented GCE Compute driver returns null for image. -- Key: LIBCLOUD-696 URL: https://issues.apache.org/jira/browse/LIBCLOUD-696 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Jason DeTiberus Assignee: Eric Johnson The GCE Compute driver is currently returning null for image. The issue appears to be related to a change in the data returned from the api. In libcloud/compute/drivers/gce.py: {code:none|title=_to_node} extra['image'] = node.get('image') ... for disk in extra['disks']: if disk.get('boot') and disk.get('type') == 'PERSISTENT': bd = self._get_components_from_path(disk['source']) extra['boot_disk'] = self.ex_get_volume(bd['name'], bd['zone']) extra['image'] = bd['name'] ... if extra['image']: image = self._get_components_from_path(extra['image'])['name'] else: image = None {code} This looks like it should be: {code:none} for disk in extra['disks']: if disk.get('boot') and disk.get('type') == 'PERSISTENT': bd = self._get_components_from_path(disk['source']) extra['boot_disk'] = self.ex_get_volume(bd['name'], bd['zone']) extra['image'] = bd['name'] ... if extra['image']: image = self._get_components_from_path(extra['image'])['name'] else: image = None {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] libcloud pull request: [google compute] LIBCLOUD-696: improve node...
GitHub user erjohnso opened a pull request: https://github.com/apache/libcloud/pull/522 [google compute] LIBCLOUD-696: improve node image detection Google removed 'image' from the instance resource a while ago, but sometimes set a sourceImage on a persistent disk. This fix will consult the sourceImage attribute on the instance's (node's) boot_disk to try and determine the image. There is still no guarantee this will be set (e.g. if a persistent disk was created from a snapshot, this will still be None), but this is an improvement. Fixes https://issues.apache.org/jira/browse/LIBCLOUD-696 You can merge this pull request into a Git repository by running: $ git pull https://github.com/erjohnso/libcloud jira696 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/libcloud/pull/522.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #522 commit b5fd83947f6fb821c38618782f1de39d73a6e0a4 Author: Eric Johnson erjoh...@google.com Date: 2015-05-13T21:52:39Z [google compute] LIBCLOUD-696: improve node image detection --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (LIBCLOUD-709) Implement retry on rate limit for nova api
[ https://issues.apache.org/jira/browse/LIBCLOUD-709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpith Prakash updated LIBCLOUD-709: Description: Nova raises exception when there are too many request, and retry-after time is being sent within the header. We should be catching that exception and retry automatically which could resolve most of downstream additional logic of implement it. (was: Nova client raises exception when there are too many request, and retry-after time is being sent within the header. We should be catching that exception and retry automatically which could resolve most of downstream additional logic of implement it.) Implement retry on rate limit for nova api -- Key: LIBCLOUD-709 URL: https://issues.apache.org/jira/browse/LIBCLOUD-709 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Arpith Prakash Priority: Trivial Nova raises exception when there are too many request, and retry-after time is being sent within the header. We should be catching that exception and retry automatically which could resolve most of downstream additional logic of implement it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] libcloud pull request: [LIBCLOUD-360] [LIBCLOUD-709] Retry on rate...
GitHub user arpithkp opened a pull request: https://github.com/apache/libcloud/pull/515 [LIBCLOUD-360] [LIBCLOUD-709] Retry on rate limit and connection timeout. [Retry Logic](https://issues.apache.org/jira/browse/LIBCLOUD-360) [Rate Limit Logic](https://issues.apache.org/jira/browse/LIBCLOUD-709) You can merge this pull request into a Git repository by running: $ git pull https://github.com/arpithkp/libcloud retry_rate_limit Alternatively you can review and apply these changes as the patch at: https://github.com/apache/libcloud/pull/515.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #515 commit 38158c8e67d3539fb8aa619adc6b726479153955 Author: Arpith Kuppur Prakash arpith.kuppurprak...@rackspace.com Date: 2015-05-06T19:02:17Z [LIBCLOUD-360] [LIBCLOUD-709] Retry on rate limit and connection timeout. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (LIBCLOUD-710) CloudSigma driver doesn't take advantage of new(er) NodeStates
[ https://issues.apache.org/jira/browse/LIBCLOUD-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542265#comment-14542265 ] ASF GitHub Bot commented on LIBCLOUD-710: - GitHub user chrisob opened a pull request: https://github.com/apache/libcloud/pull/517 [LIBCLOUD-710] Updated CloudSigma's NODE_STATE_MAP to use new-ish Nod… …eStates Fixes issue: https://issues.apache.org/jira/browse/LIBCLOUD-710 You can merge this pull request into a Git repository by running: $ git pull https://github.com/chrisob/libcloud LIBCLOUD-710_cloudsigma_update_node_state_map Alternatively you can review and apply these changes as the patch at: https://github.com/apache/libcloud/pull/517.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #517 commit 011c6767c40a52588138477f22a8fe11b223da08 Author: Chris O'Brien chriso...@gmail.com Date: 2015-05-13T17:14:47Z [LIBCLOUD-710] Updated CloudSigma's NODE_STATE_MAP to use new-ish NodeStates CloudSigma driver doesn't take advantage of new(er) NodeStates -- Key: LIBCLOUD-710 URL: https://issues.apache.org/jira/browse/LIBCLOUD-710 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Chris O'Brien Priority: Minor The NodeStates 'ERROR', and 'PAUSED' are not utilized for the corresponding CloudSigma statuses (in CloudSigma_2_0_NodeDriver's NODE_STATE_MAP) 'unavailable' and 'paused'. I propose having CloudSigma's 'unavailable' state translate to NodeState.ERROR and the 'paused' state translate to NodeState.PAUSED (duh...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] libcloud pull request: [LIBCLOUD-710] Updated CloudSigma's NODE_ST...
GitHub user chrisob opened a pull request: https://github.com/apache/libcloud/pull/517 [LIBCLOUD-710] Updated CloudSigma's NODE_STATE_MAP to use new-ish Nod⦠â¦eStates Fixes issue: https://issues.apache.org/jira/browse/LIBCLOUD-710 You can merge this pull request into a Git repository by running: $ git pull https://github.com/chrisob/libcloud LIBCLOUD-710_cloudsigma_update_node_state_map Alternatively you can review and apply these changes as the patch at: https://github.com/apache/libcloud/pull/517.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #517 commit 011c6767c40a52588138477f22a8fe11b223da08 Author: Chris O'Brien chriso...@gmail.com Date: 2015-05-13T17:14:47Z [LIBCLOUD-710] Updated CloudSigma's NODE_STATE_MAP to use new-ish NodeStates --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (LIBCLOUD-699) GCE Compute driver cannot accept ex_nic_gce_struct for create_node
[ https://issues.apache.org/jira/browse/LIBCLOUD-699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542367#comment-14542367 ] ASF GitHub Bot commented on LIBCLOUD-699: - GitHub user erjohnso opened a pull request: https://github.com/apache/libcloud/pull/518 [google compute] fix network options for create_node Minor fix to detect network option conflicts as described in https://issues.apache.org/jira/browse/LIBCLOUD-699 You can merge this pull request into a Git repository by running: $ git pull https://github.com/erjohnso/libcloud jira699 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/libcloud/pull/518.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #518 GCE Compute driver cannot accept ex_nic_gce_struct for create_node -- Key: LIBCLOUD-699 URL: https://issues.apache.org/jira/browse/LIBCLOUD-699 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Jason DeTiberus Assignee: Eric Johnson If you specify the {{ex_nic_gce_struct}} param for the {{create_node}} or {{ex_create_multiple_nodes}} Any value other than {{None}} and the test for {{if network and ex_nic_gce_struct:}} in {{_create_node_req}} fails due to network having a default value of 'default'. As a workaround I tried to set network to '', since that would have caused it to pass the comparison test, however that leads to an error from the following snippet: {code} if not hasattr(ex_network, 'name'): ex_network = self.ex_get_network(ex_network) {code} Which returns with the following error: {code} error: {'locationType': 'parameter', 'domain': 'global', 'message': Invalid value 'None'. Values must match the following regular expression: '[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?', 'reason': 'invalidParameter', 'location': 'network'} {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] libcloud pull request: [google compute] fix network options for cr...
GitHub user erjohnso opened a pull request: https://github.com/apache/libcloud/pull/518 [google compute] fix network options for create_node Minor fix to detect network option conflicts as described in https://issues.apache.org/jira/browse/LIBCLOUD-699 You can merge this pull request into a Git repository by running: $ git pull https://github.com/erjohnso/libcloud jira699 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/libcloud/pull/518.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #518 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (LIBCLOUD-699) GCE Compute driver cannot accept ex_nic_gce_struct for create_node
[ https://issues.apache.org/jira/browse/LIBCLOUD-699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Johnson resolved LIBCLOUD-699. --- Resolution: Implemented GCE Compute driver cannot accept ex_nic_gce_struct for create_node -- Key: LIBCLOUD-699 URL: https://issues.apache.org/jira/browse/LIBCLOUD-699 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Jason DeTiberus Assignee: Eric Johnson If you specify the {{ex_nic_gce_struct}} param for the {{create_node}} or {{ex_create_multiple_nodes}} Any value other than {{None}} and the test for {{if network and ex_nic_gce_struct:}} in {{_create_node_req}} fails due to network having a default value of 'default'. As a workaround I tried to set network to '', since that would have caused it to pass the comparison test, however that leads to an error from the following snippet: {code} if not hasattr(ex_network, 'name'): ex_network = self.ex_get_network(ex_network) {code} Which returns with the following error: {code} error: {'locationType': 'parameter', 'domain': 'global', 'message': Invalid value 'None'. Values must match the following regular expression: '[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?', 'reason': 'invalidParameter', 'location': 'network'} {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (LIBCLOUD-697) GCE Compute driver create_node does not accept str for ex_boot_disk
[ https://issues.apache.org/jira/browse/LIBCLOUD-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542580#comment-14542580 ] ASF GitHub Bot commented on LIBCLOUD-697: - GitHub user erjohnso opened a pull request: https://github.com/apache/libcloud/pull/520 [google compute] bugfix: allow string for ex_boot_disk on create_node Bugfix to allow a `str` argument for `ex_boot_disk` on `create_node()`. This is a fix for https://issues.apache.org/jira/browse/LIBCLOUD-697 You can merge this pull request into a Git repository by running: $ git pull https://github.com/erjohnso/libcloud jira697 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/libcloud/pull/520.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #520 commit 69ec7aa0bae33ad77d668b7c10e123f8a30e1e83 Author: Eric Johnson erjoh...@google.com Date: 2015-05-13T19:53:32Z [google compute] bugfix: allow string for ex_boot_disk on create_node GCE Compute driver create_node does not accept str for ex_boot_disk --- Key: LIBCLOUD-697 URL: https://issues.apache.org/jira/browse/LIBCLOUD-697 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Jason DeTiberus Assignee: Eric Johnson The documentation for the GCE compute driver states that the ex_boot_disk param for create_node takes either a StorageVolume or str. If you pass a string, it results in the following traceback: {code:none} failed: [localhost] = {failed: true, parsed: false} Traceback (most recent call last): File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 2278, in module main() File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 577, in main module.exit_json(**(node_mgr.execute())) File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 352, in execute metadata) File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 461, in create_instances_by_names lc_network, lc_zone, metadata) File /home/jdetiber/.ansible/tmp/ansible-tmp-1429541392.81-113800638314850/gce, line 433, in create_instance_by_name ex_nic_gce_struct=nics File /usr/lib/python2.7/site-packages/libcloud/compute/drivers/gce.py, line 2202, in create_node ex_automatic_restart) File /usr/lib/python2.7/site-packages/libcloud/compute/drivers/gce.py, line 4507, in _create_node_req 'deviceName': boot_disk.name, AttributeError: 'str' object has no attribute 'name' {code} The issue appears to be that create_node is passing the ex_boot_disk argument directly to _create_node_req (where the boot_disk param expects a StorageVolume or None). Either the docs need to be updated or create_node should instantiate a StorageVolume from ex_boot_disk if it is a str. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] libcloud pull request: [google compute] bugfix: allow string for e...
GitHub user erjohnso opened a pull request: https://github.com/apache/libcloud/pull/520 [google compute] bugfix: allow string for ex_boot_disk on create_node Bugfix to allow a `str` argument for `ex_boot_disk` on `create_node()`. This is a fix for https://issues.apache.org/jira/browse/LIBCLOUD-697 You can merge this pull request into a Git repository by running: $ git pull https://github.com/erjohnso/libcloud jira697 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/libcloud/pull/520.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #520 commit 69ec7aa0bae33ad77d668b7c10e123f8a30e1e83 Author: Eric Johnson erjoh...@google.com Date: 2015-05-13T19:53:32Z [google compute] bugfix: allow string for ex_boot_disk on create_node --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
libcloud git commit: [google compute] fixes LIBCLOUD-697: allow string for ex_boot_disk on create_node
Repository: libcloud Updated Branches: refs/heads/trunk 5c1a95bf1 - 8664118c8 [google compute] fixes LIBCLOUD-697: allow string for ex_boot_disk on create_node Closes #520 Signed-off-by: Eric Johnson erjoh...@google.com Project: http://git-wip-us.apache.org/repos/asf/libcloud/repo Commit: http://git-wip-us.apache.org/repos/asf/libcloud/commit/8664118c Tree: http://git-wip-us.apache.org/repos/asf/libcloud/tree/8664118c Diff: http://git-wip-us.apache.org/repos/asf/libcloud/diff/8664118c Branch: refs/heads/trunk Commit: 8664118c877a43f1434a576bd46223707589b078 Parents: 5c1a95b Author: Eric Johnson erjoh...@google.com Authored: Wed May 13 19:53:32 2015 + Committer: Eric Johnson erjoh...@google.com Committed: Wed May 13 20:01:46 2015 + -- libcloud/compute/drivers/gce.py | 10 ++ 1 file changed, 6 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/libcloud/blob/8664118c/libcloud/compute/drivers/gce.py -- diff --git a/libcloud/compute/drivers/gce.py b/libcloud/compute/drivers/gce.py index 22f3f9a..4f15715 100644 --- a/libcloud/compute/drivers/gce.py +++ b/libcloud/compute/drivers/gce.py @@ -2253,7 +2253,7 @@ class GCENodeDriver(NodeDriver): :type ex_metadata: ``dict`` or ``None`` :keyword ex_boot_disk: The boot disk to attach to the instance. -:type ex_boot_disk: :class:`StorageVolume` or ``str`` +:type ex_boot_disk: :class:`StorageVolume` or ``str`` or ``None`` :keyword use_existing_disk: If True and if an existing disk with the same name/location is found, use that @@ -2265,7 +2265,7 @@ class GCENodeDriver(NodeDriver): used. If 'None', then no external address will be used. To use an existing static IP address, a GCEAddress object should be passed in. -:type external_ip: :class:`GCEAddress` or ``str`` or None +:type external_ip: :class:`GCEAddress` or ``str`` or ``None`` :keyword ex_disk_type: Specify a pd-standard (default) disk or pd-ssd for an SSD disk. @@ -2356,6 +2356,8 @@ n image = self.ex_get_image(image) if not hasattr(ex_disk_type, 'name'): ex_disk_type = self.ex_get_disktype(ex_disk_type, zone=location) +if ex_boot_disk and not hasattr(ex_boot_disk, 'name'): +ex_boot_disk = self.ex_get_disk(ex_boot_disk, zone=location) # Use disks[].initializeParams to auto-create the boot disk if not ex_disks_gce_struct and not ex_boot_disk: @@ -4601,7 +4603,7 @@ n is made to ensure proper formatting of the disks[] structure. Using this structure obviates the need of using - other disk params like 'ex_boot_disk', + other disk params like 'boot_disk', etc. See the GCE docs for specific details. :type ex_disks_gce_struct: ``list`` or ``None`` @@ -4840,7 +4842,7 @@ n request, node_data = self._create_node_req( status['name'], node_attrs['size'], node_attrs['image'], node_attrs['location'], node_attrs['network'], node_attrs['tags'], -node_attrs['metadata'], boot_disk=status['disk'], +node_attrs['metadata'], external_ip=node_attrs['external_ip'], ex_service_accounts=node_attrs['ex_service_accounts'], description=node_attrs['description'],
[jira] [Resolved] (LIBCLOUD-687) Unable to set GCE metadata if no metadata already exists
[ https://issues.apache.org/jira/browse/LIBCLOUD-687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Johnson resolved LIBCLOUD-687. --- Resolution: Implemented Unable to set GCE metadata if no metadata already exists Key: LIBCLOUD-687 URL: https://issues.apache.org/jira/browse/LIBCLOUD-687 Project: Libcloud Issue Type: Bug Components: Compute Reporter: Eli Courtwright Assignee: Eric Johnson The {{libcloud.compute.drivers.gce.GCENodeDriver}} class has a nonstandard method {{ex_set_common_metadata}} which we can use to (among other things) set SSH keys. When calling this method on the latest libcloud 0.17.0 (which is not currently a listed version in Jira), I get the following error: {code} conn.ex_set_common_instance_metadata({}) Traceback (most recent call last): File stdin, line 1, in module File /opt/sideboard/lib/python2.7/site-packages/libcloud/compute/drivers/gce.py, line 1129, in ex_set_common_instance_metadata for md in current_metadata['items']: KeyError: 'items' {code} I get this error regardless of what data I pass to the {{ex_set_common_instance_metadata}} method. From delving into the libcloud code, I find the following block of code: {code} project = self.ex_get_project() current_metadata = project.extra['commonInstanceMetadata'] fingerprint = current_metadata['fingerprint'] # grab copy of current 'sshKeys' in case we want to retain them current_keys = for md in current_metadata['items']: {code} So it looks as if the method assumes that {{self.ex_get_project()}} will return: - an object with a {{.extra}} dictionary (which it does) - that dictionary will have a commonInstanceMetadata key which maps to another dictionary (which is correct) - that sub-dictionary will have an items key whose value is iterable (which is wrong) That last assumption is only wrong if there is no project metadata already associated with the project, e.g. for a new project. I'm working around this problem currently by monkeypatching libcloud: {code} def patch_gce_ex_get_project(): orig_get_project = GCENodeDriver.ex_get_project def ex_get_project(self): project = orig_get_project(self) project.extra['commonInstanceMetadata'].setdefault('items', []) return project GCENodeDriver.ex_get_project = ex_get_project {code} but having a real fix would be great for the next version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)