[Cloud-init-dev] [Merge] ~tribaal/cloud-init:migrate-lp-to-github into cloud-init:master
Chris Glass has proposed merging ~tribaal/cloud-init:migrate-lp-to-github into cloud-init:master. Commit message: lp-to-git-users: adding chrisglass Mapped from tribaal Requested reviews: cloud-init Commiters (cloud-init-dev) For more details, see: https://code.launchpad.net/~tribaal/cloud-init/+git/cloud-init/+merge/375453 -- Your team cloud-init Commiters is requested to review the proposed merge of ~tribaal/cloud-init:migrate-lp-to-github into cloud-init:master. diff --git a/tools/.lp-to-git-user b/tools/.lp-to-git-user index 04c7a9e..ec1586b 100644 --- a/tools/.lp-to-git-user +++ b/tools/.lp-to-git-user @@ -2,5 +2,6 @@ "chad.smith": "blackboxsw", "legovini": "paride", "powersj": "powersj", - "raharper": "raharper" + "raharper": "raharper", + "tribaal": "chrisglass" } \ No newline at end of file ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
Re: [Cloud-init-dev] [Merge] ~tribaal/cloud-init:fix/exoscale-datasource-wait-timeout into cloud-init:master
Note: retrying 12 times on average might be overkill here - but we rather be pessimistic and add a lot of buffer. -- https://code.launchpad.net/~tribaal/cloud-init/+git/cloud-init/+merge/374643 Your team cloud-init Commiters is requested to review the proposed merge of ~tribaal/cloud-init:fix/exoscale-datasource-wait-timeout into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
[Cloud-init-dev] [Merge] ~tribaal/cloud-init:fix/exoscale-datasource-wait-timeout into cloud-init:master
Chris Glass has proposed merging
~tribaal/cloud-init:fix/exoscale-datasource-wait-timeout into cloud-init:master.
Requested reviews:
cloud-init Commiters (cloud-init-dev)
For more details, see:
https://code.launchpad.net/~tribaal/cloud-init/+git/cloud-init/+merge/374643
This change sets the url_max_wait for the exoscale datasource to 120 (similar
to what is done in the EC2/Cloudstack datasources).
In some rare cases the route to the datasource IP address is not available
under 10 seconds (the datasource's default timeout), and not setting
url_max_wait results in only a single attempt being made (the default value
being -1).
--
Your team cloud-init Commiters is requested to review the proposed merge of
~tribaal/cloud-init:fix/exoscale-datasource-wait-timeout into cloud-init:master.
diff --git a/cloudinit/sources/DataSourceExoscale.py b/cloudinit/sources/DataSourceExoscale.py
index fdfb4ed..4616daa 100644
--- a/cloudinit/sources/DataSourceExoscale.py
+++ b/cloudinit/sources/DataSourceExoscale.py
@@ -26,6 +26,8 @@ class DataSourceExoscale(sources.DataSource):
dsname = 'Exoscale'
+url_max_wait = 120
+
def __init__(self, sys_cfg, distro, paths):
super(DataSourceExoscale, self).__init__(sys_cfg, distro, paths)
LOG.debug("Initializing the Exoscale datasource")
___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help : https://help.launchpad.net/ListHelp
Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:bug/1841454-exoscale-config-modules into cloud-init:master
Review: Approve Building a deb with this branch and grafting it onto an Eoan machine does what it says on the tin: the set-password module is called with frequency "always", yet the list of modules to run is not overwritten. In particular, running with userdata making use of runcmd still works (which is how we initially discovered the problem. Relevant logs on an Eoan machine: https://pastebin.ubuntu.com/p/MYZR56675X/ -- https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/371823 Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:bug/1841454-exoscale-config-modules into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
Re: [Cloud-init-dev] [Merge] ~tribaal/cloud-init:feat/datasource-exoscale into cloud-init:master
A first pass fixing most review comments is done.
One open point here is the extra_config overwriting user-data - it seems to me
that perhaps we should be using some other mechanism instead (my assumption was
that extra_config would be overwritten by user data - but that doesn't seem to
be the case).
Should we use "vendordata" instead for this particular case?
Diff comments:
> diff --git a/cloudinit/sources/DataSourceExoscale.py
> b/cloudinit/sources/DataSourceExoscale.py
> new file mode 100644
> index 000..4588c9d
> --- /dev/null
> +++ b/cloudinit/sources/DataSourceExoscale.py
> @@ -0,0 +1,126 @@
> +import time
> +
> +from cloudinit import ec2_utils as ec2
> +from cloudinit import log as logging
> +from cloudinit import sources
> +from cloudinit import url_helper
> +
> +API_VERSION = "latest"
Excellent point - I switched the version to "1.0", which happens to be an alias
for "latest" for now. At least this way the version won't change under our
user's feet as you pointed out.
> +LOG = logging.getLogger(__name__)
> +SERVICE_ADDRESS = "http://169.254.169.254;
> +
> +
> +class DataSourceExoscale(sources.DataSource):
> +
> +dsname = 'Exoscale'
> +url_timeout = 10
> +url_retries = 6
> +url_max_wait = 60
> +
> +def __init__(self, sys_cfg, distro, paths):
> +sources.DataSource.__init__(self, sys_cfg, distro, paths)
> +LOG.info("Initializing the Exoscale datasource")
> +self.extra_config = {}
> +
> +def get_password(self):
We would rather not reuse any code from the Cloudstack datasource, if it isn't
a blocker.
In particular: we prefer using python (via read_file_or_url()) to perform HTTP
requests, instead of shelling out to wget as the Cloudstack source currently
does.
> +"""Return the VM's passwords."""
> +LOG.info("Fetching password from metadata service")
Done.
> +password_url = "{}:8080".format(SERVICE_ADDRESS)
> +response = url_helper.read_file_or_url(
> +password_url,
> +ssl_details=None,
> +headers={"DomU_Request": "send_my_password"},
> +timeout=self.url_timeout,
> +retries=self.url_retries)
> +password = response.contents.decode('utf-8')
> +# the password is empty or already saved
> +if password in ['', 'saved_password']:
> +LOG.info("Password is missing or already saved")
> +return None
> +LOG.info("Found the password in metadata service")
> +# save the password
> +url_helper.read_file_or_url(
> +password_url,
> +ssl_details=None,
> +headers={"DomU_Request": "saved_password"},
> +timeout=self.url_timeout,
> +retries=self.url_retries)
> +LOG.info("password saved")
> +return password
> +
> +def wait_for_metadata_service(self):
> +"""Wait for the metadata service to be reachable."""
> +LOG.info("waiting for the metadata service")
> +start_time = time.time()
> +
> +metadata_url = "{}/{}/meta-data/instance-id".format(
> +SERVICE_ADDRESS,
> +API_VERSION)
> +
> +start_time = time.time()
Done.
This was tackled by letting url_helper.readurl() do the logging when timing out
(it was redundant).
The general time it took for us to get the metadata is now logged in
_get_data() (using util.log_time() as recommended).
> +url = url_helper.wait_for_url(
> +urls=[metadata_url],
> +max_wait=self.url_max_wait,
> +timeout=self.url_timeout,
> +status_cb=LOG.critical)
> +
> +if url:
> +LOG.info("metadata service ok")
Done.
> +return True
> +else:
> +wait_time = int(time.time() - start_time)
> +LOG.critical(("Giving up on waiting for the metadata from %s"
> + " after %s seconds"),
> + url,
> + wait_time)
> +return False
> +
> +def _get_data(self):
> +"""Fetch the user data, the metadata and the VM password
> +from the metadata service."""
> +LOG.info("fetching data")
Done.
> +if not self.wait_for_metadata_service():
> +return False
> +start_time = time.time()
> +self.userdata_raw = ec2.get_instance_userdata(API_VERSION,
Done.
The logic now goes:
self._get_data() -> self.crawl_metadata() -> read_data() -> get_password()
Added a simple __main__ to call read_data() directly (that performs a
json.dumps on stdout)
> + SERVICE_ADDRESS,
> +
> timeout=self.url_timeout,
> +
> retries=self.url_retries)
> +self.metadata = ec2.get_instance_metadata(API_VERSION,
> +
[Cloud-init-dev] [Merge] ~tribaal/cloud-init:feat/datasource-exoscale into cloud-init:master
The proposal to merge ~tribaal/cloud-init:feat/datasource-exoscale into cloud-init:master has been updated. Commit message changed to: New data source for the Exoscale.com cloud platform - dsidentify switches to the new Exoscale datasource on matching DMI name - New Exoscale datasource added Signed-off-by: Chris Glass Author: Mathieu Corbin For more details, see: https://code.launchpad.net/~tribaal/cloud-init/+git/cloud-init/+merge/369516 -- Your team cloud-init commiters is requested to review the proposed merge of ~tribaal/cloud-init:feat/datasource-exoscale into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
[Cloud-init-dev] [Merge] ~tribaal/cloud-init:feat/datasource-exoscale into cloud-init:master
Chris Glass has proposed merging ~tribaal/cloud-init:feat/datasource-exoscale
into cloud-init:master.
Commit message:
New data source for the Exoscale.com cloud platform
- dsidentify switches to the new Exoscale datasource on matching DMI name
- New Exoscale datasource added
Requested reviews:
cloud-init commiters (cloud-init-dev)
For more details, see:
https://code.launchpad.net/~tribaal/cloud-init/+git/cloud-init/+merge/369516
New data source for the Exoscale.com cloud platform
While the Exoscale cloud platform started as a Cloudstack fork, over the years
more and more of the codebase was rewritten, replaced, or otherwise
fundamentally changed.
Our recent feature development around networking made it clear that while we
intend for our cloud platform to remain as compatible with the Cloudstack
datasource in the forseeable future (once forced - see below), it is time for
us to bite the proverbial bullet and submit our own datasource for review.
Networking glitch:
The Cloudstack datasource doesn't work for us in the scenario where a user boots
a VM attached to the default network and one or more private network(s) running
DHCP (hosted by us, or customer-deployed): in some race condition the private
interface will be the lease on which the datasource will try to query for
metadata - which can't work. The submitted datasource fixes the problem by
forcing the datasource IP address to always be the same, so that we don't need
to depend on DHCP at all to determine it, fixing the race (we ensure routes to
that address are provided).
DMI product name change:
Along with this change, we intend to switch the DMI product name from
"CloudStack KVM Hypervisor" to "Exoscale Compute Platform" and the DMI
manufacturer string from "Apache Software Foundation" to "Exoscale".
Current templates situation:
We currently do not advertise custom templates to our customers, but intend to
do
so more widely in the near future.
All of our current production templates force the Cloudstack datasource usage
(via configuration in /etc/cloud*).
Once this code lands, and our DMI product name change is performed, we will
document the requirements for (future) custom templates to be either:
- A recent enough cloud-init (containing the proposed code)
- Forcing the datasource to be "Cloudstack" for older cloud-init versions. This
will trigger the above-mentioned race, but that is a specific, relatively rare
use-case.
- Certain (future) features might only be available to customers running a
recent-enough cloud-init version.
Testing this code:
Inclusion in the cloud-init test suite in addition to the submitted unit tests
would be welcome.
--
Your team cloud-init commiters is requested to review the proposed merge of
~tribaal/cloud-init:feat/datasource-exoscale into cloud-init:master.
diff --git a/cloudinit/apport.py b/cloudinit/apport.py
index 22cb7fd..003ff1f 100644
--- a/cloudinit/apport.py
+++ b/cloudinit/apport.py
@@ -23,6 +23,7 @@ KNOWN_CLOUD_NAMES = [
'CloudStack',
'DigitalOcean',
'GCE - Google Compute Engine',
+'Exoscale',
'Hetzner Cloud',
'IBM - (aka SoftLayer or BlueMix)',
'LXD',
diff --git a/cloudinit/settings.py b/cloudinit/settings.py
index b1ebaad..2060d81 100644
--- a/cloudinit/settings.py
+++ b/cloudinit/settings.py
@@ -39,6 +39,7 @@ CFG_BUILTIN = {
'Hetzner',
'IBMCloud',
'Oracle',
+'Exoscale',
# At the end to act as a 'catch' when none of the above work...
'None',
],
diff --git a/cloudinit/sources/DataSourceExoscale.py b/cloudinit/sources/DataSourceExoscale.py
new file mode 100644
index 000..4588c9d
--- /dev/null
+++ b/cloudinit/sources/DataSourceExoscale.py
@@ -0,0 +1,126 @@
+import time
+
+from cloudinit import ec2_utils as ec2
+from cloudinit import log as logging
+from cloudinit import sources
+from cloudinit import url_helper
+
+API_VERSION = "latest"
+LOG = logging.getLogger(__name__)
+SERVICE_ADDRESS = "http://169.254.169.254;
+
+
+class DataSourceExoscale(sources.DataSource):
+
+dsname = 'Exoscale'
+url_timeout = 10
+url_retries = 6
+url_max_wait = 60
+
+def __init__(self, sys_cfg, distro, paths):
+sources.DataSource.__init__(self, sys_cfg, distro, paths)
+LOG.info("Initializing the Exoscale datasource")
+self.extra_config = {}
+
+def get_password(self):
+"""Return the VM's passwords."""
+LOG.info("Fetching password from metadata service")
+password_url = "{}:8080".format(SERVICE_ADDRESS)
+response = url_helper.read_file_or_url(
+password_url,
+ssl_details=None,
+headers={"DomU_Request": "send_my_password"},
+timeout=self.url_timeout,
+retries=self.url_retries)
+password = response.contents.decode('utf
[Cloud-init-dev] [Merge] ~tribaal/cloud-init:dont-overwrite-etc-hosts-in-example into cloud-init:master
Chris Glass has proposed merging ~tribaal/cloud-init:dont-overwrite-etc-hosts-in-example into cloud-init:master. Commit message: Make the "command on first boot" example append to /etc/hosts instead of overwriting. Requested reviews: cloud init development team (cloud-init-dev) For more details, see: https://code.launchpad.net/~tribaal/cloud-init/+git/cloud-init/+merge/310043 This branch simply fixes the "Run commands on first boot" example to append to /etc/hosts instead of overwriting it. It's only an example, so hopefully people don't run it as-is, but it is probably good to have an accurate example. This also served me as a learning experience to "kick the tires" for the contribution process, should the need to contribute further arise. Should my addition be wrong or misguided please feel free to close this merge request without further notice. -- Your team cloud init development team is requested to review the proposed merge of ~tribaal/cloud-init:dont-overwrite-etc-hosts-in-example into cloud-init:master. diff --git a/doc/examples/cloud-config-boot-cmds.txt b/doc/examples/cloud-config-boot-cmds.txt index 3e59755..5c05bef 100644 --- a/doc/examples/cloud-config-boot-cmds.txt +++ b/doc/examples/cloud-config-boot-cmds.txt @@ -11,5 +11,5 @@ # - the INSTANCE_ID variable will be set to the current instance id. # - you can use 'cloud-init-boot-per' command to help only run once bootcmd: - - echo 192.168.1.130 us.archive.ubuntu.com > /etc/hosts + - echo 192.168.1.130 us.archive.ubuntu.com >> /etc/hosts - [ cloud-init-per, once, mymkfs, mkfs, /dev/vdb ] ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
