[Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
The proposal to merge ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master has been updated. Description changed to: Some sample instance-data.json on lxd EC2 and OpenStack - https://pastebin.ubuntu.com/p/52BvK7Yr3V/ More sample instance-data.json: azure: http://paste.ubuntu.com/p/tTQrwJMPBt/ rackspace: http://paste.ubuntu.com/p/73bhkBTx83/ For more details, see: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 -- Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
[Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
The proposal to merge ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master has been updated. Status: Approved => Merged For more details, see: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 -- Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
Review: Approve continuous-integration -- https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
[Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
The proposal to merge ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master has been updated. Status: Needs review => Approved For more details, see: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 -- Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
[Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
The proposal to merge ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master has been updated. Commit message changed to: instance-data: Add standard keys platform and subplatform. Refactor ec2. Add the following instance-data.json standardized keys: * v1._beta_keys: List any v1 keys in beta development, e.g. ['subplatform']. * v1.public_ssh_keys: List of any cloud-provided ssh keys for the instance. * v1.platform: String representing the cloud platform api supporting the datasource. For example: 'ec2' for aws, aliyun and brightbox cloud names. * v1.subplatform: String with more details about the source of the metadata consumed. For example, metadata uri, config drive device path or seed directory. To support the new platform and subplatform standardized instance-data, DataSource and its subclasses grew platform and subplatform attributes. The platform attribute defaults to the lowercase string datasource name at self.dsname. This method is overridden in NoCloud, Ec2 and ConfigDrive datasources. The subplatform attribute calls a _get_subplatform method which will return a string containing a simple slug for subplatform type such as metadata, seed-dir or config-drive followed by a detailed uri, device or directory path where the datasource consumed its configuration. As part of this work, DatasourceEC2 methods _get_data and _crawl_metadata have been refactored for a few reasons: - crawl_metadata is now a read-only operation, persisting no attributes on the datasource instance and returns a dictionary of consumed metadata. - crawl_metadata now closely represents the raw stucture of the ec2 metadata consumed, so that end-users can leverage public ec2 metadata documentation where possible. - crawl_metadata adds a '_metadata_api_version' key to the crawled ds.metadata to advertise what version of EC2's api was consumed by cloud-init. - _get_data now does all the processing of crawl_metadata and saves datasource instance attributes userdata_raw, metadata etc. Additional drive-bys: * unit test rework for test_altcloud and test_azure to simplify mocks and make use of existing util and test_helpers functions. For more details, see: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 -- Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
[Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
The proposal to merge ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master has been updated. Status: Needs review => Approved For more details, see: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 -- Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
Review: Approve continuous-integration PASSED: Continuous integration, rev:e3fb8a4f3a1438bf02189217005ca77b33c624b5 https://jenkins.ubuntu.com/server/job/cloud-init-ci/380/ Executed test runs: SUCCESS: Checkout SUCCESS: Unit & Style Tests SUCCESS: Ubuntu LTS: Build SUCCESS: Ubuntu LTS: Integration IN_PROGRESS: Declarative: Post Actions Click here to trigger a rebuild: https://jenkins.ubuntu.com/server/job/cloud-init-ci/380/rebuild -- https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
Diff comments:
> diff --git a/cloudinit/sources/DataSourceEc2.py
> b/cloudinit/sources/DataSourceEc2.py
> index 968ab3f..28f08e4 100644
> --- a/cloudinit/sources/DataSourceEc2.py
> +++ b/cloudinit/sources/DataSourceEc2.py
> @@ -306,13 +303,11 @@ class DataSourceEc2(sources.DataSource):
> result = None
> no_network_metadata_on_aws = bool(
> 'network' not in self.metadata and
> -self.cloud_platform == Platforms.AWS)
> +self.cloud_name == CloudNames.AWS)
> if no_network_metadata_on_aws:
> LOG.debug("Metadata 'network' not present:"
>" Refreshing stale metadata from prior to upgrade.")
> -util.log_time(
> -logfunc=LOG.debug, msg='Re-crawl of metadata service',
> -func=self._crawl_metadata)
> +self.get_data()
ahh good point. Originally since Ec2, _get_data already did a time wrapper
around crawl_metadata inside _get_data. So, I didn't want the double wrapped
timer, but there is a path potentially where we would also need to perform
sendboxed dhcp before crawl_metadata, so it'd be nice to see timestamps around
that effort as well as the crawl_metadata so I'll put it back in. Also
discovered a path where we didn't log_time of crawl_metadata if no-network
sandbox was required so I added that too.
>
> # Limit network configuration to only the primary/fallback nic
> iface = self.fallback_interface
> @@ -340,28 +335,32 @@ class DataSourceEc2(sources.DataSource):
> return super(DataSourceEc2, self).fallback_interface
> return self._fallback_interface
>
> -def _crawl_metadata(self):
> +def crawl_metadata(self):
> """Crawl metadata service when available.
>
> -@returns: True on success, False otherwise.
> +@returns: Dictionary of craweled metadata content containing the
> keys:
fixed.
> + meta-data, user-data and dynamic.
> """
> if not self.wait_for_metadata_service():
> -return False
> +return {}
> api_version = self.get_metadata_api_version()
> +crawled_metadata = {}
> try:
> -self.userdata_raw = ec2.get_instance_userdata(
> +crawled_metadata['user-data'] = ec2.get_instance_userdata(
> api_version, self.metadata_address)
> -self.metadata = ec2.get_instance_metadata(
> +crawled_metadata['meta-data'] = ec2.get_instance_metadata(
> api_version, self.metadata_address)
> -if self.cloud_platform == Platforms.AWS:
> -self.identity = ec2.get_instance_identity(
> -api_version, self.metadata_address).get('document', {})
> +if self.cloud_name == CloudNames.AWS:
> +identity = ec2.get_instance_identity(
> +api_version, self.metadata_address)
> +crawled_metadata['dynamic'] = {'instance-identity': identity}
> except Exception:
> util.logexc(
> LOG, "Failed reading from metadata address %s",
> self.metadata_address)
> -return False
> -return True
> +return {}
> +crawled_metadata['_metadata_api_version'] = api_version
> +return crawled_metadata
>
>
> class DataSourceEc2Local(DataSourceEc2):
> diff --git a/doc/rtd/topics/instancedata.rst b/doc/rtd/topics/instancedata.rst
> index 634e180..9c30bc1 100644
> --- a/doc/rtd/topics/instancedata.rst
> +++ b/doc/rtd/topics/instancedata.rst
> @@ -90,24 +90,39 @@ There are three basic top-level keys:
>
> The standardized keys present:
>
> -+--+---+---+
> -| Key path| Description |
> Examples |
> -+==+===+===+
> -| v1.cloud_name| The name of the cloud provided by metadata|
> aws, openstack, azure,|
> -| | key 'cloud-name' or the cloud-init datasource |
> configdrive, nocloud, |
> -| | name which was discovered.|
> ovf, etc. |
> -+--+---+---+
> -| v1.instance_id | Unique instance_id allocated by the cloud |
> i- |
> -+--+---+---+
> -| v1.local_hostname| The internal or local hostname of the system |
> ip-10-41-41-70, |
> -| | |
> |
> -+--+---+---+
> -| v1.region
Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
if manual_cache_clean is set, then we should be careful not to re-crawl
anything on upgrade. This is most likely a case when there was a "config disk"
in the generic sense. Ie, where a datasource provided a disk (cdrom or block
device) and then later pulled it or user formatted it.
Diff comments:
> diff --git a/cloudinit/sources/DataSourceEc2.py
> b/cloudinit/sources/DataSourceEc2.py
> index 968ab3f..28f08e4 100644
> --- a/cloudinit/sources/DataSourceEc2.py
> +++ b/cloudinit/sources/DataSourceEc2.py
> @@ -306,13 +303,11 @@ class DataSourceEc2(sources.DataSource):
> result = None
> no_network_metadata_on_aws = bool(
> 'network' not in self.metadata and
> -self.cloud_platform == Platforms.AWS)
> +self.cloud_name == CloudNames.AWS)
> if no_network_metadata_on_aws:
> LOG.debug("Metadata 'network' not present:"
>" Refreshing stale metadata from prior to upgrade.")
> -util.log_time(
> -logfunc=LOG.debug, msg='Re-crawl of metadata service',
> -func=self._crawl_metadata)
> +self.get_data()
any reason you dropped the time on that?
probably not important, but curious.
>
> # Limit network configuration to only the primary/fallback nic
> iface = self.fallback_interface
> @@ -340,28 +335,32 @@ class DataSourceEc2(sources.DataSource):
> return super(DataSourceEc2, self).fallback_interface
> return self._fallback_interface
>
> -def _crawl_metadata(self):
> +def crawl_metadata(self):
> """Crawl metadata service when available.
>
> -@returns: True on success, False otherwise.
> +@returns: Dictionary of craweled metadata content containing the
> keys:
craweled -> crawled
> + meta-data, user-data and dynamic.
> """
> if not self.wait_for_metadata_service():
> -return False
> +return {}
> api_version = self.get_metadata_api_version()
> +crawled_metadata = {}
> try:
> -self.userdata_raw = ec2.get_instance_userdata(
> +crawled_metadata['user-data'] = ec2.get_instance_userdata(
> api_version, self.metadata_address)
> -self.metadata = ec2.get_instance_metadata(
> +crawled_metadata['meta-data'] = ec2.get_instance_metadata(
> api_version, self.metadata_address)
> -if self.cloud_platform == Platforms.AWS:
> -self.identity = ec2.get_instance_identity(
> -api_version, self.metadata_address).get('document', {})
> +if self.cloud_name == CloudNames.AWS:
> +identity = ec2.get_instance_identity(
> +api_version, self.metadata_address)
> +crawled_metadata['dynamic'] = {'instance-identity': identity}
> except Exception:
> util.logexc(
> LOG, "Failed reading from metadata address %s",
> self.metadata_address)
> -return False
> -return True
> +return {}
> +crawled_metadata['_metadata_api_version'] = api_version
> +return crawled_metadata
>
>
> class DataSourceEc2Local(DataSourceEc2):
> diff --git a/doc/rtd/topics/instancedata.rst b/doc/rtd/topics/instancedata.rst
> index 634e180..9c30bc1 100644
> --- a/doc/rtd/topics/instancedata.rst
> +++ b/doc/rtd/topics/instancedata.rst
> @@ -90,24 +90,39 @@ There are three basic top-level keys:
>
> The standardized keys present:
>
> -+--+---+---+
> -| Key path| Description |
> Examples |
> -+==+===+===+
> -| v1.cloud_name| The name of the cloud provided by metadata|
> aws, openstack, azure,|
> -| | key 'cloud-name' or the cloud-init datasource |
> configdrive, nocloud, |
> -| | name which was discovered.|
> ovf, etc. |
> -+--+---+---+
> -| v1.instance_id | Unique instance_id allocated by the cloud |
> i- |
> -+--+---+---+
> -| v1.local_hostname| The internal or local hostname of the system |
> ip-10-41-41-70, |
> -| | |
> |
> -+--+---+---+
> -| v1.region| The physical region/datacenter in which the |
> us-east-2 |
> -| |
[Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
The proposal to merge ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master has been updated. Commit message changed to: instance-data: Add standardized keys platform and subplatform. Refactor ec2. Add the following instance-data.json standardized keys: * v1._beta_keys: List any v1 keys in beta development, e.g. ['subplatform']. * v1.public_ssh_keys: List of any cloud-provided ssh keys for the instance. * v1.platform: String representing the cloud platform api supporting the datasource. For example: 'ec2' for aws, aliyun and brightbox cloud names. * v1.subplatform: String with more details about the source of the metadata consumed. For example, metadata uri, config drive device path or seed directory. To support the new platform and subplatform standardized instance-data, DataSource and its subclasses grew platform and subplatform attributes. The platform attribute defaults to the lowercase string datasource name at self.dsname. This method is overridden in NoCloud, Ec2 and ConfigDrive datasources. The subplatform attribute calls a _get_subplatform method which will return a string containing a simple slug for subplatform type such as metadata, seed-dir or config-drive followed by a detailed uri, device or directory path where the datasource consumed its configuration. As part of this work, DatasourceEC2 methods _get_data and _crawl_metadata have been refactored for a few reasons: - crawl_metadata is now a read-only operation, persisting no attributes on the datasource instance and returns a dictionary of consumed metadata. - crawl_metadata now closely represents the raw stucture of the ec2 metadata consumed, so that end-users can leverage public ec2 metadata documentation where possible. - crawl_metadata adds a '_metadata_api_version' key to the crawled ds.metadata to advertise what version of EC2's api was consumed by cloud-init. - _get_data now does all the processing of crawl_metadata and saves datasource instance attributes userdata_raw, metadata etc. Additional drive-bys: * unit test rework for test_altcloud and test_azure to simplify mocks and make use of existing util and test_helpers functions. For more details, see: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 -- Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
Review: Approve continuous-integration PASSED: Continuous integration, rev:94e3f120ce8f85a21c9ef7e256cfac440dc7578d https://jenkins.ubuntu.com/server/job/cloud-init-ci/376/ Executed test runs: SUCCESS: Checkout SUCCESS: Unit & Style Tests SUCCESS: Ubuntu LTS: Build SUCCESS: Ubuntu LTS: Integration IN_PROGRESS: Declarative: Post Actions Click here to trigger a rebuild: https://jenkins.ubuntu.com/server/job/cloud-init-ci/376/rebuild -- https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
Review: Approve continuous-integration PASSED: Continuous integration, rev:02a81e66ccecebb4a624a6ac6346968b77831124 https://jenkins.ubuntu.com/server/job/cloud-init-ci/375/ Executed test runs: SUCCESS: Checkout SUCCESS: Unit & Style Tests SUCCESS: Ubuntu LTS: Build SUCCESS: Ubuntu LTS: Integration IN_PROGRESS: Declarative: Post Actions Click here to trigger a rebuild: https://jenkins.ubuntu.com/server/job/cloud-init-ci/375/rebuild -- https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
[Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
The proposal to merge ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master has been updated. Commit message changed to: instance-data: Add standardized keys platform and subplatform. Refactor ec2. Add the following instance-data.json standardized keys: * v1._beta_keys: List any v1 keys in beta development, e.g. ['subplatform']. * v1.public_ssh_keys: List of any cloud-provided ssh keys for the instance. * v1.platform: String representing the cloud platform api supporting the datasource. For example: 'ec2' for aws, aliyun and brightbox cloud names. * v1.subplatform: String with more details about the source of the metadata consumed. For example, metadata uri, config drive device path or seed directory. To support the new platform and subplatform standardized instance-data, DataSource and its subclasses grew platform and subplatform attributes. The platform attribute defaults to the lowercase string datasource name at self.dsname. This method is overridden in NoCloud, Ec2 and ConfigDrive datasources. The subplatform attribute calls a _get_subplatform method which will return a string containing a simple slug for subplatform type such as metadata, seed-dir or config-drive followed my a detailed uri, device or directory path where the datasource consumed its configuration. As part of this work, DatasourceEC2 methods _get_data and _crawl_metadata have been refactored for a few reasons: - crawl_metadata is now a read-only operation, persisting no attributes on the datasource instance and returns a dictionary of consumed metadata. - crawl_metadata now closely represents the raw stucture of the ec2 metadata consumed, so that end-users can leverage public ec2 metadata documentation where possible. - crawl_metadata adds a '_metadata_api_version' key to the crawled ds.metadata to advertise what version of EC2's api was consumed by cloud-init. - _get_data now does all the processing of crawl_metadata and saves datasource instance attributes userdata_raw, metadata etc. Additional drive-bys: * unit test rework for test_altcloud and test_azure to simplify mocks and make use of existing util and test_helpers functions. For more details, see: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 -- Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
Review: Needs Fixing continuous-integration FAILED: Continuous integration, rev:27b46c18c1384ce8b238c34cde0824340bc8fb40 https://jenkins.ubuntu.com/server/job/cloud-init-ci/374/ Executed test runs: SUCCESS: Checkout FAILED: Unit & Style Tests Click here to trigger a rebuild: https://jenkins.ubuntu.com/server/job/cloud-init-ci/374/rebuild -- https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
[Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
The proposal to merge ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master has been updated. Commit message changed to: instance-data: Add standardized keys platform and subplatform. Refactor ec2. Add the following instance-data.json standardized keys: * v1._beta_keys: List any v1 keys in beta development, e.g. ['subplatform']. * v1.public_ssh_keys: List of any cloud-provided ssh keys for the instance. * v1.platform: String representing the cloud platform api supporting the datasource. For example: 'ec2' for aws, aliyun and brightbox cloud names. * v1.subplatform: String with more details about the source of the metadata consumed. For example, metadata uri, config drive device path or seed directory. To support the new platform and subplatform standardized instance-data, DataSource and its subclasses grew platform and subplatform attributes. The platform attribute defaults to the lowercase string datasource name at self.dsname. This method is overridden in NoCloud, Ec2 and ConfigDrive datasources. The subplatform attribute calls a _get_subplatform method which will return a string containing a simple slug for subplatform type such as metadata, seed-dir or config-drive followed my a detailed uri, device or directory path where the datasource consumed its configuration. As part of this work, DatasourceEC2 methods _get_data and _crawl_metadata have been refactored for a few reasons: - crawl_metadata is now a read-only operation, persisting no attributes on the datasource instance and returns a dictionary of consumed metadata. - crawl_metadata now closely represents the raw stucture of the ec2 metadata consumed, so that end-users can leverage public ec2 metadata documentation where possible. - crawl_metadata adds a '_metadata_api_version' key to the crawled ds.metadata to advertise what version of EC2's api was consumed by cloud-init. - _get_data now does all the processing of crawl_metadata and saves datasource instance attributes userdata_raw, metadata etc. For more details, see: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 -- Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
[Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
The proposal to merge ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master has been updated. Commit message changed to: instance-data: Add standardized keys platform and subplatform. Refactor ec2. Add the following instance-data.json standardized keys: * v1._beta_keys: List any v1 keys in beta development, e.g. ['subplatform']. * v1.public_ssh_keys: List of any cloud-provided ssh keys for the instance. * v1.platform: String representing the cloud platform api supporting the datasource. For example: 'ec2' for aws, aliyun and brightbox cloud names. * v1.subplatform: String with more details about the source of the metadata consumed. For example, metadata uri, config drive device path or seed directory. To support the new platform and subplatform standardized instance-data, DataSource and its subclasses grew platform and subplatform attributes. The platform attribute defaults to the lowercase string datasource name at self.dsname. This method is overridden in NoCloud, Ec2 and ConfigDrive datasources. The subplatform attribute calls a _get_subplatform method which will return a string containing a simple slug for subplatform type such as metadata, seed-dir or config-drive followed my a detailed uri, device or directory path where the datasource consumed its configuration. As part of this work, DatasourceEC2 methods _get_data and _crawl_metadata have been refactored for a few reasons: - crawl_metadata is now a read-only operation, persisting no attributes on the datasource instance and returns a dictionary of consumed metadata. - crawl_metadata more closely represents the raw stucture of the ec2 metadata consumed, so that end-users can leverage public ec2 metadata documentation where possible. - crawl_metadata adds an _apt_metadata_version key to the crawled ds.metadata to advertise what version of EC2's api was consumed by cloud-init. - _get_data now does all the processing of crawl_metadata and saves datasource instance attributes userdata_raw, metadata etc. For more details, see: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 -- Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
[Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
The proposal to merge ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master has been updated. Commit message changed to: instance-data: Add standardized keys platform and subplatform. Refactor ec2. Add the following instance-data.json standardized keys: * v1._beta_keys: List any v1 keys in beta development, e.g. ['subplatform']. * v1.public_ssh_keys: List of any cloud-provided ssh keys for the instance. * v1.platform: String representing the cloud platform api supporting the datasource. For example: 'ec2' for aws, aliyun and brightbox cloud names. * v1.subplatform: String with more details about the source of the metadata consumed. For example, metadata uri, config drive device path or seed directory. To support the new platform and subplatform standardized instance-data, DataSource and its subclasses grew platform and subplatform attributes. The platform attribute defaults to the lowercase string datasource name at self.dsname. This method is overridden in NoCloud, Ec2 and ConfigDrive datasources. The subplatform attribute calls a _get_subplaform method which will return a string containing a simple slug for subplatform type such as metadata, seed-dir or config-drive followed my a detailed uri, device or directory path where the datasource consumed its configuration. As part of this work, DatasourceEC2 methods _get_data and _crawl_metadata have been refactored for a few reasons: - crawl_metadata is now a read-only operation, persisting no attributes on the datasource instance and returns a dictionary of consumed metadata. - crawl_metadata more closely represents the raw stucture of the ec2 metadata consumed, so that end-users can leverage public ec2 metadata documentation where possible. - crawl_metadata adds an _apt_metadata_version key to the crawled ds.metadata to advertise what version of EC2's api was consumed by cloud-init. - _get_data now does all the processing of crawl_metadata and saves datasource instance attributes userdata_raw, metadata etc. For more details, see: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 -- Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
[Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
The proposal to merge ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master has been updated. Commit message changed to: ec2: update crawled metadata. Add standardized keys * refactor EC2 _crawl_metadata to return a dictionary * refactor EC2._get_data to process crawled_data and cache ds._crawled_metadata * add v1 standardized instance-data keys: _beta_keys, ssh_public_keys, platform and subplatform * drop ec2.cloud_platform method from aliyun and ec2 datasources - instead implement _get_cloud_name method For more details, see: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 -- Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
Review: Needs Fixing continuous-integration FAILED: Continuous integration, rev:730bcf8f6033155383a8e732f26c48811c29b499 https://jenkins.ubuntu.com/server/job/cloud-init-ci/371/ Executed test runs: SUCCESS: Checkout SUCCESS: Unit & Style Tests SUCCESS: Ubuntu LTS: Build FAILED: Ubuntu LTS: Integration Click here to trigger a rebuild: https://jenkins.ubuntu.com/server/job/cloud-init-ci/371/rebuild -- https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
document beta_keys somwhere ?
Diff comments:
> diff --git a/cloudinit/sources/DataSourceAliYun.py
> b/cloudinit/sources/DataSourceAliYun.py
> index 858e082..16abd09 100644
> --- a/cloudinit/sources/DataSourceAliYun.py
> +++ b/cloudinit/sources/DataSourceAliYun.py
> @@ -18,25 +18,17 @@ class DataSourceAliYun(EC2.DataSourceEc2):
> min_metadata_version = '2016-01-01'
> extended_metadata_versions = []
>
> -def __init__(self, sys_cfg, distro, paths):
> -super(DataSourceAliYun, self).__init__(sys_cfg, distro, paths)
> -self.seed_dir = os.path.join(paths.seed_dir, "AliYun")
> -
> def get_hostname(self, fqdn=False, resolve_ip=False,
> metadata_only=False):
> return self.metadata.get('hostname', 'localhost.localdomain')
>
> def get_public_ssh_keys(self):
> return parse_public_keys(self.metadata.get('public-keys', {}))
>
> -@property
> -def cloud_platform(self):
> -if self._cloud_platform is None:
> -if _is_aliyun():
> -self._cloud_platform = EC2.Platforms.ALIYUN
> -else:
> -self._cloud_platform = EC2.Platforms.NO_EC2_METADATA
> -
> -return self._cloud_platform
> +def _get_cloud_name(self):
> +if _is_aliyun():
you dropped the caching of _cloud_platform. was that intended ?
> +return EC2.CloudNames.ALIYUN
> +else:
> +return EC2.CloudNames.NO_EC2_METADATA
>
>
> def _is_aliyun():
> diff --git a/cloudinit/sources/DataSourceAltCloud.py
> b/cloudinit/sources/DataSourceAltCloud.py
> index 8cd312d..5270fda 100644
> --- a/cloudinit/sources/DataSourceAltCloud.py
> +++ b/cloudinit/sources/DataSourceAltCloud.py
> @@ -99,7 +101,14 @@ class DataSourceAltCloud(sources.DataSource):
> 'RHEV', 'VSPHERE' or 'UNKNOWN'
>
> '''
> -
> +if os.path.exists(CLOUD_INFO_FILE):
these changes are hard... as can't test this anywhere.
I think at one poitn ovirt was using this, but I think it might be completely
gone from current use (ie, we could/should drop).
not sure how to figure that out.
> +try:
> +cloud_type = util.load_file(CLOUD_INFO_FILE).strip().upper()
> +except IOError:
> +util.logexc(LOG, 'Unable to access cloud info file at %s.',
> +CLOUD_INFO_FILE)
> +return 'UNKNOWN'
> +return cloud_type
> system_name = util.read_dmi_data("system-product-name")
> if not system_name:
> return 'UNKNOWN'
> diff --git a/cloudinit/sources/DataSourceAzure.py
> b/cloudinit/sources/DataSourceAzure.py
> index 783445e..39391d0 100644
> --- a/cloudinit/sources/DataSourceAzure.py
> +++ b/cloudinit/sources/DataSourceAzure.py
> @@ -351,6 +351,14 @@ class DataSourceAzure(sources.DataSource):
> metadata['public-keys'] = key_value or
> pubkeys_from_crt_files(fp_files)
> return metadata
>
> +def _get_subplatform(self):
> +"""Return the subplatform metadata source details."""
> +if self.seed.startswith('/dev'):
> +subplatform_type = 'config-disk'
> +else:
> +subplatform_type = 'seed-dir'
in reality this willi always say 'config-disk' now.
right?
> +return '%s (%s)' % (subplatform_type, self.seed)
> +
> def crawl_metadata(self):
> """Walk all instance metadata sources returning a dict on success.
>
> diff --git a/cloudinit/sources/DataSourceEc2.py
> b/cloudinit/sources/DataSourceEc2.py
> index 968ab3f..618714b 100644
> --- a/cloudinit/sources/DataSourceEc2.py
> +++ b/cloudinit/sources/DataSourceEc2.py
> @@ -28,18 +28,16 @@ STRICT_ID_PATH = ("datasource", "Ec2", "strict_id")
> STRICT_ID_DEFAULT = "warn"
>
>
> -class Platforms(object):
> -# TODO Rename and move to cloudinit.cloud.CloudNames
> -ALIYUN = "AliYun"
> -AWS = "AWS"
> -BRIGHTBOX = "Brightbox"
> -SEEDED = "Seeded"
I'm wondering about 'aliyun' and 'brightbox' as platforms (and as cloudnames).
they're *not* ec2 api clones (just the metadata service portion). at least
brightbox is not.
dont worry too much about this.
> +class CloudNames(object):
> +ALIYUN = "aliyun"
> +AWS = "aws"
> +BRIGHTBOX = "brightbox"
> # UNKNOWN indicates no positive id. If strict_id is 'warn' or 'false',
> # then an attempt at the Ec2 Metadata service will be made.
> -UNKNOWN = "Unknown"
> +UNKNOWN = "unknown"
> # NO_EC2_METADATA indicates this platform does not have a Ec2 metadata
> # service available. No attempt at the Ec2 Metadata service will be made.
> -NO_EC2_METADATA = "No-EC2-Metadata"
> +NO_EC2_METADATA = "no-ec2-metadata"
>
>
> class DataSourceEc2(sources.DataSource):
> diff --git a/cloudinit/sources/DataSourceOVF.py
> b/cloudinit/sources/DataSourceOVF.py
> index 178ccb0..045291e 100644
> --- a/cloudinit/sources/DataSourceOVF.py
> +++ b/cloudinit/sources/DataSourceOVF.py
>
[Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
The proposal to merge ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master has been updated. Description changed to: Some sample instance-data.json on lxd EC2 and OpenStack - https://pastebin.ubuntu.com/p/52BvK7Yr3V/ For more details, see: https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 -- Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
Review: Needs Fixing continuous-integration FAILED: Continuous integration, rev:58cdbe97d685c9d39f1cdb2d5e115cb616fcb95d https://jenkins.ubuntu.com/server/job/cloud-init-ci/370/ Executed test runs: FAILED: Checkout Click here to trigger a rebuild: https://jenkins.ubuntu.com/server/job/cloud-init-ci/370/rebuild -- https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
Review: Needs Fixing continuous-integration FAILED: Continuous integration, rev:ab2fe13ae1aa40edaf4d943dc826ad3063f74cd0 https://jenkins.ubuntu.com/server/job/cloud-init-ci/369/ Executed test runs: FAILED: Checkout Click here to trigger a rebuild: https://jenkins.ubuntu.com/server/job/cloud-init-ci/369/rebuild -- https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
Review: Needs Fixing continuous-integration FAILED: Continuous integration, rev:f5914600a4e501acbe326597058e8de59af1193c https://jenkins.ubuntu.com/server/job/cloud-init-ci/368/ Executed test runs: SUCCESS: Checkout FAILED: Unit & Style Tests Click here to trigger a rebuild: https://jenkins.ubuntu.com/server/job/cloud-init-ci/368/rebuild -- https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
Review: Needs Fixing continuous-integration FAILED: Continuous integration, rev:1dc0c9f7757e8c682308ef93e20054931f83ecf7 https://jenkins.ubuntu.com/server/job/cloud-init-ci/357/ Executed test runs: FAILED: Checkout Click here to trigger a rebuild: https://jenkins.ubuntu.com/server/job/cloud-init-ci/357/rebuild -- https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999 Your team cloud-init commiters is requested to review the proposed merge of ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master. ___ Mailing list: https://launchpad.net/~cloud-init-dev Post to : cloud-init-dev@lists.launchpad.net Unsubscribe : https://launchpad.net/~cloud-init-dev More help : https://help.launchpad.net/ListHelp
[Cloud-init-dev] [Merge] ~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master
Chad Smith has proposed merging
~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master.
Commit message:
ec2: update crawled metadata. Add standardized keys
* refactor _crawl_metadata to return a dictionary
* refactor _get_data to process crawled_data and cache ds._crawled_metadata
* add v1.ssh_public_keys and v1.platform_time standardized instance data keys
* drop ec2.cloud_platform method from aliyun and ec2 datasources
- instead implement _get_cloud_name method
Requested reviews:
cloud-init commiters (cloud-init-dev)
For more details, see:
https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/355999
--
Your team cloud-init commiters is requested to review the proposed merge of
~chad.smith/cloud-init:cleanup/metadata-cloud-platform into cloud-init:master.
diff --git a/cloudinit/sources/DataSourceAliYun.py b/cloudinit/sources/DataSourceAliYun.py
index 858e082..38ec44a 100644
--- a/cloudinit/sources/DataSourceAliYun.py
+++ b/cloudinit/sources/DataSourceAliYun.py
@@ -12,6 +12,7 @@ ALIYUN_PRODUCT = "Alibaba Cloud ECS"
class DataSourceAliYun(EC2.DataSourceEc2):
dsname = 'AliYun'
+_platform_type = 'ec2'
metadata_urls = ['http://100.100.100.200']
# The minimum supported metadata_version from the ec2 metadata apis
@@ -28,15 +29,11 @@ class DataSourceAliYun(EC2.DataSourceEc2):
def get_public_ssh_keys(self):
return parse_public_keys(self.metadata.get('public-keys', {}))
-@property
-def cloud_platform(self):
-if self._cloud_platform is None:
-if _is_aliyun():
-self._cloud_platform = EC2.Platforms.ALIYUN
-else:
-self._cloud_platform = EC2.Platforms.NO_EC2_METADATA
-
-return self._cloud_platform
+def _get_cloud_name(self):
+if _is_aliyun():
+return EC2.CloudNames.ALIYUN
+else:
+return EC2.CloudNames.NO_EC2_METADATA
def _is_aliyun():
diff --git a/cloudinit/sources/DataSourceEc2.py b/cloudinit/sources/DataSourceEc2.py
index 968ab3f..482eb21 100644
--- a/cloudinit/sources/DataSourceEc2.py
+++ b/cloudinit/sources/DataSourceEc2.py
@@ -28,18 +28,17 @@ STRICT_ID_PATH = ("datasource", "Ec2", "strict_id")
STRICT_ID_DEFAULT = "warn"
-class Platforms(object):
-# TODO Rename and move to cloudinit.cloud.CloudNames
-ALIYUN = "AliYun"
-AWS = "AWS"
-BRIGHTBOX = "Brightbox"
-SEEDED = "Seeded"
+class CloudNames(object):
+ALIYUN = "aliyun"
+AWS = "aws"
+BRIGHTBOX = "brightbox"
+SEEDED = "ec2-Seeded"
# UNKNOWN indicates no positive id. If strict_id is 'warn' or 'false',
# then an attempt at the Ec2 Metadata service will be made.
-UNKNOWN = "Unknown"
+UNKNOWN = "unknown"
# NO_EC2_METADATA indicates this platform does not have a Ec2 metadata
# service available. No attempt at the Ec2 Metadata service will be made.
-NO_EC2_METADATA = "No-EC2-Metadata"
+NO_EC2_METADATA = "no-ec2-metadata"
class DataSourceEc2(sources.DataSource):
@@ -61,8 +60,6 @@ class DataSourceEc2(sources.DataSource):
url_max_wait = 120
url_timeout = 50
-_cloud_platform = None
-
_network_config = sources.UNSET # Used to cache calculated network cfg v1
# Whether we want to get network configuration from the metadata service.
@@ -75,7 +72,7 @@ class DataSourceEc2(sources.DataSource):
def _get_cloud_name(self):
"""Return the cloud name as identified during _get_data."""
-return self.cloud_platform
+return identify_platform()
def _get_data(self):
seed_ret = {}
@@ -83,7 +80,7 @@ class DataSourceEc2(sources.DataSource):
self.userdata_raw = seed_ret['user-data']
self.metadata = seed_ret['meta-data']
LOG.debug("Using seeded ec2 data from %s", self.seed_dir)
-self._cloud_platform = Platforms.SEEDED
+self._cloud_name = CloudNames.SEEDED
return True
strict_mode, _sleep = read_strict_mode(
@@ -91,10 +88,10 @@ class DataSourceEc2(sources.DataSource):
STRICT_ID_DEFAULT), ("warn", None))
LOG.debug("strict_mode: %s, cloud_platform=%s",
- strict_mode, self.cloud_platform)
-if strict_mode == "true" and self.cloud_platform == Platforms.UNKNOWN:
+ strict_mode, self.cloud_name)
+if strict_mode == "true" and self.cloud_name == CloudNames.UNKNOWN:
return False
-elif self.cloud_platform == Platforms.NO_EC2_METADATA:
+elif self.cloud_name == CloudNames.NO_EC2_METADATA:
return False
if self.perform_dhcp_setup: # Setup networking in init-local stage.
@@ -103,13 +100,20 @@ class DataSourceEc2(sources.DataSource):
return False
try:
with EphemeralDHCPv4(self.fallback_interface):
-return util.log_time(
+
