[Cloud-init-dev] [Merge] ~louis/cloud-init:enable_Scaleway_network_config into cloud-init:master

2018-06-14 Thread Louis Bouchard
Louis Bouchard has proposed merging 
~louis/cloud-init:enable_Scaleway_network_config into cloud-init:master.

Commit message:
Scaleway: Add network configuration to the DataSource

DEP_NETWORK is removed since the network_config must
run at each boot.

Network is brought up early to fetch the metadata which
is required to configure the network (ipv4 and/or v6).

Adds unittests for the following and fixes test_common for
LOCAL and NETWORK sets.

Requested reviews:
  cloud-init commiters (cloud-init-dev)

For more details, see:
https://code.launchpad.net/~louis/cloud-init/+git/cloud-init/+merge/347973
-- 
Your team cloud-init commiters is requested to review the proposed merge of 
~louis/cloud-init:enable_Scaleway_network_config into cloud-init:master.
diff --git a/cloudinit/sources/DataSourceScaleway.py b/cloudinit/sources/DataSourceScaleway.py
index e2502b0..79fef47 100644
--- a/cloudinit/sources/DataSourceScaleway.py
+++ b/cloudinit/sources/DataSourceScaleway.py
@@ -29,7 +29,8 @@ from cloudinit import log as logging
 from cloudinit import sources
 from cloudinit import url_helper
 from cloudinit import util
-
+from cloudinit import net
+from cloudinit.net.dhcp import EphemeralDHCPv4, NoDHCPLeaseError
 
 LOG = logging.getLogger(__name__)
 
@@ -168,7 +169,6 @@ def query_data_api(api_type, api_address, retries, timeout):
 
 
 class DataSourceScaleway(sources.DataSource):
-
 dsname = "Scaleway"
 
 def __init__(self, sys_cfg, distro, paths):
@@ -185,11 +185,10 @@ class DataSourceScaleway(sources.DataSource):
 
 self.retries = int(self.ds_cfg.get('retries', DEF_MD_RETRIES))
 self.timeout = int(self.ds_cfg.get('timeout', DEF_MD_TIMEOUT))
+self._fallback_interface = None
+self._network_config = None
 
-def _get_data(self):
-if not on_scaleway():
-return False
-
+def _crawl_metadata(self):
 resp = url_helper.readurl(self.metadata_address,
   timeout=self.timeout,
   retries=self.retries)
@@ -203,8 +202,45 @@ class DataSourceScaleway(sources.DataSource):
 'vendor-data', self.vendordata_address,
 self.retries, self.timeout
 )
+
+def _get_data(self):
+if not on_scaleway():
+return False
+
+if self._fallback_interface is None:
+self._fallback_interface = net.find_fallback_nic()
+try:
+with EphemeralDHCPv4(self._fallback_interface):
+results = util.log_time(
+logfunc=LOG.debug, msg='Crawl of metadata service',
+func=self._crawl_metadata)
+except (NoDHCPLeaseError) as e:
+util.logexc(LOG, str(e))
+return False
 return True
 
+def network_config(self):
+"""
+Configure networking according to data received from the
+metadata API.
+"""
+if self._network_config:
+return self._network_config
+
+if self._fallback_interface is None:
+self._fallback_interface = net.find_fallback_nic()
+
+netcfg = {'type': 'physical', 'name': '%s' % self._fallback_interface}
+subnets = [{'type': 'dhcp4'}]
+if self.metadata['ipv6']:
+subnets += [{'type': 'static',
+ 'address': '%s' % self.metadata['ipv6']['address'],
+ 'gateway': '%s' % self.metadata['ipv6']['gateway'],
+ 'netmask': '%s' % self.metadata['ipv6']['netmask'],
+ }]
+netcfg['subnets'] = subnets
+return {'version': 1, 'config': [netcfg]}
+
 @property
 def launch_index(self):
 return None
@@ -228,7 +264,7 @@ class DataSourceScaleway(sources.DataSource):
 
 
 datasources = [
-(DataSourceScaleway, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
+(DataSourceScaleway, (sources.DEP_FILESYSTEM,)),
 ]
 
 
diff --git a/tests/unittests/test_datasource/test_common.py b/tests/unittests/test_datasource/test_common.py
index 0d35dc2..1a5a3db 100644
--- a/tests/unittests/test_datasource/test_common.py
+++ b/tests/unittests/test_datasource/test_common.py
@@ -41,6 +41,7 @@ DEFAULT_LOCAL = [
 SmartOS.DataSourceSmartOS,
 Ec2.DataSourceEc2Local,
 OpenStack.DataSourceOpenStackLocal,
+Scaleway.DataSourceScaleway,
 ]
 
 DEFAULT_NETWORK = [
@@ -55,7 +56,6 @@ DEFAULT_NETWORK = [
 NoCloud.DataSourceNoCloudNet,
 OpenStack.DataSourceOpenStack,
 OVF.DataSourceOVFNet,
-Scaleway.DataSourceScaleway,
 ]
 
 
diff --git a/tests/unittests/test_datasource/test_scaleway.py b/tests/unittests/test_datasource/test_scaleway.py
index e4e9bb2..59e820b 100644
--- a/tests/unittests/test_datasource/test_scaleway.py
+++ b/tests/unittests/test_datasource/test_scaleway.py
@@ -176,11 +176,12 @@ class TestDataSourceScaleway(HttprettyTestCase):
 self.vendordata_url = \
 DataSourceScaleway.B

[Cloud-init-dev] [Merge] ~louis/cloud-init:enable_Scaleway_network_config into cloud-init:master

2018-06-14 Thread Scott Moser
The proposal to merge ~louis/cloud-init:enable_Scaleway_network_config into 
cloud-init:master has been updated.

Commit message changed to:

Scaleway: Add network configuration to the DataSource

DEP_NETWORK is removed since the network_config must
run at each boot.

Network is brought up early to fetch the metadata which
is required to configure the network (ipv4 and/or v6).

Adds unittests for the following and fixes test_common for
LOCAL and NETWORK sets.

For more details, see:
https://code.launchpad.net/~louis/cloud-init/+git/cloud-init/+merge/347973
-- 
Your team cloud-init commiters is requested to review the proposed merge of 
~louis/cloud-init:enable_Scaleway_network_config into cloud-init:master.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp


Re: [Cloud-init-dev] [Merge] ~louis/cloud-init:enable_Scaleway_network_config into cloud-init:master

2018-06-14 Thread Server Team CI bot
Review: Needs Fixing continuous-integration

FAILED: Continuous integration, rev:ff0c6a2a4805ee8abe5e753886aa74218aedcec0
https://jenkins.ubuntu.com/server/job/cloud-init-ci/81/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/81/rebuild

-- 
https://code.launchpad.net/~louis/cloud-init/+git/cloud-init/+merge/347973
Your team cloud-init commiters is requested to review the proposed merge of 
~louis/cloud-init:enable_Scaleway_network_config into cloud-init:master.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp


Re: [Cloud-init-dev] [Merge] ~louis/cloud-init:enable_Scaleway_network_config into cloud-init:master

2018-06-14 Thread Louis Bouchard
Branch fixed & force pushed.
-- 
https://code.launchpad.net/~louis/cloud-init/+git/cloud-init/+merge/347973
Your team cloud-init commiters is requested to review the proposed merge of 
~louis/cloud-init:enable_Scaleway_network_config into cloud-init:master.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp


Re: [Cloud-init-dev] [Merge] ~louis/cloud-init:enable_Scaleway_network_config into cloud-init:master

2018-06-14 Thread Server Team CI bot
Review: Needs Fixing continuous-integration

FAILED: Continuous integration, rev:d99396dd52b69d3ee54beeaf97c99e23620b7a0c
https://jenkins.ubuntu.com/server/job/cloud-init-ci/82/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/82/rebuild

-- 
https://code.launchpad.net/~louis/cloud-init/+git/cloud-init/+merge/347973
Your team cloud-init commiters is requested to review the proposed merge of 
~louis/cloud-init:enable_Scaleway_network_config into cloud-init:master.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp


Re: [Cloud-init-dev] [Merge] ~louis/cloud-init:enable_Scaleway_network_config into cloud-init:master

2018-06-14 Thread Server Team CI bot
Review: Needs Fixing continuous-integration

FAILED: Continuous integration, rev:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/83/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/83/rebuild

-- 
https://code.launchpad.net/~louis/cloud-init/+git/cloud-init/+merge/347973
Your team cloud-init commiters is requested to review the proposed merge of 
~louis/cloud-init:enable_Scaleway_network_config into cloud-init:master.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp


[Cloud-init-dev] [Merge] ~chad.smith/cloud-init:feature/maintain-network-on-boot into cloud-init:master

2018-06-14 Thread Chad Smith
Chad Smith has proposed merging 
~chad.smith/cloud-init:feature/maintain-network-on-boot into cloud-init:master.

Commit message:
work-in-progress Strawman for MaintenanceEvent discussion.


Base-level MaintenanceEvent class and DataSource.maintain_metadata behavior to 
allow clearing cached instance data and re-crawling all metadata sources.



I'll build this out today and review any comments/suggestions folks want to 
attach while I'm developing it.

Requested reviews:
  cloud-init commiters (cloud-init-dev)

For more details, see:
https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/348000
-- 
Your team cloud-init commiters is requested to review the proposed merge of 
~chad.smith/cloud-init:feature/maintain-network-on-boot into cloud-init:master.
diff --git a/cloudinit/hotplug.py b/cloudinit/hotplug.py
new file mode 100644
index 000..c5ba1af
--- /dev/null
+++ b/cloudinit/hotplug.py
@@ -0,0 +1,15 @@
+# This file is part of cloud-init. See LICENSE file for license information.
+"""Classes and functions related to hotplug and eventing."""
+
+# Maintenance events describing the source generating a maintenance request.
+class MaintenanceEvent(object):
+NONE = 0x0   # React to no maintenance events
+BOOT = 0x1   # Any system boot or reboot event
+DEVICE_ADD = 0x2 # Any new device added
+DEVICE_REMOVE = 0x4  # Any device removed
+DEVICE_CHANGE = 0x8  # Any device metadata change
+ANY = 0xF# Match any defined MaintenanceEvents
+
+MAINTENANCE_EVENT_STR = dict(
+(attr, getattr(MaintenanceEvent, attr))
+for attr in MaintenanceEvent.__dict__.keys() if attr.isupper())
diff --git a/cloudinit/sources/DataSourceAltCloud.py b/cloudinit/sources/DataSourceAltCloud.py
index 24fd65f..cd6ab17 100644
--- a/cloudinit/sources/DataSourceAltCloud.py
+++ b/cloudinit/sources/DataSourceAltCloud.py
@@ -114,7 +114,7 @@ class DataSourceAltCloud(sources.DataSource):
 
 return 'UNKNOWN'
 
-def _get_data(self):
+def _get_data(self, clear_cache=False):
 '''
 Description:
 User Data is passed to the launching instance which
diff --git a/cloudinit/sources/DataSourceBigstep.py b/cloudinit/sources/DataSourceBigstep.py
index 699a85b..bc0d53a 100644
--- a/cloudinit/sources/DataSourceBigstep.py
+++ b/cloudinit/sources/DataSourceBigstep.py
@@ -25,7 +25,7 @@ class DataSourceBigstep(sources.DataSource):
 self.vendordata_raw = ""
 self.userdata_raw = ""
 
-def _get_data(self, apply_filter=False):
+def _get_data(self, clear_cache=False, apply_filter=False):
 url = get_url_from_file()
 if url is None:
 return False
diff --git a/cloudinit/sources/DataSourceCloudSigma.py b/cloudinit/sources/DataSourceCloudSigma.py
index c816f34..e67ff7c 100644
--- a/cloudinit/sources/DataSourceCloudSigma.py
+++ b/cloudinit/sources/DataSourceCloudSigma.py
@@ -49,7 +49,7 @@ class DataSourceCloudSigma(sources.DataSource):
 LOG.warning("failed to query dmi data for system product name")
 return False
 
-def _get_data(self):
+def _get_data(self, clear_cache=False):
 """
 Metadata is the whole server context and /meta/cloud-config is used
 as userdata.
diff --git a/cloudinit/sources/DataSourceCloudStack.py b/cloudinit/sources/DataSourceCloudStack.py
index d4b758f..3912bc8 100644
--- a/cloudinit/sources/DataSourceCloudStack.py
+++ b/cloudinit/sources/DataSourceCloudStack.py
@@ -109,7 +109,7 @@ class DataSourceCloudStack(sources.DataSource):
 def get_config_obj(self):
 return self.cfg
 
-def _get_data(self):
+def _get_data(self, clear_cache=False):
 seed_ret = {}
 if util.read_optional_seed(seed_ret, base=(self.seed_dir + "/")):
 self.userdata_raw = seed_ret['user-data']
diff --git a/cloudinit/sources/DataSourceConfigDrive.py b/cloudinit/sources/DataSourceConfigDrive.py
index 4cb2897..99ff30c 100644
--- a/cloudinit/sources/DataSourceConfigDrive.py
+++ b/cloudinit/sources/DataSourceConfigDrive.py
@@ -54,7 +54,7 @@ class DataSourceConfigDrive(openstack.SourceMixin, sources.DataSource):
 mstr += "[source=%s]" % (self.source)
 return mstr
 
-def _get_data(self):
+def _get_data(self, clear_cache=False):
 found = None
 md = {}
 results = {}
diff --git a/cloudinit/sources/DataSourceDigitalOcean.py b/cloudinit/sources/DataSourceDigitalOcean.py
index e0ef665..25e8c87 100644
--- a/cloudinit/sources/DataSourceDigitalOcean.py
+++ b/cloudinit/sources/DataSourceDigitalOcean.py
@@ -47,7 +47,7 @@ class DataSourceDigitalOcean(sources.DataSource):
 def _get_sysinfo(self):
 return do_helper.read_sysinfo()
 
-def _get_data(self):
+def _get_data(self, clear_cache=False):
 (is_do, droplet_id) = self._get_sysinfo()
 
 # only proceed if we know we are on DigitalOcean
diff --git a/cloudinit/sources/DataSourceEc2.py b/cloudinit/sources/DataSourc

Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:feature/maintain-network-on-boot into cloud-init:master

2018-06-14 Thread Server Team CI bot
Review: Needs Fixing continuous-integration

FAILED: Continuous integration, rev:ca3b0d7c1a6fc23bbea29458855c6c9550ea08c4
https://jenkins.ubuntu.com/server/job/cloud-init-ci/84/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/84/rebuild

-- 
https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/348000
Your team cloud-init commiters is requested to review the proposed merge of 
~chad.smith/cloud-init:feature/maintain-network-on-boot into cloud-init:master.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp


[Cloud-init-dev] [Merge] ~chad.smith/cloud-init:feature/maintain-network-on-boot into cloud-init:master

2018-06-14 Thread Scott Moser
The proposal to merge ~chad.smith/cloud-init:feature/maintain-network-on-boot 
into cloud-init:master has been updated.

Status: Needs review => Work in progress

For more details, see:
https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/348000
-- 
Your team cloud-init commiters is requested to review the proposed merge of 
~chad.smith/cloud-init:feature/maintain-network-on-boot into cloud-init:master.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp


Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:feature/maintain-network-on-boot into cloud-init:master

2018-06-14 Thread Scott Moser
so where would the logic of "apply if changed" be placed?


Diff comments:

> diff --git a/cloudinit/hotplug.py b/cloudinit/hotplug.py
> new file mode 100644
> index 000..c5ba1af
> --- /dev/null
> +++ b/cloudinit/hotplug.py
> @@ -0,0 +1,15 @@
> +# This file is part of cloud-init. See LICENSE file for license information.
> +"""Classes and functions related to hotplug and eventing."""
> +
> +# Maintenance events describing the source generating a maintenance request.
> +class MaintenanceEvent(object):
> +NONE = 0x0   # React to no maintenance events
> +BOOT = 0x1   # Any system boot or reboot event
> +DEVICE_ADD = 0x2 # Any new device added
> +DEVICE_REMOVE = 0x4  # Any device removed

i think these ADD/REMOVE are not relevant anymore.
right?

> +DEVICE_CHANGE = 0x8  # Any device metadata change
> +ANY = 0xF# Match any defined MaintenanceEvents
> +
> +MAINTENANCE_EVENT_STR = dict(
> +(attr, getattr(MaintenanceEvent, attr))
> +for attr in MaintenanceEvent.__dict__.keys() if attr.isupper())
> diff --git a/cloudinit/sources/DataSourceAltCloud.py 
> b/cloudinit/sources/DataSourceAltCloud.py
> index 24fd65f..cd6ab17 100644
> --- a/cloudinit/sources/DataSourceAltCloud.py
> +++ b/cloudinit/sources/DataSourceAltCloud.py
> @@ -114,7 +114,7 @@ class DataSourceAltCloud(sources.DataSource):
>  
>  return 'UNKNOWN'
>  
> -def _get_data(self):
> +def _get_data(self, clear_cache=False):

ultimately i think we want *some* method on a datasource that actually just 
gets the data and doesnt modify the state of the datasource.
then, that thing doesnt have a 'clear_cache' it just gets called.

when i first read this without looking, i was thinking that _get_data() was 
that thing.  but it appears some  of our datasources modify their state there.

>  '''
>  Description:
>  User Data is passed to the launching instance which
> diff --git a/cloudinit/sources/__init__.py b/cloudinit/sources/__init__.py
> index 90d7457..af876ad 100644
> --- a/cloudinit/sources/__init__.py
> +++ b/cloudinit/sources/__init__.py
> @@ -416,6 +440,32 @@ class DataSource(object):
>  def get_package_mirror_info(self):
>  return self.distro.get_package_mirror_info(data_source=self)
>  
> +def maintain_metadata(self, maintenance_event):
> +"""Refresh cached metadata if the datasource handles this event.
> +
> +The datasource defines a network_maintenance_mask attribute which
> +authorizes refreshing all cached metadata due to any number of
> +supported MaintenenanceEvent types.
> +
> +@param maintenance_event: The source MaintenanceEvent type
> +observed to which the datasource may react.
> +
> +@return True if the datasource has updated cached metadata due to the
> +   the provided maintenance_event type. MaintenanceEvents will be
> +   something like boot, configchange, device_add, device_remove etc.
> +"""
> +if bool(maintenance_event & self.network_maintenance_mask):
> +LOG.debug(
> +"Re-crawling datasource metadata due to maintenance event: 
> '%s'",
> +MAINTENANCE_EVENT_STR.get(maintenance_event, 
> maintenance_event))

we probably should never hit the default value here... our maintenance event 
should always be in MAINTENANCE_EVENT_STR.

> +result = self.get_data(clear_cache=True)
> +if result:
> +return True
> +else:
> +LOG.warning(
> +'Re-crawling metadata reported invalid datasource type')
> +return False
> +
>  def check_instance_id(self, sys_cfg):
>  # quickly (local check only) if self.instance_id is still
>  return False


-- 
https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/348000
Your team cloud-init commiters is requested to review the proposed merge of 
~chad.smith/cloud-init:feature/maintain-network-on-boot into cloud-init:master.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp


Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:feature/maintain-network-on-boot into cloud-init:master

2018-06-14 Thread Ryan Harper
I'm not super happy with the Maintenance name.  Maybe SystemEvent?  Also, I 
think we need other classes of Event types;  DatasourceEvents 
(METADATA_REFRESHED) or PlatformEvents;  Maybe we'll be notified of a pending 
instance migration, pending shutdown, etc.  

Diff comments:

> diff --git a/cloudinit/hotplug.py b/cloudinit/hotplug.py
> new file mode 100644
> index 000..c5ba1af
> --- /dev/null
> +++ b/cloudinit/hotplug.py
> @@ -0,0 +1,15 @@
> +# This file is part of cloud-init. See LICENSE file for license information.
> +"""Classes and functions related to hotplug and eventing."""
> +
> +# Maintenance events describing the source generating a maintenance request.
> +class MaintenanceEvent(object):
> +NONE = 0x0   # React to no maintenance events
> +BOOT = 0x1   # Any system boot or reboot event
> +DEVICE_ADD = 0x2 # Any new device added
> +DEVICE_REMOVE = 0x4  # Any device removed
> +DEVICE_CHANGE = 0x8  # Any device metadata change

I think we should expand the list of events to be very granular, and we can 
combine some flags into common groups.
For example, we may want to react to REBOOT differently than BOOT.
I think the comment 'metadata' is unclear;  Do you mean any changes to the 
device itself or changes to metadata about a particular device?

> +ANY = 0xF# Match any defined MaintenanceEvents
> +
> +MAINTENANCE_EVENT_STR = dict(
> +(attr, getattr(MaintenanceEvent, attr))
> +for attr in MaintenanceEvent.__dict__.keys() if attr.isupper())
> diff --git a/cloudinit/sources/__init__.py b/cloudinit/sources/__init__.py
> index 90d7457..af876ad 100644
> --- a/cloudinit/sources/__init__.py
> +++ b/cloudinit/sources/__init__.py
> @@ -19,6 +19,7 @@ from cloudinit.atomic_helper import write_json
>  from cloudinit import importer
>  from cloudinit import log as logging
>  from cloudinit import net
> +from cloudinit.hotplug import MaintenanceEvent

cloudinit.events ?

>  from cloudinit import type_utils
>  from cloudinit import user_data as ud
>  from cloudinit import util
> @@ -102,6 +103,13 @@ class DataSource(object):
>  url_timeout = 10# timeout for each metadata url read attempt
>  url_retries = 5 # number of times to retry url upon 404
>  
> +# Subclasses can define a mask of supported MaintenanceEvents during
> +# which the datasource will regenerate network_configuration. For 
> example:
> +# network_maintenance_mask = MEvent.BOOT|MEvent.DEVICE_ADD
> +
> +# Default behavior, perform no network update for any maintenance event
> +network_maintenance_mask = MaintenanceEvent.NONE

I think the default is .BOOT right?  If not how do we reconcile the current 
default behavior of per-instance;  maybe we need a NEW_INSTANCE event?

> +
>  def __init__(self, sys_cfg, distro, paths, ud_proc=None):
>  self.sys_cfg = sys_cfg
>  self.distro = distro
> @@ -134,12 +142,25 @@ class DataSource(object):
>  'region': self.region,
>  'availability-zone': self.availability_zone}}
>  
> -def get_data(self):
> +def get_data(self, clear_cache=False):
>  """Datasources implement _get_data to setup metadata and 
> userdata_raw.
>  
>  Minimally, the datasource should return a boolean True on success.
> +@param use_cache: Boolean set true to re-use data cache if present.
> +   Value of False, will clear any cached data, re-crawling all
> +   instance metadata.
>  """
> -return_value = self._get_data()
> +if clear_cache:
> +if hasattr(self, '_network_config'):

I think we need to open this up to allow the Datasource subclasses to define a 
set of attrs that should be cleared; and we have a clear_cache() base method so 
the subclass can override implementation.

> +# Clear network config property so it is regenerated from md.
> +setattr(self, '_network_config', None)
> +self.userdata = None
> +self.metadata = {}
> +self.userdata_raw = None
> +self.vendordata = None
> +self.vendordata_raw = None
> +
> +return_value = self._get_data(clear_cache=clear_cache)
>  json_file = os.path.join(self.paths.run_dir, INSTANCE_JSON_FILE)
>  if not return_value:
>  return return_value
> @@ -416,6 +440,32 @@ class DataSource(object):
>  def get_package_mirror_info(self):
>  return self.distro.get_package_mirror_info(data_source=self)
>  
> +def maintain_metadata(self, maintenance_event):

refresh_metadata?

> +"""Refresh cached metadata if the datasource handles this event.
> +
> +The datasource defines a network_maintenance_mask attribute which
> +authorizes refreshing all cached metadata due to any number of
> +supported MaintenenanceEvent types.
> +
> +@param maintenance_event: The source MaintenanceEvent type
> +observed t

Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:feature/maintain-network-on-boot into cloud-init:master

2018-06-14 Thread Chad Smith



Diff comments:

> diff --git a/cloudinit/sources/DataSourceAltCloud.py 
> b/cloudinit/sources/DataSourceAltCloud.py
> index 24fd65f..cd6ab17 100644
> --- a/cloudinit/sources/DataSourceAltCloud.py
> +++ b/cloudinit/sources/DataSourceAltCloud.py
> @@ -114,7 +114,7 @@ class DataSourceAltCloud(sources.DataSource):
>  
>  return 'UNKNOWN'
>  
> -def _get_data(self):
> +def _get_data(self, clear_cache=False):

+1 on this suggestion, I've started to distill crawl_data from get_data in 
other Ec2 and OpenStack datasources.It's going to be a quite large branch to 
move all datasources to those distinct operations (read(crawl_metadata) vs 
persist(get_data)). So I don't want to make that move in this branch if we can 
avoid it.

>  '''
>  Description:
>  User Data is passed to the launching instance which
> diff --git a/cloudinit/sources/__init__.py b/cloudinit/sources/__init__.py
> index 90d7457..af876ad 100644
> --- a/cloudinit/sources/__init__.py
> +++ b/cloudinit/sources/__init__.py
> @@ -134,12 +142,25 @@ class DataSource(object):
>  'region': self.region,
>  'availability-zone': self.availability_zone}}
>  
> -def get_data(self):
> +def get_data(self, clear_cache=False):
>  """Datasources implement _get_data to setup metadata and 
> userdata_raw.
>  
>  Minimally, the datasource should return a boolean True on success.
> +@param use_cache: Boolean set true to re-use data cache if present.
> +   Value of False, will clear any cached data, re-crawling all
> +   instance metadata.
>  """
> -return_value = self._get_data()
> +if clear_cache:
> +if hasattr(self, '_network_config'):

Ryan I thought about this too, though those datasources might also have to 
provide default values expected on clear as well. Maybe the 
datasource.cached_attribute_defaults could be an n-tuple ('attr-name', 
)  and the DataSource.clear_cached_data would set values back to 
their datasource default.

> +# Clear network config property so it is regenerated from md.
> +setattr(self, '_network_config', None)
> +self.userdata = None
> +self.metadata = {}
> +self.userdata_raw = None
> +self.vendordata = None
> +self.vendordata_raw = None
> +
> +return_value = self._get_data(clear_cache=clear_cache)
>  json_file = os.path.join(self.paths.run_dir, INSTANCE_JSON_FILE)
>  if not return_value:
>  return return_value
> @@ -416,6 +440,32 @@ class DataSource(object):
>  def get_package_mirror_info(self):
>  return self.distro.get_package_mirror_info(data_source=self)
>  
> +def maintain_metadata(self, maintenance_event):

+1, changing

> +"""Refresh cached metadata if the datasource handles this event.
> +
> +The datasource defines a network_maintenance_mask attribute which
> +authorizes refreshing all cached metadata due to any number of
> +supported MaintenenanceEvent types.
> +
> +@param maintenance_event: The source MaintenanceEvent type
> +observed to which the datasource may react.
> +
> +@return True if the datasource has updated cached metadata due to the
> +   the provided maintenance_event type. MaintenanceEvents will be
> +   something like boot, configchange, device_add, device_remove etc.
> +"""
> +if bool(maintenance_event & self.network_maintenance_mask):
> +LOG.debug(
> +"Re-crawling datasource metadata due to maintenance event: 
> '%s'",
> +MAINTENANCE_EVENT_STR.get(maintenance_event, 
> maintenance_event))

fair, was trying to catch a bad traceback if for some-reason we have an 
unexpected caller. for an datasource which extends for events not defined in 
the stock MaintenanceEvent class. A corner case we shouldn't worry about I 
suppose.

> +result = self.get_data(clear_cache=True)
> +if result:
> +return True
> +else:
> +LOG.warning(
> +'Re-crawling metadata reported invalid datasource type')
> +return False
> +
>  def check_instance_id(self, sys_cfg):
>  # quickly (local check only) if self.instance_id is still
>  return False
> @@ -442,7 +492,7 @@ class DataSource(object):
>  return default
>  
>  @property
> -def network_config(self):
> +def network_config(self, regenerate=False):

Hrm, not sure how this got in here, I was probably going at this earlier and 
then relized it was a @property and as such we can't provide additional params.

>  return None
>  
>  @property


-- 
https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/348000
Your team cloud-init commiters is requested to review the proposed merge of 
~chad.smith/cloud-init:fea

Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:bug/1776701-openstack-local-no-probe-on-ec2 into cloud-init:master

2018-06-14 Thread Ryan Harper
If not already done so, we should update the OpenStack datasource docs to 
indicate which product strings/chassis ids are used to positively identify 
OpenStack datasource on VMs and containers.

Diff comments:

> diff --git a/cloudinit/sources/DataSourceOpenStack.py 
> b/cloudinit/sources/DataSourceOpenStack.py
> index 1a12a3f..e7b0b41 100644
> --- a/cloudinit/sources/DataSourceOpenStack.py
> +++ b/cloudinit/sources/DataSourceOpenStack.py
> @@ -205,6 +209,25 @@ def read_metadata_service(base_url, ssl_details=None,
>  return reader.read_v2()
>  
>  
> +def detect_openstack():
> +"""Return True when a potential OpenStack platform is detected."""
> +cpu_arch = os.uname()[4]
> +if not re.match(r'i.86|x86_64', cpu_arch):
> +return True  # Non-Intel cpus don't properly report dmi product names
> +product_name = util.read_dmi_data('system-product-name')
> +if product_name in ('OpenStack Nova', 'OpenStack Compute'):

Magic values, move to top of file in a variable?

> +return True
> +elif util.read_dmi_data('chassis-asset-tag') == 'OpenTelekomCloud':
> +return True

Same here

> +elif os.path.exists('/proc/1/environ'):
> +environ_content = util.load_file('/proc/1/environ')
> +env_items = re.split(r'\0|=', environ_content)
> +pid1_environ = dict(zip(env_items[::2], env_items[1::2]))
> +if pid1_environ.get('product_name') == 'OpenStack Nova':
> +return True
> +return False
> +
> +
>  # Used to match classes to dependencies
>  datasources = [
>  (DataSourceOpenStackLocal, (sources.DEP_FILESYSTEM,)),


-- 
https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/347937
Your team cloud-init commiters is requested to review the proposed merge of 
~chad.smith/cloud-init:bug/1776701-openstack-local-no-probe-on-ec2 into 
cloud-init:master.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp


Re: [Cloud-init-dev] [Merge] ~smoser/cloud-init:fix/lxd-only-create-network-if-noexist into cloud-init:master

2018-06-14 Thread Ryan Harper



Diff comments:

> diff --git a/cloudinit/config/cc_lxd.py b/cloudinit/config/cc_lxd.py
> index 09374d2..b6ee756 100644
> --- a/cloudinit/config/cc_lxd.py
> +++ b/cloudinit/config/cc_lxd.py
> @@ -251,4 +257,47 @@ def bridge_to_cmd(bridge_cfg):
>  
>  return cmd_create, cmd_attach
>  
> +
> +def _network_exists(name):
> +"""Return boolean indicating if network exists."""
> +try:
> +_out, _err = util.subp(["lxc", "network", "show", name],

won't this trigger the initial client registration? 

Is there no other way to check for this info?

Also, older lxc doesn't have lxc network (like lxd-client 2.x on Xenial)
that exits 1 and says 'unknown command'; 

Wouldn't it be better to check via iproute2?

% ip link show type bridge
9: virbr0:  mtu 1500 qdisc noqueue state UP 
mode DEFAULT group default qlen 1000
link/ether 52:54:00:d4:d2:41 brd ff:ff:ff:ff:ff:ff
126: lxdbr0:  mtu 1500 qdisc noqueue state UP 
mode DEFAULT group default qlen 1000
link/ether fe:1d:80:7f:ab:f7 brd ff:ff:ff:ff:ff:ff

It's possible that lxd doesn't have a config for lxdbr0, but it's already 
defined.

> +   update_env={'LC_ALL': 'C'})
> +return True
> +except util.ProcessExecutionError as e:
> +if e.exit_code != 1 or "not found" not in e.stderr.lower():
> +raise e
> +return False
> +
> +
> +def _network_delete(name):
> +util.subp(["lxc", "network", "delete", name])
> +
> +
> +def maybe_delete_network(cfg, default=None):
> +"""Some versions of lxd init automatically create a lxdbr0.
> +If cfg specified to create that name, then we need to delete it
> +so that the creation will work.  If cfg specified a name other
> +than default, or mode == existing, then do not do anything.
> +
> +https://github.com/lxc/lxd/issues/4649
> +"""
> +
> +if default is None:
> +default = _DEFAULT_NETWORK_NAME
> +
> +if cfg.get("mode", "new") == "existing":
> +return
> +
> +name = cfg.get("name", default)
> +if name != default:
> +return
> +
> +if not _network_exists(name):
> +return
> +
> +LOG.debug("Removing lxd network '%s'", name)
> +_network_delete(name)
> +
> +
>  # vi: ts=4 expandtab


-- 
https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/348005
Your team cloud-init commiters is requested to review the proposed merge of 
~smoser/cloud-init:fix/lxd-only-create-network-if-noexist into 
cloud-init:master.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp


Re: [Cloud-init-dev] [Merge] ~smoser/cloud-init:fix/lxd-only-create-network-if-noexist into cloud-init:master

2018-06-14 Thread Server Team CI bot
Review: Approve continuous-integration

PASSED: Continuous integration, rev:c16f9104673e1e8a1fdaa024cc3511d5ad147c17
https://jenkins.ubuntu.com/server/job/cloud-init-ci/85/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/85/rebuild

-- 
https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/348005
Your team cloud-init commiters is requested to review the proposed merge of 
~smoser/cloud-init:fix/lxd-only-create-network-if-noexist into 
cloud-init:master.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp


Re: [Cloud-init-dev] [Merge] ~smoser/cloud-init:fix/lxd-only-create-network-if-noexist into cloud-init:master

2018-06-14 Thread Server Team CI bot
Review: Approve continuous-integration

PASSED: Continuous integration, rev:0cede0e8ee836f150b6382fddc551735fbae52c2
https://jenkins.ubuntu.com/server/job/cloud-init-ci/86/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/86/rebuild

-- 
https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/348005
Your team cloud-init commiters is requested to review the proposed merge of 
~smoser/cloud-init:fix/lxd-only-create-network-if-noexist into 
cloud-init:master.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp


Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:bug/1776701-openstack-local-no-probe-on-ec2 into cloud-init:master

2018-06-14 Thread Scott Moser
some minor things.


Diff comments:

> diff --git a/cloudinit/sources/DataSourceOpenStack.py 
> b/cloudinit/sources/DataSourceOpenStack.py
> index 1a12a3f..e7b0b41 100644
> --- a/cloudinit/sources/DataSourceOpenStack.py
> +++ b/cloudinit/sources/DataSourceOpenStack.py
> @@ -205,6 +209,25 @@ def read_metadata_service(base_url, ssl_details=None,
>  return reader.read_v2()
>  
>  
> +def detect_openstack():
> +"""Return True when a potential OpenStack platform is detected."""
> +cpu_arch = os.uname()[4]
> +if not re.match(r'i.86|x86_64', cpu_arch):
> +return True  # Non-Intel cpus don't properly report dmi product names

in util.read_dmi_data we have:

if not (uname_arch == 'x86_64' or (uname_arch.startswith("i") and 
uname_arch[2:] == "86")

over engineered, but faster.
$ CR=$'\n'; python3 -m timeit --setup="import re${CR}def check(cpu): return 
re.match(r'i.86|x86_64', cpu)" "check('x86_64')"
100 loops, best of 3: 0.775 usec per loop

$ CR=$'\n'; python3 -m timeit --setup="import re${CR}def check(cpu): return 
cpu.startswith('i') and cpu[2:] == '86'" "check('x86_64')"
1000 loops, best of 3: 0.178 usec per loop

Perhaps pull that out and make a util.is_x86 ?

> +product_name = util.read_dmi_data('system-product-name')
> +if product_name in ('OpenStack Nova', 'OpenStack Compute'):
> +return True
> +elif util.read_dmi_data('chassis-asset-tag') == 'OpenTelekomCloud':
> +return True
> +elif os.path.exists('/proc/1/environ'):

this is 'get_proc_env'.
you can then write this as:

elif get_proc_env(1).get('product_name') == 'OpenStack Nova'

> +environ_content = util.load_file('/proc/1/environ')
> +env_items = re.split(r'\0|=', environ_content)
> +pid1_environ = dict(zip(env_items[::2], env_items[1::2]))
> +if pid1_environ.get('product_name') == 'OpenStack Nova':
> +return True
> +return False
> +
> +
>  # Used to match classes to dependencies
>  datasources = [
>  (DataSourceOpenStackLocal, (sources.DEP_FILESYSTEM,)),


-- 
https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/347937
Your team cloud-init commiters is requested to review the proposed merge of 
~chad.smith/cloud-init:bug/1776701-openstack-local-no-probe-on-ec2 into 
cloud-init:master.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp


[Cloud-init-dev] [Merge] ~smoser/cloud-init:fix/lxd-only-create-network-if-noexist into cloud-init:master

2018-06-14 Thread Scott Moser
Scott Moser has proposed merging 
~smoser/cloud-init:fix/lxd-only-create-network-if-noexist into 
cloud-init:master.

Commit message:
lxd: Delete lxdbr0 network if it exists and is to be created.

Newer versions (3.0.1+) of lxd create the 'lxdbr0' network when
'lxd init --auto' is invoked.

When cloud-init is given a network configuration to pass on to
lxc and that config had no name specified or 'lxdbr0', then cloud-init
would fail to create the network as it already exists.

The change here is to check if it exists and delete it if it does.

LP: #1776958

Requested reviews:
  cloud-init commiters (cloud-init-dev)
Related bugs:
  Bug #1776958 in cloud-init: "error creating lxdbr0."
  https://bugs.launchpad.net/cloud-init/+bug/1776958

For more details, see:
https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/348005

see commit message
-- 
Your team cloud-init commiters is requested to review the proposed merge of 
~smoser/cloud-init:fix/lxd-only-create-network-if-noexist into 
cloud-init:master.
diff --git a/cloudinit/config/cc_lxd.py b/cloudinit/config/cc_lxd.py
index 09374d2..b6ee756 100644
--- a/cloudinit/config/cc_lxd.py
+++ b/cloudinit/config/cc_lxd.py
@@ -47,11 +47,16 @@ lxd-bridge will be configured accordingly.
 domain: 
 """
 
+from cloudinit import log as logging
 from cloudinit import util
 import os
 
 distros = ['ubuntu']
 
+LOG = logging.getLogger(__name__)
+
+_DEFAULT_NETWORK_NAME = "lxdbr0"
+
 
 def handle(name, cfg, cloud, log, args):
 # Get config
@@ -134,6 +139,7 @@ def handle(name, cfg, cloud, log, args):
'--frontend=noninteractive'])
 else:
 # Built-in LXD bridge support
+maybe_delete_network(bridge_cfg)
 cmd_create, cmd_attach = bridge_to_cmd(bridge_cfg)
 if cmd_create:
 log.debug("Creating lxd bridge: %s" %
@@ -204,7 +210,7 @@ def bridge_to_cmd(bridge_cfg):
 if bridge_cfg.get("mode") == "none":
 return None, None
 
-bridge_name = bridge_cfg.get("name", "lxdbr0")
+bridge_name = bridge_cfg.get("name", _DEFAULT_NETWORK_NAME)
 cmd_create = []
 cmd_attach = ["lxc", "network", "attach-profile", bridge_name,
   "default", "eth0", "--force-local"]
@@ -251,4 +257,47 @@ def bridge_to_cmd(bridge_cfg):
 
 return cmd_create, cmd_attach
 
+
+def _network_exists(name):
+"""Return boolean indicating if network exists."""
+try:
+_out, _err = util.subp(["lxc", "network", "show", name],
+   update_env={'LC_ALL': 'C'})
+return True
+except util.ProcessExecutionError as e:
+if e.exit_code != 1 or "not found" not in e.stderr.lower():
+raise e
+return False
+
+
+def _network_delete(name):
+util.subp(["lxc", "network", "delete", name])
+
+
+def maybe_delete_network(cfg, default=None):
+"""Some versions of lxd init automatically create a lxdbr0.
+If cfg specified to create that name, then we need to delete it
+so that the creation will work.  If cfg specified a name other
+than default, or mode == existing, then do not do anything.
+
+https://github.com/lxc/lxd/issues/4649
+"""
+
+if default is None:
+default = _DEFAULT_NETWORK_NAME
+
+if cfg.get("mode", "new") == "existing":
+return
+
+name = cfg.get("name", default)
+if name != default:
+return
+
+if not _network_exists(name):
+return
+
+LOG.debug("Removing lxd network '%s'", name)
+_network_delete(name)
+
+
 # vi: ts=4 expandtab
diff --git a/tests/unittests/test_handler/test_handler_lxd.py b/tests/unittests/test_handler/test_handler_lxd.py
index a205498..02b0c8d 100644
--- a/tests/unittests/test_handler/test_handler_lxd.py
+++ b/tests/unittests/test_handler/test_handler_lxd.py
@@ -4,6 +4,7 @@ from cloudinit.config import cc_lxd
 from cloudinit.sources import DataSourceNoCloud
 from cloudinit import (distros, helpers, cloud)
 from cloudinit.tests import helpers as t_help
+from cloudinit.util import ProcessExecutionError
 
 try:
 from unittest import mock
@@ -33,12 +34,16 @@ class TestLxd(t_help.CiTestCase):
 cc = cloud.Cloud(ds, paths, {}, d, None)
 return cc
 
+@mock.patch("cloudinit.config.cc_lxd.maybe_delete_network")
 @mock.patch("cloudinit.config.cc_lxd.util")
-def test_lxd_init(self, mock_util):
+def test_lxd_init(self, mock_util, m_maybe_del):
 cc = self._get_cloud('ubuntu')
 mock_util.which.return_value = True
+m_maybe_del.return_value = None
 cc_lxd.handle('cc_lxd', self.lxd_cfg, cc, self.logger, [])
 self.assertTrue(mock_util.which.called)
+# no bridge config, so maybe_delete should not be called.
+self.assertFalse(m_maybe_del.called)
 init_call = mock_util.subp.call_args_list[0][0][0]
 self.assertEqual(init_call,
  ['lxd', 'init', '--auto',
@@ -46,32 +51,39 @@ class TestLx

Re: [Cloud-init-dev] [Merge] ~smoser/cloud-init:fix/lxd-only-create-network-if-noexist into cloud-init:master

2018-06-14 Thread Chad Smith



Diff comments:

> diff --git a/cloudinit/config/cc_lxd.py b/cloudinit/config/cc_lxd.py
> index 09374d2..b6ee756 100644
> --- a/cloudinit/config/cc_lxd.py
> +++ b/cloudinit/config/cc_lxd.py
> @@ -251,4 +257,47 @@ def bridge_to_cmd(bridge_cfg):
>  
>  return cmd_create, cmd_attach
>  
> +
> +def _network_exists(name):
> +"""Return boolean indicating if network exists."""
> +try:
> +_out, _err = util.subp(["lxc", "network", "show", name],

Instead of either option, could we just use cloudinit.net.is_bridge('lxdbr0')?

> +   update_env={'LC_ALL': 'C'})
> +return True
> +except util.ProcessExecutionError as e:
> +if e.exit_code != 1 or "not found" not in e.stderr.lower():
> +raise e
> +return False
> +
> +
> +def _network_delete(name):
> +util.subp(["lxc", "network", "delete", name])
> +
> +
> +def maybe_delete_network(cfg, default=None):
> +"""Some versions of lxd init automatically create a lxdbr0.
> +If cfg specified to create that name, then we need to delete it
> +so that the creation will work.  If cfg specified a name other
> +than default, or mode == existing, then do not do anything.
> +
> +https://github.com/lxc/lxd/issues/4649
> +"""
> +
> +if default is None:
> +default = _DEFAULT_NETWORK_NAME
> +
> +if cfg.get("mode", "new") == "existing":
> +return
> +
> +name = cfg.get("name", default)
> +if name != default:
> +return
> +
> +if not _network_exists(name):
> +return
> +
> +LOG.debug("Removing lxd network '%s'", name)
> +_network_delete(name)
> +
> +
>  # vi: ts=4 expandtab


-- 
https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/348005
Your team cloud-init commiters is requested to review the proposed merge of 
~smoser/cloud-init:fix/lxd-only-create-network-if-noexist into 
cloud-init:master.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp


Re: [Cloud-init-dev] [Merge] ~smoser/cloud-init:fix/lxd-only-create-network-if-noexist into cloud-init:master

2018-06-14 Thread Scott Moser



Diff comments:

> diff --git a/cloudinit/config/cc_lxd.py b/cloudinit/config/cc_lxd.py
> index 09374d2..b6ee756 100644
> --- a/cloudinit/config/cc_lxd.py
> +++ b/cloudinit/config/cc_lxd.py
> @@ -251,4 +257,47 @@ def bridge_to_cmd(bridge_cfg):
>  
>  return cmd_create, cmd_attach
>  
> +
> +def _network_exists(name):
> +"""Return boolean indicating if network exists."""
> +try:
> +_out, _err = util.subp(["lxc", "network", "show", name],

@ryan,
a.) older lxc wont take this path.. it will take the debconf-communicate path. 
this path is only taken if we have anlxd that has a 'network' subcommand.
b.) checking via any mechanism other than asking lxc is just asking for 
problems.  The thing that knows is lxd, so ask it.  even simpler than 'ip 
route' is just os.path.exists("/sys/class/net/lxdbr0") but that is baking in 
knowledge of lxc that I idon't have to k now.
c.) "initial client registration" was going to happen anyway when we *create* 
the network.

if you all are insistent on not using lxc to ask lxd if it has a network, then 
i'd just use the is_bridge path, but honestly that could be brittle to lxd 
implementations behavior.

> +   update_env={'LC_ALL': 'C'})
> +return True
> +except util.ProcessExecutionError as e:
> +if e.exit_code != 1 or "not found" not in e.stderr.lower():
> +raise e
> +return False
> +
> +
> +def _network_delete(name):
> +util.subp(["lxc", "network", "delete", name])
> +
> +
> +def maybe_delete_network(cfg, default=None):
> +"""Some versions of lxd init automatically create a lxdbr0.
> +If cfg specified to create that name, then we need to delete it
> +so that the creation will work.  If cfg specified a name other
> +than default, or mode == existing, then do not do anything.
> +
> +https://github.com/lxc/lxd/issues/4649
> +"""
> +
> +if default is None:
> +default = _DEFAULT_NETWORK_NAME
> +
> +if cfg.get("mode", "new") == "existing":
> +return
> +
> +name = cfg.get("name", default)
> +if name != default:
> +return
> +
> +if not _network_exists(name):
> +return
> +
> +LOG.debug("Removing lxd network '%s'", name)
> +_network_delete(name)
> +
> +
>  # vi: ts=4 expandtab


-- 
https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/348005
Your team cloud-init commiters is requested to review the proposed merge of 
~smoser/cloud-init:fix/lxd-only-create-network-if-noexist into 
cloud-init:master.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp


Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:bug/1776701-openstack-local-no-probe-on-ec2 into cloud-init:master

2018-06-14 Thread Server Team CI bot
Review: Needs Fixing continuous-integration

FAILED: Continuous integration, rev:c84f6798829de3a5b6817057dbea4ba1bdf46081
https://jenkins.ubuntu.com/server/job/cloud-init-ci/87/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/87/rebuild

-- 
https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/347937
Your team cloud-init commiters is requested to review the proposed merge of 
~chad.smith/cloud-init:bug/1776701-openstack-local-no-probe-on-ec2 into 
cloud-init:master.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp


Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:bug/1776701-openstack-local-no-probe-on-ec2 into cloud-init:master

2018-06-14 Thread Server Team CI bot
Review: Needs Fixing continuous-integration

FAILED: Continuous integration, rev:2effe5c3a1d89c38e3890c2f275046354bfd1b98
https://jenkins.ubuntu.com/server/job/cloud-init-ci/88/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/88/rebuild

-- 
https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/347937
Your team cloud-init commiters is requested to review the proposed merge of 
~chad.smith/cloud-init:bug/1776701-openstack-local-no-probe-on-ec2 into 
cloud-init:master.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp


Re: [Cloud-init-dev] [Merge] ~chad.smith/cloud-init:bug/1776701-openstack-local-no-probe-on-ec2 into cloud-init:master

2018-06-14 Thread Server Team CI bot
Review: Needs Fixing continuous-integration

FAILED: Continuous integration, rev:44caa4afd5810f34091d218638de35c0aa6aaf19
https://jenkins.ubuntu.com/server/job/cloud-init-ci/89/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/89/rebuild

-- 
https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/347937
Your team cloud-init commiters is requested to review the proposed merge of 
~chad.smith/cloud-init:bug/1776701-openstack-local-no-probe-on-ec2 into 
cloud-init:master.

___
Mailing list: https://launchpad.net/~cloud-init-dev
Post to : cloud-init-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~cloud-init-dev
More help   : https://help.launchpad.net/ListHelp