** Description changed:
+ === Begin SRU Template ===
+ [Impact]
+ When attempting to launch a Bionic instance on Oracle Cloud Infrastructure,
with an explicitly set datasource: [ Oracle ], the instance fails to run the
OracleDataSource. This eventually leads to cloud-init falling back to
NoDataSource. The root cause is cloud-init attempting to add routes to create
an Ephemeral DHCP network. We can instead check for a response from the
hardcoded metadata URL and skip adding unnecessary routes.
+
+ [Test Case]
+ 1. Launch Oracle Bionic instance
+ 2. Verify the datasource listed via `cloud-init status -l` shows
DataSourceOracle and not DataSourceNoCloud
+
+ [Regression Potential]
+ If the metadata service is down, we'll fall back to the erroneous behavior.
However, cloud-init will fail in other ways if the metadata service is
inaccessible.
+
+ [Other Info]
+ Github PR: https://github.com/canonical/cloud-init/pull/988
+ Upstream commit:
+
https://github.com/canonical/cloud-init/commit/612e39087aee3b1242765e7c4f463f54a6ebd723
+
+ === End SRU Template ===
+
+ Initial bug:
+
When attempting to launch a Bionic instance on Oracle Cloud
Infrastructure, with an explicitly set datasource: [ Oracle ], the
instance fails to run the OracleDataSource. This leads to the instance
not having SSH keys imported from the metadata service. The failure is
related to the command:
-
- Running command ['ip', '-4', 'route', 'add', '0.0.0.0/0', 'via', '10.0.0.1',
'dev', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
+ Running command ['ip', '-4', 'route', 'add', '0.0.0.0/0', 'via',
+ '10.0.0.1', 'dev', 'ens3'] with allowed return codes [0] (shell=False,
+ capture=True)
Which showed up in the logs :
2021-08-11 13:56:13,289 - util.py[DEBUG]: Reading from
/var/tmp/cloud-init/cloud-init-dhcp-p8n35ztd/dhcp.leases (quiet=False)
2021-08-11 13:56:13,289 - util.py[DEBUG]: Read 519 bytes from
/var/tmp/cloud-init/cloud-init-dhcp-p8n35ztd/dhcp.leases
2021-08-11 13:56:13,289 - dhcp.py[DEBUG]: Received dhcp lease on ens3 for
10.0.0.66/255.255.255.0
2021-08-11 13:56:13,289 - __init__.py[DEBUG]: Attempting setup of ephemeral
network on ens3 with 10.0.0.66/24 brd 10.0.0.255
2021-08-11 13:56:13,289 - subp.py[DEBUG]: Running command ['ip', '-family',
'inet', 'addr', 'add', '10.0.0.66/24', 'broadcast', '10.0.0.255', 'dev',
'ens3'] with allowed return codes [0] (shell=False, capture=True)
2021-08-11 13:56:13,291 - __init__.py[DEBUG]: Skip ephemeral network setup,
ens3 already has address 10.0.0.66
2021-08-11 13:56:13,291 - subp.py[DEBUG]: Running command ['ip', '-4',
'route', 'add', '0.0.0.0/0', 'via', '10.0.0.1', 'dev', 'ens3'] with allowed
return codes [0] (shell=False, capture=True)
2021-08-11 13:56:13,293 - handlers.py[DEBUG]: finish:
init-local/search-Oracle: FAIL: no local data found from DataSourceOracle
2021-08-11 13:56:13,293 - util.py[WARNING]: Getting data from <class
'cloudinit.sources.DataSourceOracle.DataSourceOracle'> failed
2021-08-11 13:56:13,293 - util.py[DEBUG]: Getting data from <class
'cloudinit.sources.DataSourceOracle.DataSourceOracle'> failed
Traceback (most recent call last):
- File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line
792, in find_source
- if s.update_metadata([EventType.BOOT_NEW_INSTANCE]):
- File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line
681, in update_metadata
- result = self.get_data()
- File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line
292, in get_data
- return_value = self._get_data()
- File
"/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceOracle.py", line
138, in _get_data
- with network_context:
- File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 57, in
__enter__
- return self.obtain_lease()
- File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 110, in
obtain_lease
- ephipv4.__enter__()
- File "/usr/lib/python3/dist-packages/cloudinit/net/__init__.py", line 1088,
in __enter__
- self._bringup_static_routes()
- File "/usr/lib/python3/dist-packages/cloudinit/net/__init__.py", line 1142,
in _bringup_static_routes
- ['dev', self.interface], capture=True)
- File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 295, in subp
- cmd=args)
+ File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line
792, in find_source
+ if s.update_metadata([EventType.BOOT_NEW_INSTANCE]):
+ File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line
681, in update_metadata
+ result = self.get_data()
+ File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line
292, in get_data
+ return_value = self._get_data()
+ File
"/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceOracle.py", line
138, in _get_data
+ with network_context:
+ File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 57, in
__enter__
+ return self.obtain_lease()
+ File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 110, in
obtain_lease
+ ephipv4.__enter__()
+ File "/usr/lib/python3/dist-packages/cloudinit/net/__init__.py", line 1088,
in __enter__
+ self._bringup_static_routes()
+ File "/usr/lib/python3/dist-packages/cloudinit/net/__init__.py", line 1142,
in _bringup_static_routes
+ ['dev', self.interface], capture=True)
+ File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 295, in subp
+ cmd=args)
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['ip', '-4', 'route', 'add', '0.0.0.0/0', 'via', '10.0.0.1', 'dev',
'ens3']
Exit code: 2
Reason: -
- Stdout:
+ Stdout:
Stderr: RTNETLINK answers: File exists
-
This eventually leads to cloud-init falling back to NoDataSource.
To create this image, I:
* Updated CPC's livecd-rootfs code for Oracle to include:
# etc/cloud/cloud.cfg.d/99-oracle-compute-infra-datasource.cfg"
# Configuration for Oracle Cloud Infrastructure
datasource_list: [ Oracle ]
* created an image using CPC's livecd-rootfs using ubuntu-bartender
* registered a custom image in OCI
* attempted to create an instance using the custom image
I was unable to connect via ssh, getting "Permission denied (publickey)"
- I attempted to create a serial connection, however, I was never able to
successfully SSH in. It just hung forever.
+ I attempted to create a serial connection, however, I was never able to
successfully SSH in. It just hung forever.
In a second attempt, I tried to pass in a username:password to cloud-
init. However, due to the failure of the datasource, and fallback to
NoDataSource, my custom data was not loaded either
I was able to collect logs by terminating the instance, but keeping the
boot volume. I then created a Bionic instance using the platform image,
and verified that it worked with the OpenStack datasource currently in
use. I then attached the boot volume from the now terminated instance as
a block volume, ran the required iscsi commands (found via the web
console after attaching the block volume), and mounted the drive to
/mnt/nods. I was then able to collect the logs in
/mnt/nods/var/log/cloud-init*. Because of how I had to collect logs, i
was unable to run `cloud-init collect-logs`. I actually could run cloud-
init in a chroot setup, like `sudo chroot /mnt/nods cloud-init collect-
logs`. This failed with being unable to find the command `cloud-init`.
Honestly not sure if that's the correct approach in the circumstance.
To reproduce, an image would need made with the datasource explicitly
set to Oracle.
** Also affects: cloud-init (Ubuntu Bionic)
Importance: Undecided
Status: New
** Description changed:
=== Begin SRU Template ===
[Impact]
When attempting to launch a Bionic instance on Oracle Cloud Infrastructure,
with an explicitly set datasource: [ Oracle ], the instance fails to run the
OracleDataSource. This eventually leads to cloud-init falling back to
NoDataSource. The root cause is cloud-init attempting to add routes to create
an Ephemeral DHCP network. We can instead check for a response from the
hardcoded metadata URL and skip adding unnecessary routes.
[Test Case]
1. Launch Oracle Bionic instance
- 2. Verify the datasource listed via `cloud-init status -l` shows
DataSourceOracle and not DataSourceNoCloud
+ 2. Verify the datasource listed via `cloud-init status -l` shows
DataSourceOracle and not DataSourceNoCloud or DataSourceOpenStack
[Regression Potential]
If the metadata service is down, we'll fall back to the erroneous behavior.
However, cloud-init will fail in other ways if the metadata service is
inaccessible.
[Other Info]
Github PR: https://github.com/canonical/cloud-init/pull/988
Upstream commit:
https://github.com/canonical/cloud-init/commit/612e39087aee3b1242765e7c4f463f54a6ebd723
=== End SRU Template ===
Initial bug:
When attempting to launch a Bionic instance on Oracle Cloud
Infrastructure, with an explicitly set datasource: [ Oracle ], the
instance fails to run the OracleDataSource. This leads to the instance
not having SSH keys imported from the metadata service. The failure is
related to the command:
Running command ['ip', '-4', 'route', 'add', '0.0.0.0/0', 'via',
'10.0.0.1', 'dev', 'ens3'] with allowed return codes [0] (shell=False,
capture=True)
Which showed up in the logs :
2021-08-11 13:56:13,289 - util.py[DEBUG]: Reading from
/var/tmp/cloud-init/cloud-init-dhcp-p8n35ztd/dhcp.leases (quiet=False)
2021-08-11 13:56:13,289 - util.py[DEBUG]: Read 519 bytes from
/var/tmp/cloud-init/cloud-init-dhcp-p8n35ztd/dhcp.leases
2021-08-11 13:56:13,289 - dhcp.py[DEBUG]: Received dhcp lease on ens3 for
10.0.0.66/255.255.255.0
2021-08-11 13:56:13,289 - __init__.py[DEBUG]: Attempting setup of ephemeral
network on ens3 with 10.0.0.66/24 brd 10.0.0.255
2021-08-11 13:56:13,289 - subp.py[DEBUG]: Running command ['ip', '-family',
'inet', 'addr', 'add', '10.0.0.66/24', 'broadcast', '10.0.0.255', 'dev',
'ens3'] with allowed return codes [0] (shell=False, capture=True)
2021-08-11 13:56:13,291 - __init__.py[DEBUG]: Skip ephemeral network setup,
ens3 already has address 10.0.0.66
2021-08-11 13:56:13,291 - subp.py[DEBUG]: Running command ['ip', '-4',
'route', 'add', '0.0.0.0/0', 'via', '10.0.0.1', 'dev', 'ens3'] with allowed
return codes [0] (shell=False, capture=True)
2021-08-11 13:56:13,293 - handlers.py[DEBUG]: finish:
init-local/search-Oracle: FAIL: no local data found from DataSourceOracle
2021-08-11 13:56:13,293 - util.py[WARNING]: Getting data from <class
'cloudinit.sources.DataSourceOracle.DataSourceOracle'> failed
2021-08-11 13:56:13,293 - util.py[DEBUG]: Getting data from <class
'cloudinit.sources.DataSourceOracle.DataSourceOracle'> failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line
792, in find_source
if s.update_metadata([EventType.BOOT_NEW_INSTANCE]):
File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line
681, in update_metadata
result = self.get_data()
File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line
292, in get_data
return_value = self._get_data()
File
"/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceOracle.py", line
138, in _get_data
with network_context:
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 57, in
__enter__
return self.obtain_lease()
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 110, in
obtain_lease
ephipv4.__enter__()
File "/usr/lib/python3/dist-packages/cloudinit/net/__init__.py", line 1088,
in __enter__
self._bringup_static_routes()
File "/usr/lib/python3/dist-packages/cloudinit/net/__init__.py", line 1142,
in _bringup_static_routes
['dev', self.interface], capture=True)
File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 295, in subp
cmd=args)
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['ip', '-4', 'route', 'add', '0.0.0.0/0', 'via', '10.0.0.1', 'dev',
'ens3']
Exit code: 2
Reason: -
Stdout:
Stderr: RTNETLINK answers: File exists
This eventually leads to cloud-init falling back to NoDataSource.
To create this image, I:
* Updated CPC's livecd-rootfs code for Oracle to include:
# etc/cloud/cloud.cfg.d/99-oracle-compute-infra-datasource.cfg"
# Configuration for Oracle Cloud Infrastructure
datasource_list: [ Oracle ]
* created an image using CPC's livecd-rootfs using ubuntu-bartender
* registered a custom image in OCI
* attempted to create an instance using the custom image
I was unable to connect via ssh, getting "Permission denied (publickey)"
I attempted to create a serial connection, however, I was never able to
successfully SSH in. It just hung forever.
In a second attempt, I tried to pass in a username:password to cloud-
init. However, due to the failure of the datasource, and fallback to
NoDataSource, my custom data was not loaded either
I was able to collect logs by terminating the instance, but keeping the
boot volume. I then created a Bionic instance using the platform image,
and verified that it worked with the OpenStack datasource currently in
use. I then attached the boot volume from the now terminated instance as
a block volume, ran the required iscsi commands (found via the web
console after attaching the block volume), and mounted the drive to
/mnt/nods. I was then able to collect the logs in
/mnt/nods/var/log/cloud-init*. Because of how I had to collect logs, i
was unable to run `cloud-init collect-logs`. I actually could run cloud-
init in a chroot setup, like `sudo chroot /mnt/nods cloud-init collect-
logs`. This failed with being unable to find the command `cloud-init`.
Honestly not sure if that's the correct approach in the circumstance.
To reproduce, an image would need made with the datasource explicitly
set to Oracle.
** Description changed:
=== Begin SRU Template ===
[Impact]
When attempting to launch a Bionic instance on Oracle Cloud Infrastructure,
with an explicitly set datasource: [ Oracle ], the instance fails to run the
OracleDataSource. This eventually leads to cloud-init falling back to
NoDataSource. The root cause is cloud-init attempting to add routes to create
an Ephemeral DHCP network. We can instead check for a response from the
hardcoded metadata URL and skip adding unnecessary routes.
[Test Case]
1. Launch Oracle Bionic instance
2. Verify the datasource listed via `cloud-init status -l` shows
DataSourceOracle and not DataSourceNoCloud or DataSourceOpenStack
+ 3. Verify /var/log/cloud-init.log has no errors due to setting up routes.
[Regression Potential]
If the metadata service is down, we'll fall back to the erroneous behavior.
However, cloud-init will fail in other ways if the metadata service is
inaccessible.
[Other Info]
Github PR: https://github.com/canonical/cloud-init/pull/988
Upstream commit:
https://github.com/canonical/cloud-init/commit/612e39087aee3b1242765e7c4f463f54a6ebd723
=== End SRU Template ===
Initial bug:
When attempting to launch a Bionic instance on Oracle Cloud
Infrastructure, with an explicitly set datasource: [ Oracle ], the
instance fails to run the OracleDataSource. This leads to the instance
not having SSH keys imported from the metadata service. The failure is
related to the command:
Running command ['ip', '-4', 'route', 'add', '0.0.0.0/0', 'via',
'10.0.0.1', 'dev', 'ens3'] with allowed return codes [0] (shell=False,
capture=True)
Which showed up in the logs :
2021-08-11 13:56:13,289 - util.py[DEBUG]: Reading from
/var/tmp/cloud-init/cloud-init-dhcp-p8n35ztd/dhcp.leases (quiet=False)
2021-08-11 13:56:13,289 - util.py[DEBUG]: Read 519 bytes from
/var/tmp/cloud-init/cloud-init-dhcp-p8n35ztd/dhcp.leases
2021-08-11 13:56:13,289 - dhcp.py[DEBUG]: Received dhcp lease on ens3 for
10.0.0.66/255.255.255.0
2021-08-11 13:56:13,289 - __init__.py[DEBUG]: Attempting setup of ephemeral
network on ens3 with 10.0.0.66/24 brd 10.0.0.255
2021-08-11 13:56:13,289 - subp.py[DEBUG]: Running command ['ip', '-family',
'inet', 'addr', 'add', '10.0.0.66/24', 'broadcast', '10.0.0.255', 'dev',
'ens3'] with allowed return codes [0] (shell=False, capture=True)
2021-08-11 13:56:13,291 - __init__.py[DEBUG]: Skip ephemeral network setup,
ens3 already has address 10.0.0.66
2021-08-11 13:56:13,291 - subp.py[DEBUG]: Running command ['ip', '-4',
'route', 'add', '0.0.0.0/0', 'via', '10.0.0.1', 'dev', 'ens3'] with allowed
return codes [0] (shell=False, capture=True)
2021-08-11 13:56:13,293 - handlers.py[DEBUG]: finish:
init-local/search-Oracle: FAIL: no local data found from DataSourceOracle
2021-08-11 13:56:13,293 - util.py[WARNING]: Getting data from <class
'cloudinit.sources.DataSourceOracle.DataSourceOracle'> failed
2021-08-11 13:56:13,293 - util.py[DEBUG]: Getting data from <class
'cloudinit.sources.DataSourceOracle.DataSourceOracle'> failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line
792, in find_source
if s.update_metadata([EventType.BOOT_NEW_INSTANCE]):
File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line
681, in update_metadata
result = self.get_data()
File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line
292, in get_data
return_value = self._get_data()
File
"/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceOracle.py", line
138, in _get_data
with network_context:
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 57, in
__enter__
return self.obtain_lease()
File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 110, in
obtain_lease
ephipv4.__enter__()
File "/usr/lib/python3/dist-packages/cloudinit/net/__init__.py", line 1088,
in __enter__
self._bringup_static_routes()
File "/usr/lib/python3/dist-packages/cloudinit/net/__init__.py", line 1142,
in _bringup_static_routes
['dev', self.interface], capture=True)
File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 295, in subp
cmd=args)
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['ip', '-4', 'route', 'add', '0.0.0.0/0', 'via', '10.0.0.1', 'dev',
'ens3']
Exit code: 2
Reason: -
Stdout:
Stderr: RTNETLINK answers: File exists
This eventually leads to cloud-init falling back to NoDataSource.
To create this image, I:
* Updated CPC's livecd-rootfs code for Oracle to include:
# etc/cloud/cloud.cfg.d/99-oracle-compute-infra-datasource.cfg"
# Configuration for Oracle Cloud Infrastructure
datasource_list: [ Oracle ]
* created an image using CPC's livecd-rootfs using ubuntu-bartender
* registered a custom image in OCI
* attempted to create an instance using the custom image
I was unable to connect via ssh, getting "Permission denied (publickey)"
I attempted to create a serial connection, however, I was never able to
successfully SSH in. It just hung forever.
In a second attempt, I tried to pass in a username:password to cloud-
init. However, due to the failure of the datasource, and fallback to
NoDataSource, my custom data was not loaded either
I was able to collect logs by terminating the instance, but keeping the
boot volume. I then created a Bionic instance using the platform image,
and verified that it worked with the OpenStack datasource currently in
use. I then attached the boot volume from the now terminated instance as
a block volume, ran the required iscsi commands (found via the web
console after attaching the block volume), and mounted the drive to
/mnt/nods. I was then able to collect the logs in
/mnt/nods/var/log/cloud-init*. Because of how I had to collect logs, i
was unable to run `cloud-init collect-logs`. I actually could run cloud-
init in a chroot setup, like `sudo chroot /mnt/nods cloud-init collect-
logs`. This failed with being unable to find the command `cloud-init`.
Honestly not sure if that's the correct approach in the circumstance.
To reproduce, an image would need made with the datasource explicitly
set to Oracle.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1939603
Title:
Oracle DataSource Fails When Used With a Bionic Image
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1939603/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
