Found this in the cloud-init.log:
in cloud-init.log:
2016-02-10 14:02:09,604 - cc_resizefs.py[DEBUG]: Resizing / (ext4) using
resize2fs /dev/disk/by-label/cloudimg-rootfs
2016-02-10 14:02:09,604 - util.py[DEBUG]: Running command ('resize2fs',
'/dev/disk/by-label/cloudimg-rootfs') with allowed return
codes[0](shell=False, capture=True)
2016-02-10 14:02:09,615 - util.py[WARNING]: Failed to resize filesystem
(cmd=('resize2fs', '/dev/disk/by-label/cloudimg-rootfs'))
2016-02-10 14:02:09,616 - util.py[DEBUG]: Failed to resize filesystem
(cmd=('resize2fs', '/dev/disk/by-label/cloudimg-rootfs'))
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/cloudinit/config/cc_resizefs.py",
line 172, in do_resize
util.subp(resize_cmd)
File "/usr/lib/python2.7/dist-packages/cloudinit/util.py", line 1539, in
subp
cmd=args)
ProcessExecutionError: Unexpected error while running command.
Command: ('resize2fs', '/dev/disk/by-label/cloudimg-rootfs')
Exit code: 1
Reason: -
Stdout: ''
Stderr: "resize2fs 1.42.9 (4-Feb-2014)\nPlease run 'e2fsck -f
/dev/disk/by-label/cloudimg-rootfs' first.\n\n"
2016-02-10 14:02:09,627 - util.py[DEBUG]: Resizing took 0.023 seconds
2016-02-10 14:02:09,627 - util.py[WARNING]: Running resizefs (<module
'cloudinit.config.cc_resizefs' from
'/usr/lib/python2.7/dist-packages/cloudinit/config/cc_resizefs.pyc'>) failed
2016-02-10 14:02:09,627 - util.py[DEBUG]: Running resizefs (<module
'cloudinit.config.cc_resizefs' from
'/usr/lib/python2.7/dist-packages/cloudinit/config/cc_resizefs.pyc'>) failed
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/cloudinit/stages.py", line 658,
in _run_modules
cc.run(run_name, mod.handle, func_args, freq=freq)
File "/usr/lib/python2.7/dist-packages/cloudinit/cloud.py", line 63, in run
return self._runners.run(name, functor, args, freq, clear_on_fail)
File "/usr/lib/python2.7/dist-packages/cloudinit/helpers.py", line 197,
in run
results = functor(*args)
File "/usr/lib/python2.7/dist-packages/cloudinit/config/cc_resizefs.py",
line 161, in handle
func=do_resize, args=(resize_cmd, log))
File "/usr/lib/python2.7/dist-packages/cloudinit/util.py", line 1875, in
log_time
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/cloudinit/config/cc_resizefs.py",
line 172, in do_resize
util.subp(resize_cmd)
File "/usr/lib/python2.7/dist-packages/cloudinit/util.py", line 1539, in
subp
cmd=args)
ProcessExecutionError: Unexpected error while running command.
Command: ('resize2fs', '/dev/disk/by-label/cloudimg-rootfs')
Exit code: 1
Reason: -
Stdout: ''
Stderr: "resize2fs 1.42.9 (4-Feb-2014)\nPlease run 'e2fsck -f
/dev/disk/by-label/cloudimg-rootfs' first.\n\n"
Is it possible that after we cleared up the disk openstack tried to
resize the disk but it didn't manage and the image got corrupted?
Google search gave a case when someone had an image with this error that
was not bootable:
https://bugs.launchpad.net/diskimage-builder/+bug/1280709/ ("Daniele
Venzano wroteon 2014-02-28:" )
Regards, Balazs
On 15/03/16 14:13, Balazs Varhegyi wrote:
Hi Steve,
Thanks for the suggestion,
I already managed to clear up some space in rescue mode (VM instance has
2,1T disk with 1T free space) and core machine has an extra 270G free
disk too.
Balazs
On 15/03/16 13:56, Stephen Davies wrote:
Hi Balazs,
Have you tried connecting to the image via NBD (network block device).
1) Mount the image.
2) Repair/free some space.
3) Start the instance.
http://blog.vmsplice.net/2011/02/how-to-access-virtual-machine-image.html
Steve
On 15/03/16 11:20, Tomas Vondra wrote:
Balazs Varhegyi <varh1ibalazs@...> writes:
Hi guys,
I have an openstack instance that run out of storage on core machine
and
the instance got paused.
I cleared up the core machine so it has more disk now but when I start
the machine I can't ssh in.
In rescue mode I'm able to ssh in but I want to unrescue the
machine to
be able to use it again without creating a new instance and migrate
everything from this failed instance manually.
I enabled virtlib debug log but nothing interesting came up (or at
least
I didn't notice), the only thing that indicates an error is:
"Domain id=53 name='instance-00000281'
uuid=f2715c2c-6d4e-5cf1-7606-f5552a59cb56 is tainted: host-cpu"
"nova list" shows the machine has the correct IP set but I'm not
able to
ping it on private IP nor via floating IP.
I tracked the network configuration from libvirt.xml and it seems it's
configured correctly to connect to the internal bridge: "br-int" on
Open
VSwitch.
do you have any idea what should I try to unrescue the machine to and
get it into active state?
Regards,
Balazs Varhegyi
Hi!
What does the log and console show when the VM is in normal state?
The rescue mode is not a magical solution to all problem. It just
runs a
clean OS and attaches your broken one as a secondary disk. YOU have
to do
the magic to rescue it.
Tomas
_______________________________________________
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : [email protected]
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : [email protected]
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
