Re: [Openstack] nova backup - instances unreachable

[John Petrini]

Adding disable_libvirt_livesnapshot = false to the nova.conf enabled live
snapshots.

Thanks everyone for the help.

___

John Petrini

NOC Systems Administrator   //   *CoreDial, LLC*   //   coredial.com
//   [image:
Twitter]    [image: LinkedIn]
   [image: Google Plus]
   [image: Blog]

Hillcrest I, 751 Arbor Way, Suite 150, Blue Bell PA, 19422
*P: *215.297.4400 x232   //   *F: *215.297.4401   //   *E: *
jpetr...@coredial.com

[image: Exceptional people. Proven Processes. Innovative Technology.
Discover CoreDial - watch our video]


The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission,  dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.

On Mon, Jan 23, 2017 at 3:55 AM, John Petrini  wrote:

> Hi Eugen,
>
> disable_libvirt_livesnapshot is not present in the nova.conf. During the
> snapshot the nova logs says "Beginning cold snapshot".
>
> I read about this option in the nova documentation but did not realize it
> was the default. In fact I assumed it wasn't since it's in the workarounds
> section. I'll try setting it to false.
>
> Thank You,
>
>
>
> ___
>
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission,  dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from any
> computer.
>
> On Mon, Jan 23, 2017 at 3:16 AM, Eugen Block  wrote:
>
>> Have you enabled live snapshots in nova.conf?
>>
>> The default for this option is "true", so you should check that:
>>
>> disable_libvirt_livesnapshot = false
>>
>> Is it really a live snaphot? What's in the nova-compute.log? It should
>> say something like
>>
>> [instance: XXX] Beginning live snapshot process
>>
>>
>>
>> Regards,
>> Eugen
>>
>>
>>
>> Zitat von John Petrini :
>>
>> Hi All,
>>>
>>> Following up after making this change. Adding write permissions to the
>>> images pool in Ceph did the trick and RBD snapshots now work. However the
>>> instance is still paused for the duration of the snapshot. Is it possible
>>> to do a live snapshot without pausing the instance?
>>>
>>> Thanks,
>>>
>>> John
>>>
>>> On Fri, Jan 13, 2017 at 5:49 AM, Eugen Block  wrote:
>>>
>>> Thanks,

 for anyone interested in this issue, I filed a bug report:
 https://bugs.launchpad.net/nova/+bug/1656242


 Regards,
 Eugen


 Zitat von Mohammed Naser :

 It is likely because this has been tested with QEMU only. I think you

> might want to bring this up with the Nova team.
>
> Sent from my iPhone
>
> On Jan 12, 2017, at 11:28 AM, Eugen Block  wrote:
>
>>
>> I'm not sure if this is the right spot, but I added some log
>> statements
>> into driver.py.
>> First, there's this if-block:
>>
>>if (self._host.has_min_version(MI
>> N_LIBVIRT_LIVESNAPSHOT_VERSION,
>>   MIN_QEMU_LIVESNAPSHOT_VERSION,
>>   host.HV_DRIVER_QEMU)
>> and source_type not in ('lvm')
>> and not CONF.ephemeral_storage_encryption.enabled
>> and not CONF.workarounds.disable_libvirt_livesnapshot):
>>live_snapshot = True
>>   [...]
>>else:
>>live_snapshot = False
>>
>> And I know that it lands in the else-statement. Turns out that
>> _host.has_min_version is "false", because of host.HV_DRIVER_QEMU. We
>> are
>> running on Xen hypervisors. So I tried it with host.HV_DRIVER_XEN and
>> now
>> nova-compute says:
>>
>> [instance: 14b75237-7619-481f-9636-792b64d1be17] instance
>> snapshotting
>> [instance: 14b75237-7619-481f-9636-792b64d1be17] Beginning live
>> snapshot process
>>
>> Now I'm waiting for the result, but at least the VM is still running,
>> so
>> it looks quite promising...
>>
>> And there it is:
>>
>> [instance: 14b75237-7619-481f-9636-792b64d1be17] Snapshot image
>> upload
>> complete
>>
>> I'm testing the image now, and it 

Re: [Openstack] nova backup - instances unreachable

[John Petrini]

Hi Eugen,

disable_libvirt_livesnapshot is not present in the nova.conf. During the
snapshot the nova logs says "Beginning cold snapshot".

I read about this option in the nova documentation but did not realize it
was the default. In fact I assumed it wasn't since it's in the workarounds
section. I'll try setting it to false.

Thank You,



___

The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission,  dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.

On Mon, Jan 23, 2017 at 3:16 AM, Eugen Block  wrote:

> Have you enabled live snapshots in nova.conf?
>
> The default for this option is "true", so you should check that:
>
> disable_libvirt_livesnapshot = false
>
> Is it really a live snaphot? What's in the nova-compute.log? It should say
> something like
>
> [instance: XXX] Beginning live snapshot process
>
>
>
> Regards,
> Eugen
>
>
>
> Zitat von John Petrini :
>
> Hi All,
>>
>> Following up after making this change. Adding write permissions to the
>> images pool in Ceph did the trick and RBD snapshots now work. However the
>> instance is still paused for the duration of the snapshot. Is it possible
>> to do a live snapshot without pausing the instance?
>>
>> Thanks,
>>
>> John
>>
>> On Fri, Jan 13, 2017 at 5:49 AM, Eugen Block  wrote:
>>
>> Thanks,
>>>
>>> for anyone interested in this issue, I filed a bug report:
>>> https://bugs.launchpad.net/nova/+bug/1656242
>>>
>>>
>>> Regards,
>>> Eugen
>>>
>>>
>>> Zitat von Mohammed Naser :
>>>
>>> It is likely because this has been tested with QEMU only. I think you
>>>
 might want to bring this up with the Nova team.

 Sent from my iPhone

 On Jan 12, 2017, at 11:28 AM, Eugen Block  wrote:

>
> I'm not sure if this is the right spot, but I added some log statements
> into driver.py.
> First, there's this if-block:
>
>if (self._host.has_min_version(MIN_LIBVIRT_LIVESNAPSHOT_VERSION
> ,
>   MIN_QEMU_LIVESNAPSHOT_VERSION,
>   host.HV_DRIVER_QEMU)
> and source_type not in ('lvm')
> and not CONF.ephemeral_storage_encryption.enabled
> and not CONF.workarounds.disable_libvirt_livesnapshot):
>live_snapshot = True
>   [...]
>else:
>live_snapshot = False
>
> And I know that it lands in the else-statement. Turns out that
> _host.has_min_version is "false", because of host.HV_DRIVER_QEMU. We
> are
> running on Xen hypervisors. So I tried it with host.HV_DRIVER_XEN and
> now
> nova-compute says:
>
> [instance: 14b75237-7619-481f-9636-792b64d1be17] instance snapshotting
> [instance: 14b75237-7619-481f-9636-792b64d1be17] Beginning live
> snapshot process
>
> Now I'm waiting for the result, but at least the VM is still running,
> so
> it looks quite promising...
>
> And there it is:
>
> [instance: 14b75237-7619-481f-9636-792b64d1be17] Snapshot image upload
> complete
>
> I'm testing the image now, and it works!
>
> Now the question is, why is it defaulting to HV_DRIVER_QEMU and is it
> really necessary to change this directly in the code? Is there any
> other
> way?
>
> Regards,
> Eugen
>
> Zitat von Eugen Block :
>
> Yes, I truncated the file and uploaded it:
>
>>
>> http://dropcanvas.com/ta7nu
>> (First time I used this service, please give me feedback if this
>> doesn't work for you)
>>
>> I see the "Beginning cold snapshot process" message, but I don't know
>> why. Any help is appreciated!
>>
>> Regards,
>> Eugen
>>
>>
>> Zitat von Mohammed Naser :
>>
>> Would you be able to share the logs of a full snapshot run with the
>>
>>> compute node in debug?
>>>
>>> Sent from my iPhone
>>>
>>> On Jan 12, 2017, at 7:47 AM, Eugen Block  wrote:
>>>

 That's strange, I also searched for this message, but nothing there.
 I have debug logs enabled on compute node but I don't see anything
 regarding ceph. No matter, what I do, my instance is always
 shutdown before
 a snapshot is taken. What else can I try?


 Zitat von John Petrini :

 Mohammed,

>
> It looks like you may be right. Just found the permissions issue in
> the

Re: [Openstack] nova backup - instances unreachable

[Eugen Block]

Have you enabled live snapshots in nova.conf?

The default for this option is "true", so you should check that:

disable_libvirt_livesnapshot = false

Is it really a live snaphot? What's in the nova-compute.log? It should  
say something like


[instance: XXX] Beginning live snapshot process


Regards,
Eugen



Zitat von John Petrini :


Hi All,

Following up after making this change. Adding write permissions to the
images pool in Ceph did the trick and RBD snapshots now work. However the
instance is still paused for the duration of the snapshot. Is it possible
to do a live snapshot without pausing the instance?

Thanks,

John

On Fri, Jan 13, 2017 at 5:49 AM, Eugen Block  wrote:


Thanks,

for anyone interested in this issue, I filed a bug report:
https://bugs.launchpad.net/nova/+bug/1656242


Regards,
Eugen


Zitat von Mohammed Naser :

It is likely because this has been tested with QEMU only. I think you

might want to bring this up with the Nova team.

Sent from my iPhone

On Jan 12, 2017, at 11:28 AM, Eugen Block  wrote:


I'm not sure if this is the right spot, but I added some log statements
into driver.py.
First, there's this if-block:

   if (self._host.has_min_version(MIN_LIBVIRT_LIVESNAPSHOT_VERSION,
  MIN_QEMU_LIVESNAPSHOT_VERSION,
  host.HV_DRIVER_QEMU)
and source_type not in ('lvm')
and not CONF.ephemeral_storage_encryption.enabled
and not CONF.workarounds.disable_libvirt_livesnapshot):
   live_snapshot = True
  [...]
   else:
   live_snapshot = False

And I know that it lands in the else-statement. Turns out that
_host.has_min_version is "false", because of host.HV_DRIVER_QEMU. We are
running on Xen hypervisors. So I tried it with host.HV_DRIVER_XEN and now
nova-compute says:

[instance: 14b75237-7619-481f-9636-792b64d1be17] instance snapshotting
[instance: 14b75237-7619-481f-9636-792b64d1be17] Beginning live
snapshot process

Now I'm waiting for the result, but at least the VM is still running, so
it looks quite promising...

And there it is:

[instance: 14b75237-7619-481f-9636-792b64d1be17] Snapshot image upload
complete

I'm testing the image now, and it works!

Now the question is, why is it defaulting to HV_DRIVER_QEMU and is it
really necessary to change this directly in the code? Is there any other
way?

Regards,
Eugen

Zitat von Eugen Block :

Yes, I truncated the file and uploaded it:


http://dropcanvas.com/ta7nu
(First time I used this service, please give me feedback if this
doesn't work for you)

I see the "Beginning cold snapshot process" message, but I don't know
why. Any help is appreciated!

Regards,
Eugen


Zitat von Mohammed Naser :

Would you be able to share the logs of a full snapshot run with the

compute node in debug?

Sent from my iPhone

On Jan 12, 2017, at 7:47 AM, Eugen Block  wrote:


That's strange, I also searched for this message, but nothing there.
I have debug logs enabled on compute node but I don't see anything
regarding ceph. No matter, what I do, my instance is always  
shutdown before

a snapshot is taken. What else can I try?


Zitat von John Petrini :

Mohammed,


It looks like you may be right. Just found the permissions issue in
the
nova log on the compute node.

4-e8f52e4fbcfb 691caf1c10354efab3e3c8ed61b7d89a
49bc5e5bf2684bd0948d9f94c7875027 - - -] Performing standard snapshot
because direct snapshot failed: no write permission on storage pool
images

I'm going to test the change and will send an update you all with the
results.

Thank You,

___

John Petrini





Yes, we are also running Mitaka and I also read Sebastien Han's

blogs ;-)

our snapshots are not happening at the RBD level,


they are being copied and uploaded to glance which takes up a lot
of space
and is very slow.



Unfortunately, that's what we are experiencing, too. I don't know if
there's something I missed in the nova configs or somewhere else,
but I'm
relieved that I'm not the only one :-)

While writing this email I searched again and found something:

https://specs.openstack.org/openstack/nova-specs/specs/mitak
a/implemented/rbd-instance-snapshots.html

https://review.openstack.org/#/c/205282/

It seems to be implemented already, I'm looking for the config
options to
set. If you manage to get nova to make rbd snapshots, please let me
know ;-)

Regards,
Eugen



Zitat von John Petrini :

Hi Eugen,



Thanks for the response! That makes a lost of sense and is what I
figured
was going on but I missed it in the documentation. We use Ceph as
well and
I had considered doing the snapshots at the RBD level but I was
hoping
there was someway to accomplish this via nova. I came across this
Sebastien
Han write-up that claims this functionality was added to Mitaka:

Re: [Openstack] nova backup - instances unreachable

[John Petrini]

Hi All,

Following up after making this change. Adding write permissions to the
images pool in Ceph did the trick and RBD snapshots now work. However the
instance is still paused for the duration of the snapshot. Is it possible
to do a live snapshot without pausing the instance?

Thanks,

John

On Fri, Jan 13, 2017 at 5:49 AM, Eugen Block  wrote:

> Thanks,
>
> for anyone interested in this issue, I filed a bug report:
> https://bugs.launchpad.net/nova/+bug/1656242
>
>
> Regards,
> Eugen
>
>
> Zitat von Mohammed Naser :
>
> It is likely because this has been tested with QEMU only. I think you
>> might want to bring this up with the Nova team.
>>
>> Sent from my iPhone
>>
>> On Jan 12, 2017, at 11:28 AM, Eugen Block  wrote:
>>>
>>> I'm not sure if this is the right spot, but I added some log statements
>>> into driver.py.
>>> First, there's this if-block:
>>>
>>>if (self._host.has_min_version(MIN_LIBVIRT_LIVESNAPSHOT_VERSION,
>>>   MIN_QEMU_LIVESNAPSHOT_VERSION,
>>>   host.HV_DRIVER_QEMU)
>>> and source_type not in ('lvm')
>>> and not CONF.ephemeral_storage_encryption.enabled
>>> and not CONF.workarounds.disable_libvirt_livesnapshot):
>>>live_snapshot = True
>>>   [...]
>>>else:
>>>live_snapshot = False
>>>
>>> And I know that it lands in the else-statement. Turns out that
>>> _host.has_min_version is "false", because of host.HV_DRIVER_QEMU. We are
>>> running on Xen hypervisors. So I tried it with host.HV_DRIVER_XEN and now
>>> nova-compute says:
>>>
>>> [instance: 14b75237-7619-481f-9636-792b64d1be17] instance snapshotting
>>> [instance: 14b75237-7619-481f-9636-792b64d1be17] Beginning live
>>> snapshot process
>>>
>>> Now I'm waiting for the result, but at least the VM is still running, so
>>> it looks quite promising...
>>>
>>> And there it is:
>>>
>>> [instance: 14b75237-7619-481f-9636-792b64d1be17] Snapshot image upload
>>> complete
>>>
>>> I'm testing the image now, and it works!
>>>
>>> Now the question is, why is it defaulting to HV_DRIVER_QEMU and is it
>>> really necessary to change this directly in the code? Is there any other
>>> way?
>>>
>>> Regards,
>>> Eugen
>>>
>>> Zitat von Eugen Block :
>>>
>>> Yes, I truncated the file and uploaded it:

 http://dropcanvas.com/ta7nu
 (First time I used this service, please give me feedback if this
 doesn't work for you)

 I see the "Beginning cold snapshot process" message, but I don't know
 why. Any help is appreciated!

 Regards,
 Eugen


 Zitat von Mohammed Naser :

 Would you be able to share the logs of a full snapshot run with the
> compute node in debug?
>
> Sent from my iPhone
>
> On Jan 12, 2017, at 7:47 AM, Eugen Block  wrote:
>>
>> That's strange, I also searched for this message, but nothing there.
>> I have debug logs enabled on compute node but I don't see anything
>> regarding ceph. No matter, what I do, my instance is always shutdown 
>> before
>> a snapshot is taken. What else can I try?
>>
>>
>> Zitat von John Petrini :
>>
>> Mohammed,
>>>
>>> It looks like you may be right. Just found the permissions issue in
>>> the
>>> nova log on the compute node.
>>>
>>> 4-e8f52e4fbcfb 691caf1c10354efab3e3c8ed61b7d89a
>>> 49bc5e5bf2684bd0948d9f94c7875027 - - -] Performing standard snapshot
>>> because direct snapshot failed: no write permission on storage pool
>>> images
>>>
>>> I'm going to test the change and will send an update you all with the
>>> results.
>>>
>>> Thank You,
>>>
>>> ___
>>>
>>> John Petrini
>>>
>>>
>>>
>>>
> Yes, we are also running Mitaka and I also read Sebastien Han's
 blogs ;-)

 our snapshots are not happening at the RBD level,

> they are being copied and uploaded to glance which takes up a lot
> of space
> and is very slow.
>
>
 Unfortunately, that's what we are experiencing, too. I don't know if
 there's something I missed in the nova configs or somewhere else,
 but I'm
 relieved that I'm not the only one :-)

 While writing this email I searched again and found something:

 https://specs.openstack.org/openstack/nova-specs/specs/mitak
 a/implemented/rbd-instance-snapshots.html

 https://review.openstack.org/#/c/205282/

 It seems to be implemented already, I'm looking for the config
 options to
 set. If you manage to get nova to make rbd snapshots, please let me
 know ;-)

 Regards,
 Eugen

Re: [Openstack] nova backup - instances unreachable

[Eugen Block]

Thanks,

for anyone interested in this issue, I filed a bug report:  
https://bugs.launchpad.net/nova/+bug/1656242


Regards,
Eugen


Zitat von Mohammed Naser :

It is likely because this has been tested with QEMU only. I think  
you might want to bring this up with the Nova team.


Sent from my iPhone


On Jan 12, 2017, at 11:28 AM, Eugen Block  wrote:

I'm not sure if this is the right spot, but I added some log  
statements into driver.py.

First, there's this if-block:

   if (self._host.has_min_version(MIN_LIBVIRT_LIVESNAPSHOT_VERSION,
  MIN_QEMU_LIVESNAPSHOT_VERSION,
  host.HV_DRIVER_QEMU)
and source_type not in ('lvm')
and not CONF.ephemeral_storage_encryption.enabled
and not CONF.workarounds.disable_libvirt_livesnapshot):
   live_snapshot = True
  [...]
   else:
   live_snapshot = False

And I know that it lands in the else-statement. Turns out that  
_host.has_min_version is "false", because of host.HV_DRIVER_QEMU.  
We are running on Xen hypervisors. So I tried it with  
host.HV_DRIVER_XEN and now nova-compute says:


[instance: 14b75237-7619-481f-9636-792b64d1be17] instance snapshotting
[instance: 14b75237-7619-481f-9636-792b64d1be17] Beginning live  
snapshot process


Now I'm waiting for the result, but at least the VM is still  
running, so it looks quite promising...


And there it is:

[instance: 14b75237-7619-481f-9636-792b64d1be17] Snapshot image  
upload complete


I'm testing the image now, and it works!

Now the question is, why is it defaulting to HV_DRIVER_QEMU and is  
it really necessary to change this directly in the code? Is there  
any other way?


Regards,
Eugen

Zitat von Eugen Block :


Yes, I truncated the file and uploaded it:

http://dropcanvas.com/ta7nu
(First time I used this service, please give me feedback if this  
doesn't work for you)


I see the "Beginning cold snapshot process" message, but I don't  
know why. Any help is appreciated!


Regards,
Eugen


Zitat von Mohammed Naser :

Would you be able to share the logs of a full snapshot run with  
the compute node in debug?


Sent from my iPhone


On Jan 12, 2017, at 7:47 AM, Eugen Block  wrote:

That's strange, I also searched for this message, but nothing  
there. I have debug logs enabled on compute node but I don't see  
anything regarding ceph. No matter, what I do, my instance is  
always shutdown before a snapshot is taken. What else can I try?



Zitat von John Petrini :


Mohammed,

It looks like you may be right. Just found the permissions issue in the
nova log on the compute node.

4-e8f52e4fbcfb 691caf1c10354efab3e3c8ed61b7d89a
49bc5e5bf2684bd0948d9f94c7875027 - - -] Performing standard snapshot
because direct snapshot failed: no write permission on storage  
pool images


I'm going to test the change and will send an update you all with the
results.

Thank You,

___

John Petrini





Yes, we are also running Mitaka and I also read Sebastien  
Han's blogs ;-)


our snapshots are not happening at the RBD level,
they are being copied and uploaded to glance which takes up a  
lot of space

and is very slow.



Unfortunately, that's what we are experiencing, too. I don't know if
there's something I missed in the nova configs or somewhere  
else, but I'm

relieved that I'm not the only one :-)

While writing this email I searched again and found something:

https://specs.openstack.org/openstack/nova-specs/specs/mitak
a/implemented/rbd-instance-snapshots.html

https://review.openstack.org/#/c/205282/

It seems to be implemented already, I'm looking for the config  
options to
set. If you manage to get nova to make rbd snapshots, please  
let me know ;-)


Regards,
Eugen



Zitat von John Petrini :

Hi Eugen,


Thanks for the response! That makes a lost of sense and is  
what I figured
was going on but I missed it in the documentation. We use  
Ceph as well and

I had considered doing the snapshots at the RBD level but I was hoping
there was someway to accomplish this via nova. I came across this
Sebastien
Han write-up that claims this functionality was added to Mitaka:
http://www.sebastien-han.fr/blog/2015/10/05/openstack-nova-
snapshots-on-ceph-rbd/

We are running Mitaka but our snapshots are not happening at the RBD
level,
they are being copied and uploaded to glance which takes up a  
lot of space

and is very slow.

Have you or anyone else implemented this in Mitaka? Other  
than Sebastian's

blog I haven't found any documentation on this.

Thank You,

___

John Petrini

On Wed, Jan 11, 2017 at 3:32 AM, Eugen Block  wrote:

Hi,


this seems to be exptected, the docs say:

"Shut down the source VM before you take the snapshot to  
ensure that all

data is flushed to disk."

So if the VM is not shut down, it's freezed to 

Re: [Openstack] nova backup - instances unreachable

[Mohammed Naser]

It is likely because this has been tested with QEMU only. I think you might 
want to bring this up with the Nova team. 

Sent from my iPhone

> On Jan 12, 2017, at 11:28 AM, Eugen Block  wrote:
> 
> I'm not sure if this is the right spot, but I added some log statements into 
> driver.py.
> First, there's this if-block:
> 
>if (self._host.has_min_version(MIN_LIBVIRT_LIVESNAPSHOT_VERSION,
>   MIN_QEMU_LIVESNAPSHOT_VERSION,
>   host.HV_DRIVER_QEMU)
> and source_type not in ('lvm')
> and not CONF.ephemeral_storage_encryption.enabled
> and not CONF.workarounds.disable_libvirt_livesnapshot):
>live_snapshot = True
>   [...]
>else:
>live_snapshot = False
> 
> And I know that it lands in the else-statement. Turns out that 
> _host.has_min_version is "false", because of host.HV_DRIVER_QEMU. We are 
> running on Xen hypervisors. So I tried it with host.HV_DRIVER_XEN and now 
> nova-compute says:
> 
> [instance: 14b75237-7619-481f-9636-792b64d1be17] instance snapshotting
> [instance: 14b75237-7619-481f-9636-792b64d1be17] Beginning live snapshot 
> process
> 
> Now I'm waiting for the result, but at least the VM is still running, so it 
> looks quite promising...
> 
> And there it is:
> 
> [instance: 14b75237-7619-481f-9636-792b64d1be17] Snapshot image upload 
> complete
> 
> I'm testing the image now, and it works!
> 
> Now the question is, why is it defaulting to HV_DRIVER_QEMU and is it really 
> necessary to change this directly in the code? Is there any other way?
> 
> Regards,
> Eugen
> 
> Zitat von Eugen Block :
> 
>> Yes, I truncated the file and uploaded it:
>> 
>> http://dropcanvas.com/ta7nu
>> (First time I used this service, please give me feedback if this doesn't 
>> work for you)
>> 
>> I see the "Beginning cold snapshot process" message, but I don't know why. 
>> Any help is appreciated!
>> 
>> Regards,
>> Eugen
>> 
>> 
>> Zitat von Mohammed Naser :
>> 
>>> Would you be able to share the logs of a full snapshot run with the compute 
>>> node in debug?
>>> 
>>> Sent from my iPhone
>>> 
 On Jan 12, 2017, at 7:47 AM, Eugen Block  wrote:
 
 That's strange, I also searched for this message, but nothing there. I 
 have debug logs enabled on compute node but I don't see anything regarding 
 ceph. No matter, what I do, my instance is always shutdown before a 
 snapshot is taken. What else can I try?
 
 
 Zitat von John Petrini :
 
> Mohammed,
> 
> It looks like you may be right. Just found the permissions issue in the
> nova log on the compute node.
> 
> 4-e8f52e4fbcfb 691caf1c10354efab3e3c8ed61b7d89a
> 49bc5e5bf2684bd0948d9f94c7875027 - - -] Performing standard snapshot
> because direct snapshot failed: no write permission on storage pool images
> 
> I'm going to test the change and will send an update you all with the
> results.
> 
> Thank You,
> 
> ___
> 
> John Petrini
> 
> 
> 
>>> 
>> Yes, we are also running Mitaka and I also read Sebastien Han's blogs ;-)
>> 
>> our snapshots are not happening at the RBD level,
>>> they are being copied and uploaded to glance which takes up a lot of 
>>> space
>>> and is very slow.
>>> 
>> 
>> Unfortunately, that's what we are experiencing, too. I don't know if
>> there's something I missed in the nova configs or somewhere else, but I'm
>> relieved that I'm not the only one :-)
>> 
>> While writing this email I searched again and found something:
>> 
>> https://specs.openstack.org/openstack/nova-specs/specs/mitak
>> a/implemented/rbd-instance-snapshots.html
>> 
>> https://review.openstack.org/#/c/205282/
>> 
>> It seems to be implemented already, I'm looking for the config options to
>> set. If you manage to get nova to make rbd snapshots, please let me know 
>> ;-)
>> 
>> Regards,
>> Eugen
>> 
>> 
>> 
>> Zitat von John Petrini :
>> 
>> Hi Eugen,
>>> 
>>> Thanks for the response! That makes a lost of sense and is what I 
>>> figured
>>> was going on but I missed it in the documentation. We use Ceph as well 
>>> and
>>> I had considered doing the snapshots at the RBD level but I was hoping
>>> there was someway to accomplish this via nova. I came across this
>>> Sebastien
>>> Han write-up that claims this functionality was added to Mitaka:
>>> http://www.sebastien-han.fr/blog/2015/10/05/openstack-nova-
>>> snapshots-on-ceph-rbd/
>>> 
>>> We are running Mitaka but our snapshots are not happening at the RBD
>>> level,
>>> they are being copied and uploaded to glance which takes up a lot of 
>>> space
>>> 

Re: [Openstack] nova backup - instances unreachable

[Eugen Block]

Yes, I truncated the file and uploaded it:

http://dropcanvas.com/ta7nu
(First time I used this service, please give me feedback if this  
doesn't work for you)


I see the "Beginning cold snapshot process" message, but I don't know  
why. Any help is appreciated!


Regards,
Eugen


Zitat von Mohammed Naser :

Would you be able to share the logs of a full snapshot run with the  
compute node in debug?


Sent from my iPhone


On Jan 12, 2017, at 7:47 AM, Eugen Block  wrote:

That's strange, I also searched for this message, but nothing  
there. I have debug logs enabled on compute node but I don't see  
anything regarding ceph. No matter, what I do, my instance is  
always shutdown before a snapshot is taken. What else can I try?



Zitat von John Petrini :


Mohammed,

It looks like you may be right. Just found the permissions issue in the
nova log on the compute node.

4-e8f52e4fbcfb 691caf1c10354efab3e3c8ed61b7d89a
49bc5e5bf2684bd0948d9f94c7875027 - - -] Performing standard snapshot
because direct snapshot failed: no write permission on storage pool images

I'm going to test the change and will send an update you all with the
results.

Thank You,

___

John Petrini






Yes, we are also running Mitaka and I also read Sebastien Han's blogs ;-)

our snapshots are not happening at the RBD level,
they are being copied and uploaded to glance which takes up a  
lot of space

and is very slow.



Unfortunately, that's what we are experiencing, too. I don't know if
there's something I missed in the nova configs or somewhere else, but I'm
relieved that I'm not the only one :-)

While writing this email I searched again and found something:

https://specs.openstack.org/openstack/nova-specs/specs/mitak
a/implemented/rbd-instance-snapshots.html

https://review.openstack.org/#/c/205282/

It seems to be implemented already, I'm looking for the config options to
set. If you manage to get nova to make rbd snapshots, please let  
me know ;-)


Regards,
Eugen



Zitat von John Petrini :

Hi Eugen,


Thanks for the response! That makes a lost of sense and is what I figured
was going on but I missed it in the documentation. We use Ceph  
as well and

I had considered doing the snapshots at the RBD level but I was hoping
there was someway to accomplish this via nova. I came across this
Sebastien
Han write-up that claims this functionality was added to Mitaka:
http://www.sebastien-han.fr/blog/2015/10/05/openstack-nova-
snapshots-on-ceph-rbd/

We are running Mitaka but our snapshots are not happening at the RBD
level,
they are being copied and uploaded to glance which takes up a  
lot of space

and is very slow.

Have you or anyone else implemented this in Mitaka? Other than  
Sebastian's

blog I haven't found any documentation on this.

Thank You,

___

John Petrini

On Wed, Jan 11, 2017 at 3:32 AM, Eugen Block  wrote:

Hi,


this seems to be exptected, the docs say:

"Shut down the source VM before you take the snapshot to ensure that all
data is flushed to disk."

So if the VM is not shut down, it's freezed to prevent data loss (I
guess). Depending on your storage backend, there are other ways to
perform
backups of your VMs.
We use Ceph as backend for nova, glance and cinder. Ceph stores the
disks,
images and volumes as Rados block device objects. We have a  
backup script

that creates snapshots of these RBDs, which are exported to our backup
drive. This way the running VM is not stopped or freezed, the user
doesn't
notice any issues. Unlike a nova snapshot, the rbd snapshot is created
immediately within a few seconds. After a successful backup the  
snapshots

are removed.

Hope this helps! If you are interested in Ceph, visit [1].

Regards,
Eugen

[1] http://docs.ceph.com/docs/giant/start/intro/


Zitat von John Petrini :


Hello,



I've just started experimenting with nova backup and discovered that
there
is a period of time during the snapshot where the instance becomes
unreachable. Is this behavior expected during a live snapshot? Is there
any
way to prevent this?

___

John Petrini





--
Eugen Block voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail  : ebl...@nde.ag

   Vorsitzende des Aufsichtsrates: Angelika Mozdzen
 Sitz und Registergericht: Hamburg, HRB 90934
 Vorstand: Jens-U. Mozdzen
  USt-IdNr. DE 814 013 983


___
Mailing list: http://lists.openstack.org/cgi
-bin/mailman/listinfo/openstac
k
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi
-bin/mailman/listinfo/openstac
k





--
Eugen Block voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
Postfach 61 03 

Re: [Openstack] nova backup - instances unreachable

[Mohammed Naser]

Would you be able to share the logs of a full snapshot run with the compute 
node in debug?

Sent from my iPhone

> On Jan 12, 2017, at 7:47 AM, Eugen Block  wrote:
> 
> That's strange, I also searched for this message, but nothing there. I have 
> debug logs enabled on compute node but I don't see anything regarding ceph. 
> No matter, what I do, my instance is always shutdown before a snapshot is 
> taken. What else can I try?
> 
> 
> Zitat von John Petrini :
> 
>> Mohammed,
>> 
>> It looks like you may be right. Just found the permissions issue in the
>> nova log on the compute node.
>> 
>> 4-e8f52e4fbcfb 691caf1c10354efab3e3c8ed61b7d89a
>> 49bc5e5bf2684bd0948d9f94c7875027 - - -] Performing standard snapshot
>> because direct snapshot failed: no write permission on storage pool images
>> 
>> I'm going to test the change and will send an update you all with the
>> results.
>> 
>> Thank You,
>> 
>> ___
>> 
>> John Petrini
>> 
>> 
>> 
 
>>> Yes, we are also running Mitaka and I also read Sebastien Han's blogs ;-)
>>> 
>>> our snapshots are not happening at the RBD level,
 they are being copied and uploaded to glance which takes up a lot of space
 and is very slow.
 
>>> 
>>> Unfortunately, that's what we are experiencing, too. I don't know if
>>> there's something I missed in the nova configs or somewhere else, but I'm
>>> relieved that I'm not the only one :-)
>>> 
>>> While writing this email I searched again and found something:
>>> 
>>> https://specs.openstack.org/openstack/nova-specs/specs/mitak
>>> a/implemented/rbd-instance-snapshots.html
>>> 
>>> https://review.openstack.org/#/c/205282/
>>> 
>>> It seems to be implemented already, I'm looking for the config options to
>>> set. If you manage to get nova to make rbd snapshots, please let me know ;-)
>>> 
>>> Regards,
>>> Eugen
>>> 
>>> 
>>> 
>>> Zitat von John Petrini :
>>> 
>>> Hi Eugen,
 
 Thanks for the response! That makes a lost of sense and is what I figured
 was going on but I missed it in the documentation. We use Ceph as well and
 I had considered doing the snapshots at the RBD level but I was hoping
 there was someway to accomplish this via nova. I came across this
 Sebastien
 Han write-up that claims this functionality was added to Mitaka:
 http://www.sebastien-han.fr/blog/2015/10/05/openstack-nova-
 snapshots-on-ceph-rbd/
 
 We are running Mitaka but our snapshots are not happening at the RBD
 level,
 they are being copied and uploaded to glance which takes up a lot of space
 and is very slow.
 
 Have you or anyone else implemented this in Mitaka? Other than Sebastian's
 blog I haven't found any documentation on this.
 
 Thank You,
 
 ___
 
 John Petrini
 
 On Wed, Jan 11, 2017 at 3:32 AM, Eugen Block  wrote:
 
 Hi,
> 
> this seems to be exptected, the docs say:
> 
> "Shut down the source VM before you take the snapshot to ensure that all
> data is flushed to disk."
> 
> So if the VM is not shut down, it's freezed to prevent data loss (I
> guess). Depending on your storage backend, there are other ways to
> perform
> backups of your VMs.
> We use Ceph as backend for nova, glance and cinder. Ceph stores the
> disks,
> images and volumes as Rados block device objects. We have a backup script
> that creates snapshots of these RBDs, which are exported to our backup
> drive. This way the running VM is not stopped or freezed, the user
> doesn't
> notice any issues. Unlike a nova snapshot, the rbd snapshot is created
> immediately within a few seconds. After a successful backup the snapshots
> are removed.
> 
> Hope this helps! If you are interested in Ceph, visit [1].
> 
> Regards,
> Eugen
> 
> [1] http://docs.ceph.com/docs/giant/start/intro/
> 
> 
> Zitat von John Petrini :
> 
> 
> Hello,
> 
>> 
>> I've just started experimenting with nova backup and discovered that
>> there
>> is a period of time during the snapshot where the instance becomes
>> unreachable. Is this behavior expected during a live snapshot? Is there
>> any
>> way to prevent this?
>> 
>> ___
>> 
>> John Petrini
>> 
>> 
> 
> 
> --
> Eugen Block voice   : +49-40-559 51 75
> NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
> Postfach 61 03 15
> D-22423 Hamburg e-mail  : ebl...@nde.ag
> 
>Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>  Sitz und Registergericht: Hamburg, HRB 90934
>  Vorstand: Jens-U. Mozdzen
>   USt-IdNr. DE 814 013 983
> 
> 
> ___
> 

Re: [Openstack] nova backup - instances unreachable

[Eugen Block]

That's strange, I also searched for this message, but nothing there. I  
have debug logs enabled on compute node but I don't see anything  
regarding ceph. No matter, what I do, my instance is always shutdown  
before a snapshot is taken. What else can I try?



Zitat von John Petrini :


Mohammed,

It looks like you may be right. Just found the permissions issue in the
nova log on the compute node.

4-e8f52e4fbcfb 691caf1c10354efab3e3c8ed61b7d89a
49bc5e5bf2684bd0948d9f94c7875027 - - -] Performing standard snapshot
because direct snapshot failed: no write permission on storage pool images

I'm going to test the change and will send an update you all with the
results.

Thank You,

___

John Petrini






Yes, we are also running Mitaka and I also read Sebastien Han's blogs ;-)

our snapshots are not happening at the RBD level,

they are being copied and uploaded to glance which takes up a lot of space
and is very slow.



Unfortunately, that's what we are experiencing, too. I don't know if
there's something I missed in the nova configs or somewhere else, but I'm
relieved that I'm not the only one :-)

While writing this email I searched again and found something:

https://specs.openstack.org/openstack/nova-specs/specs/mitak
a/implemented/rbd-instance-snapshots.html

https://review.openstack.org/#/c/205282/

It seems to be implemented already, I'm looking for the config options to
set. If you manage to get nova to make rbd snapshots, please let me know ;-)

Regards,
Eugen



Zitat von John Petrini :

Hi Eugen,


Thanks for the response! That makes a lost of sense and is what I figured
was going on but I missed it in the documentation. We use Ceph as well and
I had considered doing the snapshots at the RBD level but I was hoping
there was someway to accomplish this via nova. I came across this
Sebastien
Han write-up that claims this functionality was added to Mitaka:
http://www.sebastien-han.fr/blog/2015/10/05/openstack-nova-
snapshots-on-ceph-rbd/

We are running Mitaka but our snapshots are not happening at the RBD
level,
they are being copied and uploaded to glance which takes up a lot of space
and is very slow.

Have you or anyone else implemented this in Mitaka? Other than Sebastian's
blog I haven't found any documentation on this.

Thank You,

___

John Petrini

On Wed, Jan 11, 2017 at 3:32 AM, Eugen Block  wrote:

Hi,


this seems to be exptected, the docs say:

"Shut down the source VM before you take the snapshot to ensure that all
data is flushed to disk."

So if the VM is not shut down, it's freezed to prevent data loss (I
guess). Depending on your storage backend, there are other ways to
perform
backups of your VMs.
We use Ceph as backend for nova, glance and cinder. Ceph stores the
disks,
images and volumes as Rados block device objects. We have a backup script
that creates snapshots of these RBDs, which are exported to our backup
drive. This way the running VM is not stopped or freezed, the user
doesn't
notice any issues. Unlike a nova snapshot, the rbd snapshot is created
immediately within a few seconds. After a successful backup the snapshots
are removed.

Hope this helps! If you are interested in Ceph, visit [1].

Regards,
Eugen

[1] http://docs.ceph.com/docs/giant/start/intro/


Zitat von John Petrini :


Hello,



I've just started experimenting with nova backup and discovered that
there
is a period of time during the snapshot where the instance becomes
unreachable. Is this behavior expected during a live snapshot? Is there
any
way to prevent this?

___

John Petrini





--
Eugen Block voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail  : ebl...@nde.ag

Vorsitzende des Aufsichtsrates: Angelika Mozdzen
  Sitz und Registergericht: Hamburg, HRB 90934
  Vorstand: Jens-U. Mozdzen
   USt-IdNr. DE 814 013 983


___
Mailing list: http://lists.openstack.org/cgi
-bin/mailman/listinfo/openstac
k
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi
-bin/mailman/listinfo/openstac
k





--
Eugen Block voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail  : ebl...@nde.ag

Vorsitzende des Aufsichtsrates: Angelika Mozdzen
  Sitz und Registergericht: Hamburg, HRB 90934
  Vorstand: Jens-U. Mozdzen
   USt-IdNr. DE 814 013 983






--
Eugen Block voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail  : ebl...@nde.ag

Vorsitzende des Aufsichtsrates: 

Re: [Openstack] nova backup - instances unreachable

[John Petrini]

Mohammed,

It looks like you may be right. Just found the permissions issue in the
nova log on the compute node.

4-e8f52e4fbcfb 691caf1c10354efab3e3c8ed61b7d89a
49bc5e5bf2684bd0948d9f94c7875027 - - -] Performing standard snapshot
because direct snapshot failed: no write permission on storage pool images

I'm going to test the change and will send an update you all with the
results.

Thank You,

___

John Petrini



>>
> Yes, we are also running Mitaka and I also read Sebastien Han's blogs ;-)
>
> our snapshots are not happening at the RBD level,
>> they are being copied and uploaded to glance which takes up a lot of space
>> and is very slow.
>>
>
> Unfortunately, that's what we are experiencing, too. I don't know if
> there's something I missed in the nova configs or somewhere else, but I'm
> relieved that I'm not the only one :-)
>
> While writing this email I searched again and found something:
>
> https://specs.openstack.org/openstack/nova-specs/specs/mitak
> a/implemented/rbd-instance-snapshots.html
>
> https://review.openstack.org/#/c/205282/
>
> It seems to be implemented already, I'm looking for the config options to
> set. If you manage to get nova to make rbd snapshots, please let me know ;-)
>
> Regards,
> Eugen
>
>
>
> Zitat von John Petrini :
>
> Hi Eugen,
>>
>> Thanks for the response! That makes a lost of sense and is what I figured
>> was going on but I missed it in the documentation. We use Ceph as well and
>> I had considered doing the snapshots at the RBD level but I was hoping
>> there was someway to accomplish this via nova. I came across this
>> Sebastien
>> Han write-up that claims this functionality was added to Mitaka:
>> http://www.sebastien-han.fr/blog/2015/10/05/openstack-nova-
>> snapshots-on-ceph-rbd/
>>
>> We are running Mitaka but our snapshots are not happening at the RBD
>> level,
>> they are being copied and uploaded to glance which takes up a lot of space
>> and is very slow.
>>
>> Have you or anyone else implemented this in Mitaka? Other than Sebastian's
>> blog I haven't found any documentation on this.
>>
>> Thank You,
>>
>> ___
>>
>> John Petrini
>>
>> On Wed, Jan 11, 2017 at 3:32 AM, Eugen Block  wrote:
>>
>> Hi,
>>>
>>> this seems to be exptected, the docs say:
>>>
>>> "Shut down the source VM before you take the snapshot to ensure that all
>>> data is flushed to disk."
>>>
>>> So if the VM is not shut down, it's freezed to prevent data loss (I
>>> guess). Depending on your storage backend, there are other ways to
>>> perform
>>> backups of your VMs.
>>> We use Ceph as backend for nova, glance and cinder. Ceph stores the
>>> disks,
>>> images and volumes as Rados block device objects. We have a backup script
>>> that creates snapshots of these RBDs, which are exported to our backup
>>> drive. This way the running VM is not stopped or freezed, the user
>>> doesn't
>>> notice any issues. Unlike a nova snapshot, the rbd snapshot is created
>>> immediately within a few seconds. After a successful backup the snapshots
>>> are removed.
>>>
>>> Hope this helps! If you are interested in Ceph, visit [1].
>>>
>>> Regards,
>>> Eugen
>>>
>>> [1] http://docs.ceph.com/docs/giant/start/intro/
>>>
>>>
>>> Zitat von John Petrini :
>>>
>>>
>>> Hello,
>>>

 I've just started experimenting with nova backup and discovered that
 there
 is a period of time during the snapshot where the instance becomes
 unreachable. Is this behavior expected during a live snapshot? Is there
 any
 way to prevent this?

 ___

 John Petrini


>>>
>>>
>>> --
>>> Eugen Block voice   : +49-40-559 51 75
>>> NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
>>> Postfach 61 03 15
>>> D-22423 Hamburg e-mail  : ebl...@nde.ag
>>>
>>> Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>   Sitz und Registergericht: Hamburg, HRB 90934
>>>   Vorstand: Jens-U. Mozdzen
>>>USt-IdNr. DE 814 013 983
>>>
>>>
>>> ___
>>> Mailing list: http://lists.openstack.org/cgi
>>> -bin/mailman/listinfo/openstac
>>> k
>>> Post to : openstack@lists.openstack.org
>>> Unsubscribe : http://lists.openstack.org/cgi
>>> -bin/mailman/listinfo/openstac
>>> k
>>>
>>>
>
>
> --
> Eugen Block voice   : +49-40-559 51 75
> NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
> Postfach 61 03 15
> D-22423 Hamburg e-mail  : ebl...@nde.ag
>
> Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>   Sitz und Registergericht: Hamburg, HRB 90934
>   Vorstand: Jens-U. Mozdzen
>USt-IdNr. DE 814 013 983
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org

Re: [Openstack] nova backup - instances unreachable

[Mohammed Naser]

Hi John,

It just works for us with Mitaka.  You might be running with issues regarding 
permissions where the Nova user might not be able to write to the images pool. 

Turn debug on in your nova compute and snapshot a machine on it, you'll see the 
logs and if it's turning it off, it's probably because your rbd snapshot failed 
(in my experience) and it fell back to the older snapshot process. 

Thanks
Mohammed 

Sent from my iPhone

> On Jan 11, 2017, at 9:22 AM, John Petrini  wrote:
> 
> Hi Eugen,
> 
> Thanks for the response! That makes a lost of sense and is what I figured was 
> going on but I missed it in the documentation. We use Ceph as well and I had 
> considered doing the snapshots at the RBD level but I was hoping there was 
> someway to accomplish this via nova. I came across this Sebastien Han 
> write-up that claims this functionality was added to Mitaka: 
> http://www.sebastien-han.fr/blog/2015/10/05/openstack-nova-snapshots-on-ceph-rbd/
> 
> We are running Mitaka but our snapshots are not happening at the RBD level, 
> they are being copied and uploaded to glance which takes up a lot of space 
> and is very slow.
> 
> Have you or anyone else implemented this in Mitaka? Other than Sebastian's 
> blog I haven't found any documentation on this.
> 
> Thank You,
> 
> ___
> 
> John Petrini
> 
>> On Wed, Jan 11, 2017 at 3:32 AM, Eugen Block  wrote:
>> Hi,
>> 
>> this seems to be exptected, the docs say:
>> 
>> "Shut down the source VM before you take the snapshot to ensure that all 
>> data is flushed to disk."
>> 
>> So if the VM is not shut down, it's freezed to prevent data loss (I guess). 
>> Depending on your storage backend, there are other ways to perform backups 
>> of your VMs.
>> We use Ceph as backend for nova, glance and cinder. Ceph stores the disks, 
>> images and volumes as Rados block device objects. We have a backup script 
>> that creates snapshots of these RBDs, which are exported to our backup 
>> drive. This way the running VM is not stopped or freezed, the user doesn't 
>> notice any issues. Unlike a nova snapshot, the rbd snapshot is created 
>> immediately within a few seconds. After a successful backup the snapshots 
>> are removed.
>> 
>> Hope this helps! If you are interested in Ceph, visit [1].
>> 
>> Regards,
>> Eugen
>> 
>> [1] http://docs.ceph.com/docs/giant/start/intro/
>> 
>> 
>> Zitat von John Petrini :
>> 
>> 
>>> Hello,
>>> 
>>> I've just started experimenting with nova backup and discovered that there
>>> is a period of time during the snapshot where the instance becomes
>>> unreachable. Is this behavior expected during a live snapshot? Is there any
>>> way to prevent this?
>>> 
>>> ___
>>> 
>>> John Petrini
>> 
>> 
>> 
>> -- 
>> Eugen Block voice   : +49-40-559 51 75
>> NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
>> Postfach 61 03 15
>> D-22423 Hamburg e-mail  : ebl...@nde.ag
>> 
>> Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>   Sitz und Registergericht: Hamburg, HRB 90934
>>   Vorstand: Jens-U. Mozdzen
>>USt-IdNr. DE 814 013 983
>> 
>> 
>> ___
>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] nova backup - instances unreachable

[Eugen Block]

Have you or anyone else implemented this in Mitaka?


Yes, we are also running Mitaka and I also read Sebastien Han's blogs ;-)


our snapshots are not happening at the RBD level,
they are being copied and uploaded to glance which takes up a lot of space
and is very slow.


Unfortunately, that's what we are experiencing, too. I don't know if  
there's something I missed in the nova configs or somewhere else, but  
I'm relieved that I'm not the only one :-)


While writing this email I searched again and found something:

https://specs.openstack.org/openstack/nova-specs/specs/mitaka/implemented/rbd-instance-snapshots.html

https://review.openstack.org/#/c/205282/

It seems to be implemented already, I'm looking for the config options  
to set. If you manage to get nova to make rbd snapshots, please let me  
know ;-)


Regards,
Eugen


Zitat von John Petrini :


Hi Eugen,

Thanks for the response! That makes a lost of sense and is what I figured
was going on but I missed it in the documentation. We use Ceph as well and
I had considered doing the snapshots at the RBD level but I was hoping
there was someway to accomplish this via nova. I came across this Sebastien
Han write-up that claims this functionality was added to Mitaka:
http://www.sebastien-han.fr/blog/2015/10/05/openstack-nova-snapshots-on-ceph-rbd/

We are running Mitaka but our snapshots are not happening at the RBD level,
they are being copied and uploaded to glance which takes up a lot of space
and is very slow.

Have you or anyone else implemented this in Mitaka? Other than Sebastian's
blog I haven't found any documentation on this.

Thank You,

___

John Petrini

On Wed, Jan 11, 2017 at 3:32 AM, Eugen Block  wrote:


Hi,

this seems to be exptected, the docs say:

"Shut down the source VM before you take the snapshot to ensure that all
data is flushed to disk."

So if the VM is not shut down, it's freezed to prevent data loss (I
guess). Depending on your storage backend, there are other ways to perform
backups of your VMs.
We use Ceph as backend for nova, glance and cinder. Ceph stores the disks,
images and volumes as Rados block device objects. We have a backup script
that creates snapshots of these RBDs, which are exported to our backup
drive. This way the running VM is not stopped or freezed, the user doesn't
notice any issues. Unlike a nova snapshot, the rbd snapshot is created
immediately within a few seconds. After a successful backup the snapshots
are removed.

Hope this helps! If you are interested in Ceph, visit [1].

Regards,
Eugen

[1] http://docs.ceph.com/docs/giant/start/intro/


Zitat von John Petrini :


Hello,


I've just started experimenting with nova backup and discovered that there
is a period of time during the snapshot where the instance becomes
unreachable. Is this behavior expected during a live snapshot? Is there
any
way to prevent this?

___

John Petrini





--
Eugen Block voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail  : ebl...@nde.ag

Vorsitzende des Aufsichtsrates: Angelika Mozdzen
  Sitz und Registergericht: Hamburg, HRB 90934
  Vorstand: Jens-U. Mozdzen
   USt-IdNr. DE 814 013 983


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
k
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
k





--
Eugen Block voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail  : ebl...@nde.ag

Vorsitzende des Aufsichtsrates: Angelika Mozdzen
  Sitz und Registergericht: Hamburg, HRB 90934
  Vorstand: Jens-U. Mozdzen
   USt-IdNr. DE 814 013 983


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] nova backup - instances unreachable

[Eugen Block]

Hi,

this seems to be exptected, the docs say:

"Shut down the source VM before you take the snapshot to ensure that  
all data is flushed to disk."


So if the VM is not shut down, it's freezed to prevent data loss (I  
guess). Depending on your storage backend, there are other ways to  
perform backups of your VMs.
We use Ceph as backend for nova, glance and cinder. Ceph stores the  
disks, images and volumes as Rados block device objects. We have a  
backup script that creates snapshots of these RBDs, which are exported  
to our backup drive. This way the running VM is not stopped or  
freezed, the user doesn't notice any issues. Unlike a nova snapshot,  
the rbd snapshot is created immediately within a few seconds. After a  
successful backup the snapshots are removed.


Hope this helps! If you are interested in Ceph, visit [1].

Regards,
Eugen

[1] http://docs.ceph.com/docs/giant/start/intro/


Zitat von John Petrini :


Hello,

I've just started experimenting with nova backup and discovered that there
is a period of time during the snapshot where the instance becomes
unreachable. Is this behavior expected during a live snapshot? Is there any
way to prevent this?

___

John Petrini




--
Eugen Block voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail  : ebl...@nde.ag

Vorsitzende des Aufsichtsrates: Angelika Mozdzen
  Sitz und Registergericht: Hamburg, HRB 90934
  Vorstand: Jens-U. Mozdzen
   USt-IdNr. DE 814 013 983


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack