Re: [ovirt-users] Convert local storage domain to shared

[Gianluca Cecchi]

Il 03 Dic 2017 07:42, "Yedidyah Bar David"  ha scritto:

Hi Gianluca,

Thanks for another great post!

Any chance you'd like to convert it to a blog/article on ovirt.org?


Thanks for your compliments Didi, much appreciated.
I can try to do something at the end of the week
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Convert local storage domain to shared

[Yedidyah Bar David]

Hi Gianluca,

Thanks for another great post!

Any chance you'd like to convert it to a blog/article on ovirt.org?

On Fri, Dec 1, 2017 at 2:25 AM, Gianluca Cecchi
 wrote:
> On Wed, Nov 29, 2017 at 6:49 PM, Demeter Tibor  wrote:
>>
>> Hi,
>>
>> Yes, I understand what do you talk about. It isn't too safe..:(
>> We have terrabytes under that VM.
>> I could make a downtime at most for eight hours (maybe), but meanwhile I
>> have to copy 3 TB of vdisks. Firstly I need export (with a gigabit nic) to
>> export domain, and back under 10gbe nic.
>> I don't know how is enough this.
>>
>> Thanks
>>
>> Tibor
>
>
> Hi Tibor,
> I'm in shortage of time these days, but I have to admit your problem was so
> intriguing that I couldn't resist and I decided to try and reproduce it.
> All happened on my laptop with Fedora 26 (time to upgrade? not enough
> time... ;-)
>
> So this is the test environment of all vms inside virt-manager:
>
> 1) Create 3.5.6 environment
>
> - CentOS 6.6 VM (this was the iso I had at hand...) with hostname c6engine35
> where I installed oVirt 3.5.6 as engine
> - CentOS 6.6 VM with hostname c6rhv35 (sorry for the rhv in the name but
> these weeks I'm also working on it so it came out quite naturally...) were I
> installed the Hypervisor of 3.5.6 repo
>
> I created a local DC on top of a directory of the hypervisor (/ldomain)
> I created a CentOS 6.6 VM in this storage domain with a 4Gb disk
>
> 2) Detach the local domain from DC
>
> HERE YOUR THEORETICAL DOWNTIME BEGINS
>
> To do so I powered off the test VM and created a fake further local domain
> based on another directory of c6rhv35
> Then put into maintenance the local domain to be imported in 4.1
> The fake local domain becomes the master.
> Detach the local domain.
>
>
> 3) Create 4.1.7 environment (in your case it is already there..)
> - CentOS 7.4 VM with hostname c7engine41 where I installed oVirt 4.1.7 as
> engine
> - CentOS 7.4 VM with hostname c7rhv41 were I installed the Hypervisor of
> 4.1.7 repo
>
> I created a shared DC NFSDC with a cluster NFSCL
> To speed things I exported a directory from the engine and used it to create
> an NFS storage domain (DATANFS) for the 4.1 host and activated it
>
> 4) Shutdown 3.5 environment and start/configure the 3.5 hypervisor to export
> its previously local storage domain directory
>
> Start c6rhv35 in single user mode
> chkconfig service_name off
>
> for this service_name:
> ebtables ip6tables iptables libvirt-guests libvirtd momd numad sanlock
> supervdsmd vdsmd wdmd
>
> reboot
> create an entry in /etc/exports
>
> /ldomain c7rhv41.localdomain.local(rw)
>
> service nfs start
>
> set up accordingly the /etc/hosts of the servers involved so that all know
> all...
>
> 5) import domain in 4.1
> Select Storage -> Import domain and put
>
> c6rhv35.localdomain.local:/ldomain
>
> You will get a warning about it being already part of another DC:
> https://drive.google.com/file/d/1HjFZhW6fCkasPak0jQH5k49Bdsg1NLSN/view?usp=sharing
>
> Approve operation and you arrive here:
> https://drive.google.com/file/d/10d1ea0TbPCZhoaAf7br5IVqnvZx0LzSu/view?usp=sharing
>
> Activate the domain and you arrive here:
> https://drive.google.com/file/d/1-4sMfVVj5WyaglPI8zhWUsdJqkVkxzAT/view?usp=sharing
>
> Now you can proceed importing your VMs; in my case only the testvm
> Select he imported storage domain and then the "VM Import" tab; select the
> VM and "Import":
>
> https://drive.google.com/file/d/18yjPvoHjTw6mOhUrlHJ2RpsdPph4qBxL/view?usp=sharing
>
> https://drive.google.com/file/d/1CrCzVUYC3vI4aQ2ly83b3uAQ3QQh1xhm/view?usp=sharing
>
> Note that it is an immediate operation, and not depending on the size of the
> disks of the VM itself
> At the end you get your VM imported; here details:
>
> https://drive.google.com/file/d/1W00TpIKAQ7cWUit_tLIQkm30wj5j56AN/view?usp=sharing
> https://drive.google.com/file/d/1qq7sZV2vwapRRdjbi21Z43OOBM0m2NuY/view?usp=sharing
>
> While you import, you can then gradually start your VMs, so that your
> downtime becomes partial and not total
> https://drive.google.com/file/d/1kwrzSJBXISC0wBTtZIpdh3yJdA0g3G0A/view?usp=sharing
>
> When you have started all your imported VMs, YOUR THEORETICAL DOWNTIME ENDS
>
> Your VMS are now running on your old local storage, exported from your old
> 3.5 host to your new 4.1 hosts via NFS
>
> You can now execute live storage migration of your disks one by one to the
> desired 4.1 storage domain:
> https://drive.google.com/file/d/1p6OOgDBbOFFGgy3uuWT-V8VnCMaxk4iP/view?usp=sharing
>
> and at the end of the move
> https://drive.google.com/file/d/1dUuKQQxI0r4Bhz-N0TmRcsnjwQl1bwU6/view?usp=sharing
>
> Obviously there are many caveats in a real environment such as:
>
> - actual oVirt origin and target version could differ from mine and behavior
> be different
> - network visibility between the two oVirt environments
> - layout of the logical networks of the two oVirt environments: when you
> import you could need to 

Re: [ovirt-users] Convert local storage domain to shared

[Gianluca Cecchi]

On Wed, Nov 29, 2017 at 6:49 PM, Demeter Tibor  wrote:

> Hi,
>
> Yes, I understand what do you talk about. It isn't too safe..:(
> We have terrabytes under that VM.
> I could make a downtime at most for eight hours (maybe), but meanwhile I
> have to copy 3 TB of vdisks. Firstly I need export (with a gigabit nic) to
> export domain, and back under 10gbe nic.
> I don't know how is enough this.
>
> Thanks
>
> Tibor


Hi Tibor,
I'm in shortage of time these days, but I have to admit your problem was so
intriguing that I couldn't resist and I decided to try and reproduce it.
All happened on my laptop with Fedora 26 (time to upgrade? not enough
time... ;-)

So this is the test environment of all vms inside virt-manager:

1) Create 3.5.6 environment

- CentOS 6.6 VM (this was the iso I had at hand...) with hostname
c6engine35 where I installed oVirt 3.5.6 as engine
- CentOS 6.6 VM with hostname c6rhv35 (sorry for the rhv in the name but
these weeks I'm also working on it so it came out quite naturally...) were
I installed the Hypervisor of 3.5.6 repo

I created a local DC on top of a directory of the hypervisor (/ldomain)
I created a CentOS 6.6 VM in this storage domain with a 4Gb disk

2) Detach the local domain from DC

HERE YOUR THEORETICAL DOWNTIME BEGINS

To do so I powered off the test VM and created a fake further local domain
based on another directory of c6rhv35
Then put into maintenance the local domain to be imported in 4.1
The fake local domain becomes the master.
Detach the local domain.


3) Create 4.1.7 environment (in your case it is already there..)
- CentOS 7.4 VM with hostname c7engine41 where I installed oVirt 4.1.7 as
engine
- CentOS 7.4 VM with hostname c7rhv41 were I installed the Hypervisor of
4.1.7 repo

I created a shared DC NFSDC with a cluster NFSCL
To speed things I exported a directory from the engine and used it to
create an NFS storage domain (DATANFS) for the 4.1 host and activated it

4) Shutdown 3.5 environment and start/configure the 3.5 hypervisor to
export its previously local storage domain directory

Start c6rhv35 in single user mode
chkconfig service_name off

for this service_name:
ebtables ip6tables iptables libvirt-guests libvirtd momd numad sanlock
supervdsmd vdsmd wdmd

reboot
create an entry in /etc/exports

/ldomain c7rhv41.localdomain.local(rw)

service nfs start

set up accordingly the /etc/hosts of the servers involved so that all know
all...

5) import domain in 4.1
Select Storage -> Import domain and put

c6rhv35.localdomain.local:/ldomain

You will get a warning about it being already part of another DC:
https://drive.google.com/file/d/1HjFZhW6fCkasPak0jQH5k49Bdsg1NLSN/view?usp=sharing

Approve operation and you arrive here:
https://drive.google.com/file/d/10d1ea0TbPCZhoaAf7br5IVqnvZx0LzSu/view?usp=sharing

Activate the domain and you arrive here:
https://drive.google.com/file/d/1-4sMfVVj5WyaglPI8zhWUsdJqkVkxzAT/view?usp=sharing

Now you can proceed importing your VMs; in my case only the testvm
Select he imported storage domain and then the "VM Import" tab; select the
VM and "Import":

https://drive.google.com/file/d/18yjPvoHjTw6mOhUrlHJ2RpsdPph4qBxL/view?usp=sharing

https://drive.google.com/file/d/1CrCzVUYC3vI4aQ2ly83b3uAQ3QQh1xhm/view?usp=sharing

Note that it is an immediate operation, and not depending on the size of
the disks of the VM itself
At the end you get your VM imported; here details:

https://drive.google.com/file/d/1W00TpIKAQ7cWUit_tLIQkm30wj5j56AN/view?usp=sharing
https://drive.google.com/file/d/1qq7sZV2vwapRRdjbi21Z43OOBM0m2NuY/view?usp=sharing

While you import, you can then gradually start your VMs, so that your
downtime becomes partial and not total
https://drive.google.com/file/d/1kwrzSJBXISC0wBTtZIpdh3yJdA0g3G0A/view?usp=sharing

When you have started all your imported VMs, YOUR THEORETICAL DOWNTIME ENDS

Your VMS are now running on your old local storage, exported from your old
3.5 host to your new 4.1 hosts via NFS

You can now execute live storage migration of your disks one by one to the
desired 4.1 storage domain:
https://drive.google.com/file/d/1p6OOgDBbOFFGgy3uuWT-V8VnCMaxk4iP/view?usp=sharing

and at the end of the move
https://drive.google.com/file/d/1dUuKQQxI0r4Bhz-N0TmRcsnjwQl1bwU6/view?usp=sharing

Obviously there are many caveats in a real environment such as:

- actual oVirt origin and target version could differ from mine and
behavior be different
- network visibility between the two oVirt environments
- layout of the logical networks of the two oVirt environments: when you
import you could need to change logical network and have conflicting MACS:
in my test scenario it was all on ovirtmgmt with the same macs range
- live storage migration of TB of disks.. not tested yet (by me at
least)
- other things that don't come to mind right now

HIH,
Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Convert local storage domain to shared

[Karli Sjöberg]

Den 29 nov. 2017 18:49 skrev Demeter Tibor :Hi,Yes, I understand what do you talk about. It isn't too safe..:(We have terrabytes under that VM.I could make a downtime at most for eight hours (maybe), but meanwhile I have to copy 3 TB of vdisks. Firstly I need export (with a gigabit nic) to export domain, and back under 10gbe nic.I don't know how is enough this.Well, just counting the numbers, let's start with the optimistic approach and say that you can move 100 MB/s for 8 hours:100*60*60*8=288 MBAnd then just divide that with 1024*1024 to get to Tera:288%(1024^2)=2.74658203125 TBSo roughly 2.7 TB in 8 hours, and that's very optimistic! If you're more pessimistic, adjust the number of MB you think (or better yet, tested) that you'll be able to send per second to get a more accurate answer.The question is how much you can do without any downtime, I don't know myself, but the devs should:@devsIs it possible to do live exports? I mean to keep exporting and just sync the delta? If not, that would be an awesome RFE, since it would drastically reduce the downtime for these kinds of operations./KThanksTibor- 2017. nov.. 29., 18:26, Christopher Cox c...@endlessnow.com írta:> On 11/29/2017 09:39 AM, Demeter Tibor wrote: Dear Users, We have an old ovirt3.5 install with a local and a shared clusters. Meanwhile we>> created a new data center, that based on 4.1 and it use only shared>> infrastructure.>> I would like to migrate an big VM from the old local datacenter to our new, but>> I don't have enough downtime. Is it possible to convert the old local storage to shared (by share via NFS) and>> attach that as new storage domain to the new cluster?>> I just want to import VM and copy (while running) with live storage migration>> function. I know, the official way for move vms between ovirt clusters is the export>> domain, but it has very big disks. What can I do?> > Just my opinion, but if you don't figure out a way to have occasional downtime,> you'll probably pay the price with unplanned downtime eventually (and it could> be painful).> > Define "large disks"?  Terabytes?> > I know for a fact that if you don't have good network segmentation that live> migrations of large disks can be very problematic.  And I'm not talking about> what you're wanting to do.  I'm just talking about storage migration.> > We successfully migrated hundreds of VMs from a 3.4 to a 3.6 (on new blades and> storage) last year over time using the NFS export domain method.> > If storage is the same across DC's, you might be able to shortcut this with> minimal downtime, but I'm pretty sure there will be some downtime.> > I've seen large storage migrations render entire nodes offline (not nice) due to> non-isolated paths or QoS.> > > > ___> Users mailing list> Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users___Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Convert local storage domain to shared

[Demeter Tibor]

Hi,

Yes, I understand what do you talk about. It isn't too safe..:(
We have terrabytes under that VM.
I could make a downtime at most for eight hours (maybe), but meanwhile I have 
to copy 3 TB of vdisks. Firstly I need export (with a gigabit nic) to export 
domain, and back under 10gbe nic.
I don't know how is enough this.

Thanks

Tibor
- 2017. nov.. 29., 18:26, Christopher Cox c...@endlessnow.com írta:

> On 11/29/2017 09:39 AM, Demeter Tibor wrote:
>>
>> Dear Users,
>>
>> We have an old ovirt3.5 install with a local and a shared clusters. 
>> Meanwhile we
>> created a new data center, that based on 4.1 and it use only shared
>> infrastructure.
>> I would like to migrate an big VM from the old local datacenter to our new, 
>> but
>> I don't have enough downtime.
>>
>> Is it possible to convert the old local storage to shared (by share via NFS) 
>> and
>> attach that as new storage domain to the new cluster?
>> I just want to import VM and copy (while running) with live storage migration
>> function.
>>
>> I know, the official way for move vms between ovirt clusters is the export
>> domain, but it has very big disks.
>>
>> What can I do?
> 
> Just my opinion, but if you don't figure out a way to have occasional 
> downtime,
> you'll probably pay the price with unplanned downtime eventually (and it could
> be painful).
> 
> Define "large disks"?  Terabytes?
> 
> I know for a fact that if you don't have good network segmentation that live
> migrations of large disks can be very problematic.  And I'm not talking about
> what you're wanting to do.  I'm just talking about storage migration.
> 
> We successfully migrated hundreds of VMs from a 3.4 to a 3.6 (on new blades 
> and
> storage) last year over time using the NFS export domain method.
> 
> If storage is the same across DC's, you might be able to shortcut this with
> minimal downtime, but I'm pretty sure there will be some downtime.
> 
> I've seen large storage migrations render entire nodes offline (not nice) due 
> to
> non-isolated paths or QoS.
> 
> 
> 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Convert local storage domain to shared

[Christopher Cox]

On 11/29/2017 09:39 AM, Demeter Tibor wrote:


Dear Users,

We have an old ovirt3.5 install with a local and a shared clusters. Meanwhile we
created a new data center, that based on 4.1 and it use only shared 
infrastructure.
I would like to migrate an big VM from the old local datacenter to our new, but
I don't have enough downtime.

Is it possible to convert the old local storage to shared (by share via NFS) and
attach that as new storage domain to the new cluster?
I just want to import VM and copy (while running) with live storage migration
function.

I know, the official way for move vms between ovirt clusters is the export
domain, but it has very big disks.

What can I do?


Just my opinion, but if you don't figure out a way to have occasional downtime, 
you'll probably pay the price with unplanned downtime eventually (and it could 
be painful).


Define "large disks"?  Terabytes?

I know for a fact that if you don't have good network segmentation that live 
migrations of large disks can be very problematic.  And I'm not talking about 
what you're wanting to do.  I'm just talking about storage migration.


We successfully migrated hundreds of VMs from a 3.4 to a 3.6 (on new blades and 
storage) last year over time using the NFS export domain method.


If storage is the same across DC's, you might be able to shortcut this with 
minimal downtime, but I'm pretty sure there will be some downtime.


I've seen large storage migrations render entire nodes offline (not nice) due to 
non-isolated paths or QoS.




___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] Convert local storage domain to shared

[Demeter Tibor]

Dear Users, 

We have an old ovirt3.5 install with a local and a shared clusters. Meanwhile 
we created a new data center, that based on 4.1 and it use only shared 
infrastructure. 
I would like to migrate an big VM from the old local datacenter to our new, but 
I don't have enough downtime. 

Is it possible to convert the old local storage to shared (by share via NFS) 
and attach that as new storage domain to the new cluster? 
I just want to import VM and copy (while running) with live storage migration 
function. 

I know, the official way for move vms between ovirt clusters is the export 
domain, but it has very big disks. 

What can I do? 

Thanks 

Tibor 





___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users