[ovirt-users] Best Practice Question: How many engines, one or more than one, for multiple physical locations
Hello all, I read with Gluster using hyper-convergence that the engine must reside on the same LAN as the nodes. I guess this makes sense by definition - ie: using Gluster storage and replicating Gluster bricks across the web sounds awful. This got me wondering about best practices for the engine setup. We have multiple physical locations (co-location data centers). In my initial plan I had expected to have my oVirt engine hosted separately from each physical location so that in the event of trouble at a remote facility the engine would still be usable. In this case, our prod sites would not have a "hyper-converged" setup if we decide to run GlusterFS for storage at any particular physical site, but I believe it would still be possible to use Gluster. In this case oVirt would have a 3 node cluster, using GlusterFS storage, but not hyper-converged since the engine would be in a separate facility. Is there any downside in this setup to having the engine off-site? Rather than having an off-site engine, should I consider one engine per physical co-location space? Thank you all for any feedback, Matt ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] [Qemu-block] import thin provisioned disks with image upload
On Fri, Dec 8, 2017 at 5:23 PM Max Reitz wrote: > On 2017-12-07 21:33, Nir Soffer wrote: > > [...] > > > > > Trying harder... > > > > > > The term "file size" is ambiguous in this context... > > > > > > It is not. file size is what you get from stat: > > > > $ truncate -s 1g empty > > > > $ stat empty > > File: 'empty' > > Size: 1073741824Blocks: 0 IO Block: 4096 regular file > > ... > > > > $ qemu-img info empty > > image: empty > > file format: raw > > virtual size: 1.0G (1073741824 bytes) > > disk size: 0 > > > > The value "disk size" used by qemu-img is confusing and not useful > > when you want to transfer the file to another host. > > > > I don't know why qemu-img display this value instead of the actual > > file size, adding qemu-block mailing list in case someone can explain > > this. > > Because I still think it's ambiguous. > > $ qemu-img create -f raw empty 1G > Formatting 'empty', fmt=raw size=1073741824 > $ qemu-img info empty > image: empty > file format: raw > virtual size: 1.0G (1073741824 bytes) > disk size: 0 > $ LANG=C stat empty > File: empty > Size: 1073741824 Blocks: 0 IO Block: 4096 regular file > [...] > $ ls -s empty > 0 empty > $ du -h empty > 0 empty > $ ls -l empty > -rw-r--r--. 1 maxx maxx 1073741824 8. Dez 16:20 empty > > What "stat" reports as "size" I'd call the length (what qemu-img info > calls "virtual size" for raw images). raw images are not an issue since the virtual size is the file size. > What I (and qemu-img info) call > "size" is how much disk space is actually used. And both ls -s and du > agree that it's 0 bytes in this case. > > By the way, yes, "stat" has a different definition of "size", so that > seems wrong. But I argue that the "ls" option is called "--size", so > there's a conflict already and qemu-img info is not the first tool to > disagree with stat on this. > I think if you ask most users what is a file size they will tell you what stat is returning. If we use the term in the way most people expect we can make tools easier to use. Users of qemu-img have no way to tell what is disk-size, the value is not documented, at least not in the manual or online help. I think the easy way to improve this is to show both the "allocated size" (st_size * block_size), and the "file size" (st_size). Nir > > Max > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] Move disk between domains
When i try to move a specific disk between storage domains i get an error. 2017-12-08 11:26:05,257-06 ERROR [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService] (default task-41) [] Permutation name: 8C01181C3B121D0AAE1312275CC96415 2017-12-08 11:26:05,257-06 ERROR [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService] (default task-41) [] Uncaught exception: com.google.gwt.core.client.JavaScriptException: (TypeError) __gwt$exception: : Cannot read property 'F' of null at org.ovirt.engine.ui.uicommonweb.models.storage.DisksAllocationModel$3.$onSuccess(DisksAllocationModel.java:120) at org.ovirt.engine.ui.uicommonweb.models.storage.DisksAllocationModel$3.onSuccess(DisksAllocationModel.java:120) at org.ovirt.engine.ui.frontend.Frontend$2.$onSuccess(Frontend.java:233) [frontend.jar:] at org.ovirt.engine.ui.frontend.Frontend$2.onSuccess(Frontend.java:233) [frontend.jar:] at org.ovirt.engine.ui.frontend.communication.OperationProcessor$2.$onSuccess(OperationProcessor.java:139) [frontend.jar:] at org.ovirt.engine.ui.frontend.communication.OperationProcessor$2.onSuccess(OperationProcessor.java:139) [frontend.jar:] at org.ovirt.engine.ui.frontend.communication.GWTRPCCommunicationProvider$5$1.$onSuccess(GWTRPCCommunicationProvider.java:269) [frontend.jar:] at org.ovirt.engine.ui.frontend.communication.GWTRPCCommunicationProvider$5$1.onSuccess(GWTRPCCommunicationProvider.java:269) [frontend.jar:] at com.google.gwt.user.client.rpc.impl.RequestCallbackAdapter.onResponseReceived(RequestCallbackAdapter.java:198) [gwt-servlet.jar:] at com.google.gwt.http.client.Request.$fireOnResponseReceived(Request.java:237) [gwt-servlet.jar:] at com.google.gwt.http.client.RequestBuilder$1.onReadyStateChange(RequestBuilder.java:409) [gwt-servlet.jar:] at Unknown.eval(webadmin-0.js@65) at com.google.gwt.core.client.impl.Impl.apply(Impl.java:296) [gwt-servlet.jar:] at com.google.gwt.core.client.impl.Impl.entry0(Impl.java:335) [gwt-servlet.jar:] at Unknown.eval(webadmin-0.js@54) All the other disks i can move. The issue here is how i got this storage domain into ovirt i think. I set up a new cluster using 4.1 coming from 3.6. I imported a domain from the 3.6 cluster. I am trying to move this disk to one of the new storage domains on the 4.1 cluster. Any help would be greatly appreciated ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] [Qemu-block] Scheduling daily Snapshot
On Fri, Dec 8, 2017 at 4:08 PM Kevin Wolf wrote: > Am 07.12.2017 um 23:19 hat Nir Soffer geschrieben: > > On Wed, Dec 6, 2017 at 6:02 PM Jason Lelievre > > wrote: > > > > > Hello, > > > > > > What is the best way to set up a daily live snapshot for all VM, and > have > > > the possibility to recover, for example, a specific VM to a specific > day? > > > > Each snapshot you create make reads and writes slower, as qemu has to > > lookup data through the entire chain. > > This is true in principle. However, as long as the lookup is purely in > memory and doesn't involve I/O, you won't even notice this in average > use cases. Whether additional I/O is necessary depends on whether the > metadata caches already cover the part of the image that you're > accessing. > > By choosing the right cache size values for the use case, it can > normally be achieved that everything is already in memory. > Can you give more details about selecting the cache size? > > > When we take a snapshot, we create a new file (or block device) and make > > the new file the active layer of the chain. > > > > For example, assuming you have a base image in raw format: > > > > image-1.raw (top) > > > > After taking a snapshot, you have: > > > > image-1.raw <- image-2.qcow2 (top) > > > > Now when qemu need to read data from the image, it will try to get the > > data from the top layer (image-2), if it does not exists it will try > > the backing file (image-1). Same when writing data, if qemu need to > > write small amount of data, it may need to get entire sector from a > > another layer in the chain and copy it to the top layer. > > Yes, though for this operation it doesn't matter whether it has to copy > it from the second image in the chain or the thirtieth. As soon as you > do a partial write to a cluster that hasn't been written yet since the > last snapshot was taken, you get to copy data, no matter the length of > the chain. > So do you think keeping 30 snapshots for backup/restore purpose is a practical solution with negligible effect on performance? > > Kevin > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] [Qemu-block] slow performance with export storage on glusterfs
On Fri, Dec 8, 2017 at 4:18 PM Kevin Wolf wrote: > Am 07.12.2017 um 23:45 hat Nir Soffer geschrieben: > > The qemu bug https://bugzilla.redhat.com/713743 explains the issue: > > qemu-img was writing disk images using writeback and fillingup the > > cache buffers which are then flushed by the kernel preventing other > > processes from accessing the storage. This is particularly bad in > > cluster environments where time-based algorithms might be in place and > > accessing the storage within certain timeouts is critical > > > > I'm not sure it this issue relevant now. We use now sanlock instead of > > safelease, (except for export domain still using safelease), and qemu > > or kernel may have better options to avoid trashing the host cache, or > > guarantee reliable access to storage. > > Non-direct means that the data goes through the kernel page cache, and > the kernel doesn't know that it won't be needed again, so it will fill > up the cache with the image. > > I'm also not aware that cache coherency is now provided by all backends > for shared storage, so O_DIRECT still seems to be the only way to avoid > using stale caches. Since the problem is about stale caches, I don't see > how the locking mechanism could make a difference. > > The only thing I can suggest, given that there is a "glusterfs" in the > subject line of the email, is that the native gluster driver in QEMU > takes a completely different path and never uses the kernel page cache, > which should make both problems disappear. Maybe it would be worth > having a look at this. > This is an interesting direction - currently oVirt does not use native glusterfs for qemu-img operation, only for running vms. We still use the fuse based glusterfs mount for storage operations like copying and converting images. > > Kevin > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] [Qemu-block] slow performance with export storage on glusterfs
Am 07.12.2017 um 23:45 hat Nir Soffer geschrieben: > The qemu bug https://bugzilla.redhat.com/713743 explains the issue: > qemu-img was writing disk images using writeback and fillingup the > cache buffers which are then flushed by the kernel preventing other > processes from accessing the storage. This is particularly bad in > cluster environments where time-based algorithms might be in place and > accessing the storage within certain timeouts is critical > > I'm not sure it this issue relevant now. We use now sanlock instead of > safelease, (except for export domain still using safelease), and qemu > or kernel may have better options to avoid trashing the host cache, or > guarantee reliable access to storage. Non-direct means that the data goes through the kernel page cache, and the kernel doesn't know that it won't be needed again, so it will fill up the cache with the image. I'm also not aware that cache coherency is now provided by all backends for shared storage, so O_DIRECT still seems to be the only way to avoid using stale caches. Since the problem is about stale caches, I don't see how the locking mechanism could make a difference. The only thing I can suggest, given that there is a "glusterfs" in the subject line of the email, is that the native gluster driver in QEMU takes a completely different path and never uses the kernel page cache, which should make both problems disappear. Maybe it would be worth having a look at this. Kevin ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] [Qemu-block] Scheduling daily Snapshot
Am 07.12.2017 um 23:19 hat Nir Soffer geschrieben: > On Wed, Dec 6, 2017 at 6:02 PM Jason Lelievre > wrote: > > > Hello, > > > > What is the best way to set up a daily live snapshot for all VM, and have > > the possibility to recover, for example, a specific VM to a specific day? > > Each snapshot you create make reads and writes slower, as qemu has to > lookup data through the entire chain. This is true in principle. However, as long as the lookup is purely in memory and doesn't involve I/O, you won't even notice this in average use cases. Whether additional I/O is necessary depends on whether the metadata caches already cover the part of the image that you're accessing. By choosing the right cache size values for the use case, it can normally be achieved that everything is already in memory. > When we take a snapshot, we create a new file (or block device) and make > the new file the active layer of the chain. > > For example, assuming you have a base image in raw format: > > image-1.raw (top) > > After taking a snapshot, you have: > > image-1.raw <- image-2.qcow2 (top) > > Now when qemu need to read data from the image, it will try to get the > data from the top layer (image-2), if it does not exists it will try > the backing file (image-1). Same when writing data, if qemu need to > write small amount of data, it may need to get entire sector from a > another layer in the chain and copy it to the top layer. Yes, though for this operation it doesn't matter whether it has to copy it from the second image in the chain or the thirtieth. As soon as you do a partial write to a cluster that hasn't been written yet since the last snapshot was taken, you get to copy data, no matter the length of the chain. Kevin ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
