Re: [Gluster-users] gluster over vdo, problem with gfapi
I don't think it is VDO, but I can be wrong. My ovirt setup is VDO + Gluster v7.7 + CentOS 7.8 . I tested libgfapi a long time ago and it worked. If you wish you can ask in the ovirt users' mailing list how qemu is using libgfapi. Best Regards, Strahil Nikolov На 13 август 2020 г. 7:19:49 GMT+03:00, dm написа: >btw, all I wrote before was about raw file format, >if it is qcow2 then, using gfapi: > > > virsh create /kvmconf/stewjon.xml >error: Failed to create domain from /kvmconf/stewjon.xml >error: internal error: process exited while connecting to monitor: >[2020-08-13 04:17:37.326933] E [MSGID: 108006] >[afr-common.c:6073:__afr_handle_child_down_event] 0-pool-replicate-0: >All subvolumes are down. Going offline until at least one of them comes > >back up. >[2020-08-13 04:17:47.220840] I [io-stats.c:4054:fini] 0-pool: io-stats >translator unloaded >2020-08-13T04:17:47.222064Z qemu-kvm: -drive >file=gluster://127.0.0.1:24007/pool/stewjon.qcow2,file.debug=4,format=qcow2,if=none,id=drive-virtio-disk0,cache=directsync: > >Could not read qcow2 header: Invalid argument > >very interesting... > >only problem here- should I report this to qemu, gluster or vdo? :-( > > >13.08.2020 08:14, Dmitry Melekhov пишет: >> 13.08.2020 07:31, Strahil Nikolov пишет: >>> I ment dis you use C7 with Gluster 7 (or older) or C7 with the new >>> Gluster 8. >> Frankly, I don't know what you mean by C7.. :-( >>> >>> Anyways, >>> if it worked before , it should run now - open an issue in github >and >>> I guess someone from the devs will take a look. >>> >> >> No, it never worked... >> >> >> But opening issue is good idea, thank you! >> >> Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster over vdo, problem with gfapi
C7 -> CentOS7 Just try with the virt group enabled on a test setup . Best Regards, Strahil Nikolov На 13 август 2020 г. 7:14:09 GMT+03:00, Dmitry Melekhov написа: >13.08.2020 07:31, Strahil Nikolov пишет: >> I ment dis you use C7 with Gluster 7 (or older) or C7 with the new >Gluster 8. >Frankly, I don't know what you mean by C7.. :-( >> >> Anyways, >> if it worked before , it should run now - open an issue in github and >I guess someone from the devs will take a look. >> > >No, it never worked... > > >But opening issue is good idea, thank you! Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster over vdo, problem with gfapi
На 13 август 2020 г. 5:23:12 GMT+03:00, Dmitry Melekhov написа: > >12.08.2020 23:25, Strahil Nikolov пишет: >> I am not sure that it is ok to use any caching (at least ovirt >doesn't uses) . >> >> Have you set the 'virt' group of settings ? They seem to be optimal >, but keep in mind that if you enable them -> you will enable sharding > which cannot be 'disabled' afterwards. > > >Sorry, I don't follow, as I said everything works until we set >cache=none or cache=directsync in libvirt, > >i.e. there is no relation with other gluster settings. > >> >> The fact that it works on C7 is strange, with wifh version of >gluster did you test. >> >> >Dunno, we run qemu on the same host as gluster itself, so we have the >same gfapi version as gluster server. I ment dis you use C7 with Gluster 7 (or older) or C7 with the new Gluster 8. Anyways, if it worked before , it should run now - open an issue in github and I guess someone from the devs will take a look. Best Regards, Strahil Nikolov Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster over vdo, problem with gfapi
13.08.2020 10:06, Strahil Nikolov пишет: I don't think it is VDO, but I can be wrong. My ovirt setup is VDO + Gluster v7.7 + CentOS 7.8 . I tested libgfapi a long time ago and it worked. If you wish you can ask in the ovirt users' mailing list how qemu is using libgfapi. As I wrote without vdo everything works just fine on the same server. And it works with vdo over fuse mount in any case and with qfapi if cache settings is default. I don't think that ovirt mail list is right place to ask- we don't use ovirt. Thank you! P.S. We decided to wait some replies here and then open issue in gluster... Thank you! Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster over vdo, problem with gfapi
btw, all I wrote before was about raw file format, if it is qcow2 then, using gfapi: virsh create /kvmconf/stewjon.xml error: Failed to create domain from /kvmconf/stewjon.xml error: internal error: process exited while connecting to monitor: [2020-08-13 04:17:37.326933] E [MSGID: 108006] [afr-common.c:6073:__afr_handle_child_down_event] 0-pool-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up. [2020-08-13 04:17:47.220840] I [io-stats.c:4054:fini] 0-pool: io-stats translator unloaded 2020-08-13T04:17:47.222064Z qemu-kvm: -drive file=gluster://127.0.0.1:24007/pool/stewjon.qcow2,file.debug=4,format=qcow2,if=none,id=drive-virtio-disk0,cache=directsync: Could not read qcow2 header: Invalid argument very interesting... only problem here- should I report this to qemu, gluster or vdo? :-( 13.08.2020 08:14, Dmitry Melekhov пишет: 13.08.2020 07:31, Strahil Nikolov пишет: I ment dis you use C7 with Gluster 7 (or older) or C7 with the new Gluster 8. Frankly, I don't know what you mean by C7.. :-( Anyways, if it worked before , it should run now - open an issue in github and I guess someone from the devs will take a look. No, it never worked... But opening issue is good idea, thank you! Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster over vdo, problem with gfapi
12.08.2020 23:25, Strahil Nikolov пишет: I am not sure that it is ok to use any caching (at least ovirt doesn't uses) . Have you set the 'virt' group of settings ? They seem to be optimal , but keep in mind that if you enable them -> you will enable sharding which cannot be 'disabled' afterwards. Sorry, I don't follow, as I said everything works until we set cache=none or cache=directsync in libvirt, i.e. there is no relation with other gluster settings. The fact that it works on C7 is strange, with wifh version of gluster did you test. Dunno, we run qemu on the same host as gluster itself, so we have the same gfapi version as gluster server. Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster over vdo, problem with gfapi
I am not sure that it is ok to use any caching (at least ovirt doesn't uses) . Have you set the 'virt' group of settings ? They seem to be optimal , but keep in mind that if you enable them -> you will enable sharding which cannot be 'disabled' afterwards. The fact that it works on C7 is strange, with wifh version of gluster did you test. Best Regards, Strahil Nikolov На 12 август 2020 г. 18:03:29 GMT+03:00, Dmitry Melekhov написа: > >12.08.2020 17:50, Strahil Nikolov пишет: >> Libgfapi brings far better performance , > >Yes, and several vms do not rely on the same mount point... > > >> but qemu has some limitations. >> >> >> If it works on FUSE , but not on libgfapi -> it seems obvious. > > >Not obvious for me, we tested vdo locally, i.e. without gluster and >qemu >works with cache=none or cache=directsync without problems, > >so problem is somewhere in gluster. > >> >> Have you tried to connect from C7 to the Gluster TSP via libgfapi. >No, but we tested the same setup with gluster 7 with the same result >before we upgraded to 8. >> >> Also, is SELINUX in enforcing or not ? > >selinux is disabled... > > >Thank you! > >> >> Best Regards, >> Strahil Nikolov >> >> На 12 август 2020 г. 16:34:26 GMT+03:00, Satheesaran Sundaramoorthi > написа: >>> On Wed, Aug 12, 2020 at 2:30 PM Dmitry Melekhov >wrote: >>> 12.08.2020 12:55, Amar Tumballi пишет: > Hi Dimitry, > > Was this working earlier and now failing on Version 8 or is this a >>> new > setup which you did first time? > Hello! This is first time we are testing gluster over vdo. Thank you! Hello Dmitry, >>> I have been testing the RHEL downstream variant of gluster with RHEL >>> 8.2, >>> where VMs are created with their images on fuse mounted gluster >volume >>> with >>> VDO. >>> This worked good. >>> >>> But I see you are using 'gfapi', so that could be different. >>> Though I don't have valuable inputs to help you, do you see 'gfapi' >>> good >>> enough than using fuse mounted volume > > >We think that gfapi is better for 2 reasons: > >1. it is faster; > >2. each qemu process connects to gluster cluster , so there is no one >point of failure- fuse mount... > > >Thank you! > >>> >>> -- Satheesaran S Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster over vdo, problem with gfapi
Libgfapi brings far better performance , but qemu has some limitations. If it works on FUSE , but not on libgfapi -> it seems obvious. Have you tried to connect from C7 to the Gluster TSP via libgfapi. Also, is SELINUX in enforcing or not ? Best Regards, Strahil Nikolov На 12 август 2020 г. 16:34:26 GMT+03:00, Satheesaran Sundaramoorthi написа: >On Wed, Aug 12, 2020 at 2:30 PM Dmitry Melekhov wrote: > >> 12.08.2020 12:55, Amar Tumballi пишет: >> > Hi Dimitry, >> > >> > Was this working earlier and now failing on Version 8 or is this a >new >> > setup which you did first time? >> > >> Hello! >> >> >> This is first time we are testing gluster over vdo. >> >> Thank you! >> >> >> Hello Dmitry, > >I have been testing the RHEL downstream variant of gluster with RHEL >8.2, >where VMs are created with their images on fuse mounted gluster volume >with >VDO. >This worked good. > >But I see you are using 'gfapi', so that could be different. >Though I don't have valuable inputs to help you, do you see 'gfapi' >good >enough than using fuse mounted volume > >-- Satheesaran S Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster over vdo, problem with gfapi
12.08.2020 17:50, Strahil Nikolov пишет: Libgfapi brings far better performance , Yes, and several vms do not rely on the same mount point... but qemu has some limitations. If it works on FUSE , but not on libgfapi -> it seems obvious. Not obvious for me, we tested vdo locally, i.e. without gluster and qemu works with cache=none or cache=directsync without problems, so problem is somewhere in gluster. Have you tried to connect from C7 to the Gluster TSP via libgfapi. No, but we tested the same setup with gluster 7 with the same result before we upgraded to 8. Also, is SELINUX in enforcing or not ? selinux is disabled... Thank you! Best Regards, Strahil Nikolov На 12 август 2020 г. 16:34:26 GMT+03:00, Satheesaran Sundaramoorthi написа: On Wed, Aug 12, 2020 at 2:30 PM Dmitry Melekhov wrote: 12.08.2020 12:55, Amar Tumballi пишет: Hi Dimitry, Was this working earlier and now failing on Version 8 or is this a new setup which you did first time? Hello! This is first time we are testing gluster over vdo. Thank you! Hello Dmitry, I have been testing the RHEL downstream variant of gluster with RHEL 8.2, where VMs are created with their images on fuse mounted gluster volume with VDO. This worked good. But I see you are using 'gfapi', so that could be different. Though I don't have valuable inputs to help you, do you see 'gfapi' good enough than using fuse mounted volume We think that gfapi is better for 2 reasons: 1. it is faster; 2. each qemu process connects to gluster cluster , so there is no one point of failure- fuse mount... Thank you! -- Satheesaran S Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster over vdo, problem with gfapi
On Wed, Aug 12, 2020 at 2:30 PM Dmitry Melekhov wrote: > 12.08.2020 12:55, Amar Tumballi пишет: > > Hi Dimitry, > > > > Was this working earlier and now failing on Version 8 or is this a new > > setup which you did first time? > > > Hello! > > > This is first time we are testing gluster over vdo. > > Thank you! > > > Hello Dmitry, I have been testing the RHEL downstream variant of gluster with RHEL 8.2, where VMs are created with their images on fuse mounted gluster volume with VDO. This worked good. But I see you are using 'gfapi', so that could be different. Though I don't have valuable inputs to help you, do you see 'gfapi' good enough than using fuse mounted volume -- Satheesaran S Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster over vdo, problem with gfapi
12.08.2020 12:55, Amar Tumballi пишет: Hi Dimitry, Was this working earlier and now failing on Version 8 or is this a new setup which you did first time? Hello! This is first time we are testing gluster over vdo. Thank you! Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster over vdo, problem with gfapi
Hi Dimitry, Was this working earlier and now failing on Version 8 or is this a new setup which you did first time? -Amar On Wed, Aug 12, 2020 at 1:17 PM dm wrote: > 12.08.2020 11:39, dm пишет: > > Some more info, really we have lvm over lvm here: > > > > lvm-vdo-lvm... > > > > Thank you! > > > > Sorry, this is wrong, I forgot we replaced this, > > vdo now is over physical drive... > > So, only one lvm layer here. > > > > > 12.08.2020 11:00, Dmitry Melekhov пишет: > >> Hello! > >> > >> > >> We are testing gluster 8 on centos 8.2 and we try to use volume > >> created over vdo. > >> > >> This is 2 nodes setup. > >> > >> There is lvm created over vdo, and xfs filesystem. > >> > >> > >> Test vm runs just fine if we run vm over fuse: > >> > >> > >> > >> > >> > >> > >> > >> > >> /root/pool/ is fuse mount. > >> > >> > >> but if we try to run: > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> then vm boot dies, qemu says- no bootable device. > >> > >> > >> It works without cache='directsync' though. > >> > >> But live migration does not work. > >> > >> > >> btw, everything work OK if we run VM on gluster volume without vdo... > >> > >> Any ideas what can cause this and how it can be fixed? > >> > >> > >> Thank you! > >> > > > > > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > -- -- https://kadalu.io Container Storage made easy! Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster over vdo, problem with gfapi
Some more info, really we have lvm over lvm here: lvm-vdo-lvm... Thank you! 12.08.2020 11:00, Dmitry Melekhov пишет: Hello! We are testing gluster 8 on centos 8.2 and we try to use volume created over vdo. This is 2 nodes setup. There is lvm created over vdo, and xfs filesystem. Test vm runs just fine if we run vm over fuse: /root/pool/ is fuse mount. but if we try to run: then vm boot dies, qemu says- no bootable device. It works without cache='directsync' though. But live migration does not work. btw, everything work OK if we run VM on gluster volume without vdo... Any ideas what can cause this and how it can be fixed? Thank you! Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster over vdo, problem with gfapi
btw, part of brick log:
[2020-08-12 07:08:32.646082] I [MSGID: 115029]
[server-handshake.c:561:server_setvolume] 0-pool-server: accepted client
from CTX_ID:9eea4bec-a522-4a29-be83-5d66c04ce6ee-GRAPH_ID:0-PID:765
2-HOST:nabu-PC_NAME:pool-client-2-RECON_NO:-0 (version: 8.0) with subvol
/wall/pool/brick
[2020-08-12 07:08:32.669522] E [MSGID: 113040]
[posix-inode-fd-ops.c:1727:posix_readv] 0-pool-posix: read failed on
gfid=231fbad6-8d8d-4555-8137-2362a06fc140, fd=0x7f342800ca38, offset=0
size=512, buf=0x7f345450f000 [Invalid argument]
[2020-08-12 07:08:32.669565] E [MSGID: 115068]
[server-rpc-fops_v2.c:1374:server4_readv_cbk] 0-pool-server: READ info
[{frame=34505}, {READV_fd_no=0}, {uuid_utoa=231fbad6-8d8d-4555-8137-2
362a06fc140},
{client=CTX_ID:9eea4bec-a522-4a29-be83-5d66c04ce6ee-GRAPH_ID:0-PID:7652-HOST:nabu-PC_NAME:pool-client-2-RECON_NO:-0},
{error-xlator=pool-posix}, {errno=22}, {error=Invalid a
rgument}]
[2020-08-12 07:08:33.241625] E [MSGID: 113040]
[posix-inode-fd-ops.c:1727:posix_readv] 0-pool-posix: read failed on
gfid=231fbad6-8d8d-4555-8137-2362a06fc140, fd=0x7f342800ca38, offset=0
size=512, buf=0x7f345450f000 [Invalid argument]
[2020-08-12 07:08:33.241669] E [MSGID: 115068]
[server-rpc-fops_v2.c:1374:server4_readv_cbk] 0-pool-server: READ info
[{frame=34507}, {READV_fd_no=0}, {uuid_utoa=231fbad6-8d8d-4555-8137-2
362a06fc140},
{client=CTX_ID:9eea4bec-a522-4a29-be83-5d66c04ce6ee-GRAPH_ID:0-PID:7652-HOST:nabu-PC_NAME:pool-client-2-RECON_NO:-0},
{error-xlator=pool-posix}, {errno=22}, {error=Invalid a
rgument}]
[2020-08-12 07:09:45.897326] W [socket.c:767:__socket_rwv]
0-tcp.pool-server: readv on 192.168.222.25:49081 failed (No data available)
[2020-08-12 07:09:45.897357] I [MSGID: 115036]
[server.c:498:server_rpc_notify] 0-pool-server: disconnecting connection
[{client-uid=CTX_ID:9eea4bec-a522-4a29-be83-5d66c04ce6ee-GRAPH_ID:0
-PID:7652-HOST:nabu-PC_NAME:pool-client-2-RECON_NO:-0}]
Thank you!
12.08.2020 11:00, Dmitry Melekhov пишет:
Hello!
We are testing gluster 8 on centos 8.2 and we try to use volume
created over vdo.
This is 2 nodes setup.
There is lvm created over vdo, and xfs filesystem.
Test vm runs just fine if we run vm over fuse:
/root/pool/ is fuse mount.
but if we try to run:
then vm boot dies, qemu says- no bootable device.
It works without cache='directsync' though.
But live migration does not work.
btw, everything work OK if we run VM on gluster volume without vdo...
Any ideas what can cause this and how it can be fixed?
Thank you!
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
