OK but what if you use it with replication? Do you still see the error? I think not. Could you give it a try and tell me what you find?
-Krutika On Tue, Mar 15, 2016 at 1:23 PM, Mahdi Adnan <[email protected]> wrote: > Hi, > > I have created the following volume; > > Volume Name: v > Type: Distribute > Volume ID: 90de6430-7f83-4eda-a98f-ad1fabcf1043 > Status: Started > Number of Bricks: 3 > Transport-type: tcp > Bricks: > Brick1: gfs001:/bricks/b001/v > Brick2: gfs001:/bricks/b002/v > Brick3: gfs001:/bricks/b003/v > Options Reconfigured: > features.shard-block-size: 128MB > features.shard: enable > cluster.server-quorum-type: server > cluster.quorum-type: auto > network.remote-dio: enable > cluster.eager-lock: enable > performance.stat-prefetch: off > performance.io-cache: off > performance.read-ahead: off > performance.quick-read: off > performance.readdir-ahead: on > > and after mounting it in ESXi and trying to clone a VM to it, i got the > same error. > > > Respectfully > *Mahdi A. Mahdi* > > > On 03/15/2016 10:44 AM, Krutika Dhananjay wrote: > > Hi, > > Do not use sharding and stripe together in the same volume because > a) It is not recommended and there is no point in using both. Using > sharding alone on your volume should work fine. > b) Nobody tested it. > c) Like Niels said, stripe feature is virtually deprecated. > > I would suggest that you create an nx3 volume where n is the number of > distribute subvols you prefer, enable group virt options on it, and enable > sharding on it, > set the shard-block-size that you feel appropriate and then just start off > with VM image creation etc. > If you run into any issues even after you do this, let us know and we'll > help you out. > > -Krutika > > On Tue, Mar 15, 2016 at 1:07 PM, Mahdi Adnan < > [email protected]> wrote: > >> Thanks Krutika, >> >> I have deleted the volume and created a new one. >> I found that it may be an issue with the NFS itself, i have created a new >> striped volume and enabled sharding and mounted it via glusterfs and it >> worked just fine, if i mount it with nfs it will fail and gives me the same >> errors. >> >> Respectfully >> *Mahdi A. Mahdi* >> >> On 03/15/2016 06:24 AM, Krutika Dhananjay wrote: >> >> Hi, >> >> So could you share the xattrs associated with the file at >> <BRICK_PATH>/.glusterfs/c3/e8/c3e88cc1-7e0a-4d46-9685-2d12131a5e1c >> >> Here's what you need to execute: >> >> # getfattr -d -m . -e hex >> /mnt/b1/v/.glusterfs/c3/e8/c3e88cc1-7e0a-4d46-9685-2d12131a5e1c on the >> first node and >> >> # getfattr -d -m . -e hex >> /mnt/b2/v/.glusterfs/c3/e8/c3e88cc1-7e0a-4d46-9685-2d12131a5e1c on the >> second. >> >> >> Also, it is normally advised to use a replica 3 volume as opposed to >> replica 2 volume to guard against split-brains. >> >> -Krutika >> >> On Mon, Mar 14, 2016 at 3:17 PM, Mahdi Adnan < >> <[email protected]>[email protected]> wrote: >> >>> sorry for serial posting but, i got new logs it might help.. >>> >>> the message appear during the migration; >>> >>> /var/log/glusterfs/nfs.log >>> >>> >>> [2016-03-14 09:45:04.573765] I [MSGID: 109036] >>> [dht-common.c:8043:dht_log_new_layout_for_dir_selfheal] 0-testv-dht: >>> Setting layout of /New Virtual Machine_1 with [Subvol_name: testv-stripe-0, >>> Err: -1 , Start: 0 , Stop: 4294967295 , Hash: 1 ], >>> [2016-03-14 09:45:04.957499] E >>> [shard.c:369:shard_modify_size_and_block_count] >>> (-->/usr/lib64/glusterfs/3.7.8/xlator/cluster/distribute.so(dht_file_setattr_cbk+0x14f) >>> [0x7f27a13c067f] >>> -->/usr/lib64/glusterfs/3.7.8/xlator/features/shard.so(shard_common_setattr_cbk+0xcc) >>> [0x7f27a116681c] >>> -->/usr/lib64/glusterfs/3.7.8/xlator/features/shard.so(shard_modify_size_and_block_count+0xdd) >>> [0x7f27a116584d] ) 0-testv-shard: Failed to get >>> trusted.glusterfs.shard.file-size for c3e88cc1-7e0a-4d46-9685-2d12131a5e1c >>> [2016-03-14 09:45:04.957577] W [MSGID: 112199] >>> [nfs3-helpers.c:3418:nfs3_log_common_res] 0-nfs-nfsv3: /New Virtual >>> Machine_1/New Virtual Machine-flat.vmdk => (XID: 3fec5a26, SETATTR: NFS: >>> 22(Invalid argument for operation), POSIX: 22(Invalid argument)) [Invalid >>> argument] >>> [2016-03-14 09:45:05.079657] E [MSGID: 112069] >>> [nfs3.c:3649:nfs3_rmdir_resume] 0-nfs-nfsv3: No such file or directory: ( >>> 192.168.221.52:826) testv : 00000000-0000-0000-0000-000000000001 >>> >>> >>> >>> Respectfully >>> >>> >>> *Mahdi A. Mahd * >>> On 03/14/2016 11:14 AM, Mahdi Adnan wrote: >>> >>> So i have deployed a new server "Cisco UCS C220M4" and created a new >>> volume; >>> >>> Volume Name: testv >>> Type: Stripe >>> Volume ID: 55cdac79-fe87-4f1f-90c0-15c9100fe00b >>> Status: Started >>> Number of Bricks: 1 x 2 = 2 >>> Transport-type: tcp >>> Bricks: >>> Brick1: 10.70.0.250:/mnt/b1/v >>> Brick2: 10.70.0.250:/mnt/b2/v >>> Options Reconfigured: >>> nfs.disable: off >>> features.shard-block-size: 64MB >>> features.shard: enable >>> cluster.server-quorum-type: server >>> cluster.quorum-type: auto >>> network.remote-dio: enable >>> cluster.eager-lock: enable >>> performance.stat-prefetch: off >>> performance.io-cache: off >>> performance.read-ahead: off >>> performance.quick-read: off >>> performance.readdir-ahead: off >>> >>> same error .. >>> >>> can anyone share with me the info of a working striped volume ? >>> >>> On 03/14/2016 09:02 AM, Mahdi Adnan wrote: >>> >>> I have a pool of two bricks in the same server; >>> >>> Volume Name: k >>> Type: Stripe >>> Volume ID: 1e9281ce-2a8b-44e8-a0c6-e3ebf7416b2b >>> Status: Started >>> Number of Bricks: 1 x 2 = 2 >>> Transport-type: tcp >>> Bricks: >>> Brick1: gfs001:/bricks/t1/k >>> Brick2: gfs001:/bricks/t2/k >>> Options Reconfigured: >>> features.shard-block-size: 64MB >>> features.shard: on >>> cluster.server-quorum-type: server >>> cluster.quorum-type: auto >>> network.remote-dio: enable >>> cluster.eager-lock: enable >>> performance.stat-prefetch: off >>> performance.io-cache: off >>> performance.read-ahead: off >>> performance.quick-read: off >>> performance.readdir-ahead: off >>> >>> same issue ... >>> glusterfs 3.7.8 built on Mar 10 2016 20:20:45. >>> >>> >>> Respectfully >>> *Mahdi A. Mahdi* >>> >>> Systems Administrator >>> IT. Department >>> Earthlink Telecommunications <https://www.facebook.com/earthlinktele> >>> >>> Cell: 07903316180 >>> Work: 3352 >>> Skype: <[email protected]>[email protected] >>> On 03/14/2016 08:11 AM, Niels de Vos wrote: >>> >>> On Mon, Mar 14, 2016 at 08:12:27AM +0530, Krutika Dhananjay wrote: >>> >>> It would be better to use sharding over stripe for your vm use case. It >>> offers better distribution and utilisation of bricks and better heal >>> performance. >>> And it is well tested. >>> >>> Basically the "striping" feature is deprecated, "sharding" is its >>> improved replacement. I expect to see "striping" completely dropped in >>> the next major release. >>> >>> Niels >>> >>> >>> >>> Couple of things to note before you do that: >>> 1. Most of the bug fixes in sharding have gone into 3.7.8. So it is advised >>> that you use 3.7.8 or above. >>> 2. When you enable sharding on a volume, already existing files in the >>> volume do not get sharded. Only the files that are newly created from the >>> time sharding is enabled will. >>> If you do want to shard the existing files, then you would need to cp >>> them to a temp name within the volume, and then rename them back to the >>> original file name. >>> >>> HTH, >>> Krutika >>> >>> On Sun, Mar 13, 2016 at 11:49 PM, Mahdi Adnan <[email protected] >>> >>> wrote: >>> >>> I couldn't find anything related to cache in the HBAs. >>> what logs are useful in my case ? i see only bricks logs which contains >>> nothing during the failure. >>> >>> ### >>> [2016-03-13 18:05:19.728614] E [MSGID: 113022] [posix.c:1232:posix_mknod] >>> 0-vmware-posix: mknod on >>> /bricks/b003/vmware/.shard/17d75e20-16f1-405e-9fa5-99ee7b1bd7f1.511 failed >>> [File exists] >>> [2016-03-13 18:07:23.337086] E [MSGID: 113022] [posix.c:1232:posix_mknod] >>> 0-vmware-posix: mknod on >>> /bricks/b003/vmware/.shard/eef2d538-8eee-4e58-bc88-fbf7dc03b263.4095 failed >>> [File exists] >>> [2016-03-13 18:07:55.027600] W [trash.c:1922:trash_rmdir] 0-vmware-trash: >>> rmdir issued on /.trashcan/, which is not permitted >>> [2016-03-13 18:07:55.027635] I [MSGID: 115056] >>> [server-rpc-fops.c:459:server_rmdir_cbk] 0-vmware-server: 41987: RMDIR >>> /.trashcan/internal_op (00000000-0000-0000-0000-000000000005/internal_op) >>> ==> (Operation not permitted) [Operation not permitted] >>> [2016-03-13 18:11:34.353441] I [login.c:81:gf_auth] 0-auth/login: allowed >>> user names: c0c72c37-477a-49a5-a305-3372c1c2f2b4 >>> [2016-03-13 18:11:34.353463] I [MSGID: 115029] >>> [server-handshake.c:612:server_setvolume] 0-vmware-server: accepted client >>> from gfs002-2727-2016/03/13-20:17:43:613597-vmware-client-4-0-0 (version: >>> 3.7.8) >>> [2016-03-13 18:11:34.591139] I [login.c:81:gf_auth] 0-auth/login: allowed >>> user names: c0c72c37-477a-49a5-a305-3372c1c2f2b4 >>> [2016-03-13 18:11:34.591173] I [MSGID: 115029] >>> [server-handshake.c:612:server_setvolume] 0-vmware-server: accepted client >>> from gfs002-2719-2016/03/13-20:17:42:609388-vmware-client-4-0-0 (version: >>> 3.7.8) >>> ### >>> >>> ESXi just keeps telling me "Cannot clone T: The virtual disk is either >>> corrupted or not a supported format. >>> error >>> 3/13/2016 9:06:20 PM >>> Clone virtual machine >>> T >>> VCENTER.LOCAL\Administrator >>> " >>> >>> My setup is 2 servers with a floating ip controlled by CTDB and my ESXi >>> server mount the NFS via the floating ip. >>> >>> >>> >>> >>> >>> On 03/13/2016 08:40 PM, pkoelle wrote: >>> >>> >>> Am 13.03.2016 um 18:22 schrieb David Gossage: >>> >>> >>> On Sun, Mar 13, 2016 at 11:07 AM, Mahdi Adnan <[email protected] >>> >>> wrote: >>> >>> >>> My HBAs are LSISAS1068E, and the filesystem is XFS. >>> >>> I tried EXT4 and it did not help. >>> I have created a stripted volume in one server with two bricks, same >>> issue. >>> and i tried a replicated volume with just "sharding enabled" same issue, >>> as soon as i disable the sharding it works just fine, niether sharding >>> nor >>> striping works for me. >>> i did follow up with some of threads in the mailing list and tried some >>> of >>> the fixes that worked with the others, none worked for me. :( >>> >>> >>> >>> Is it possible the LSI has write-cache enabled? >>> >>> >>> Why is that relevant? Even the backing filesystem has no idea if there is >>> a RAID or write cache or whatever. There are blocks and sync(), end of >>> story. >>> If you lose power and screw up your recovery OR do funky stuff with SAS >>> multipathing that might be an issue with a controller cache. AFAIK thats >>> not what we are talking about. >>> >>> I'm afraid but unless the OP has some logs from the server, a >>> reproducible testcase or a backtrace from client or server this isn't >>> getting us anywhere. >>> >>> cheers >>> Paul >>> >>> >>> >>> On 03/13/2016 06:54 PM, David Gossage wrote: >>> >>> On Sun, Mar 13, 2016 at 8:16 AM, Mahdi Adnan >>> <[email protected]> wrote: >>> >>> Okay so i have enabled shard in my test volume and it did not help, >>> >>> stupidly enough, i have enabled it in a production volume >>> "Distributed-Replicate" and it currpted half of my VMs. >>> I have updated Gluster to the latest and nothing seems to be changed in >>> my situation. >>> below the info of my volume; >>> >>> >>> >>> I was pointing at the settings in that email as an example for >>> corruption >>> fixing. I wouldn't recommend enabling sharding if you haven't gotten the >>> base working yet on that cluster. What HBA's are you using and what is >>> layout of filesystem for bricks? >>> >>> >>> Number of Bricks: 3 x 2 = 6 >>> >>> Transport-type: tcp >>> Bricks: >>> Brick1: gfs001:/bricks/b001/vmware >>> Brick2: gfs002:/bricks/b004/vmware >>> Brick3: gfs001:/bricks/b002/vmware >>> Brick4: gfs002:/bricks/b005/vmware >>> Brick5: gfs001:/bricks/b003/vmware >>> Brick6: gfs002:/bricks/b006/vmware >>> Options Reconfigured: >>> performance.strict-write-ordering: on >>> cluster.server-quorum-type: server >>> cluster.quorum-type: auto >>> network.remote-dio: enable >>> performance.stat-prefetch: disable >>> performance.io-cache: off >>> performance.read-ahead: off >>> performance.quick-read: off >>> cluster.eager-lock: enable >>> features.shard-block-size: 16MB >>> features.shard: on >>> performance.readdir-ahead: off >>> >>> >>> On 03/12/2016 08:11 PM, David Gossage wrote: >>> >>> >>> On Sat, Mar 12, 2016 at 10:21 AM, Mahdi Adnan >>> <<[email protected]> >>> <[email protected]>[email protected]> wrote: >>> >>> Both servers have HBA no RAIDs and i can setup a replicated or >>> >>> dispensers without any issues. >>> Logs are clean and when i tried to migrate a vm and got the error, >>> nothing showed up in the logs. >>> i tried mounting the volume into my laptop and it mounted fine but, >>> if i >>> use dd to create a data file it just hang and i cant cancel it, and i >>> cant >>> unmount it or anything, i just have to reboot. >>> The same servers have another volume on other bricks in a distributed >>> replicas, works fine. >>> I have even tried the same setup in a virtual environment (created two >>> vms and install gluster and created a replicated striped) and again >>> same >>> thing, data corruption. >>> >>> >>> >>> I'd look through mail archives for a topic "Shard in Production" I >>> think >>> it's called. The shard portion may not be relevant but it does discuss >>> certain settings that had to be applied with regards to avoiding >>> corruption >>> with VM's. You may want to try and disable the >>> performance.readdir-ahead >>> also. >>> >>> >>> >>> >>> On 03/12/2016 07:02 PM, David Gossage wrote: >>> >>> >>> >>> On Sat, Mar 12, 2016 at 9:51 AM, Mahdi Adnan >>> <<[email protected]> >>> <[email protected]>[email protected]> wrote: >>> >>> Thanks David, >>> >>> My settings are all defaults, i have just created the pool and >>> started >>> it. >>> I have set the settings as your recommendation and it seems to be the >>> same issue; >>> >>> Type: Striped-Replicate >>> Volume ID: 44adfd8c-2ed1-4aa5-b256-d12b64f7fc14 >>> Status: Started >>> Number of Bricks: 1 x 2 x 2 = 4 >>> Transport-type: tcp >>> Bricks: >>> Brick1: gfs001:/bricks/t1/s >>> Brick2: gfs002:/bricks/t1/s >>> Brick3: gfs001:/bricks/t2/s >>> Brick4: gfs002:/bricks/t2/s >>> Options Reconfigured: >>> performance.stat-prefetch: off >>> network.remote-dio: on >>> cluster.eager-lock: enable >>> performance.io-cache: off >>> performance.read-ahead: off >>> performance.quick-read: off >>> performance.readdir-ahead: on >>> >>> >>> >>> Is their a raid controller perhaps doing any caching? >>> >>> In the gluster logs any errors being reported during migration >>> process? >>> Since they aren't in use yet have you tested making just mirrored >>> bricks >>> using different pairings of servers two at a time to see if problem >>> follows >>> certain machine or network ports? >>> >>> >>> >>> >>> >>> On 03/12/2016 03:25 PM, David Gossage wrote: >>> >>> >>> >>> On Sat, Mar 12, 2016 at 1:55 AM, Mahdi Adnan >>> <<[email protected]> >>> <[email protected]>[email protected]> wrote: >>> >>> Dears, >>> >>> I have created a replicated striped volume with two bricks and two >>> servers but I can't use it because when I mount it in ESXi and try >>> to >>> migrate a VM to it, the data get corrupted. >>> Is any one have any idea why is this happening ? >>> >>> Dell 2950 x2 >>> Seagate 15k 600GB >>> CentOS 7.2 >>> Gluster 3.7.8 >>> >>> Appreciate your help. >>> >>> >>> >>> Most reports of this I have seen end up being settings related. Post >>> gluster volume info. Below is what I have seen as most common >>> recommended >>> settings. >>> I'd hazard a guess you may have some the read ahead cache or prefetch >>> on. >>> >>> quick-read=off >>> read-ahead=off >>> io-cache=off >>> stat-prefetch=off >>> eager-lock=enable >>> remote-dio=on >>> >>> >>> >>> Mahdi Adnan >>> System Admin >>> >>> >>> _______________________________________________ >>> Gluster-users mailing list<[email protected]> >>> <[email protected]>[email protected]<http://www.gluster.org/mailman/listinfo/gluster-users> >>> >>> <http://www.gluster.org/mailman/listinfo/gluster-users>http://www.gluster.org/mailman/listinfo/gluster-users >>> >>> _______________________________________________ >>> Gluster-users mailing >>> [email protected]http://www.gluster.org/mailman/listinfo/gluster-users >>> >>> _______________________________________________ >>> Gluster-users mailing >>> [email protected]http://www.gluster.org/mailman/listinfo/gluster-users >>> >>> _______________________________________________ >>> Gluster-users mailing >>> [email protected]http://www.gluster.org/mailman/listinfo/gluster-users >>> >>> _______________________________________________ >>> Gluster-users mailing >>> [email protected]http://www.gluster.org/mailman/listinfo/gluster-users >>> >>> >>> >>> >>> _______________________________________________ >>> Gluster-users mailing >>> [email protected]http://www.gluster.org/mailman/listinfo/gluster-users >>> >>> >>> >>> >>> _______________________________________________ >>> Gluster-users mailing >>> [email protected]http://www.gluster.org/mailman/listinfo/gluster-users >>> >>> >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> [email protected] >>> http://www.gluster.org/mailman/listinfo/gluster-users >>> >> >> >> > >
_______________________________________________ Gluster-users mailing list [email protected] http://www.gluster.org/mailman/listinfo/gluster-users
