On 5 September 2018 at 14:10, Nithya Balachandran <[email protected]> wrote:
> > > On 5 September 2018 at 14:02, Nithya Balachandran <[email protected]> > wrote: > >> Hi, >> >> Please try turning off cluster.readdir-optimize and see it if helps. >> > > You can also try turning off parallel-readdir. > Please ignore this - it is not enabled on your volume. > > >> If not, please send us the client mount logs and a tcpdump of when the >> *ls* is performed from the client. Use the following to capture the >> dump: >> >> tcpdump -i any -s 0 -w /var/tmp/dirls.pcap tcp and not port 22 >> >> >> >> Thanks, >> Nithya >> >> >>> >>> Raghavendra Gowdappa <[email protected]> 于2018年9月5日周三 下午12:40写道： >>> >>>> >>>> >>>> >>>>> >>>>>> >>>>>> On Tue, Sep 4, 2018 at 5:28 PM, yu sun <[email protected]> wrote: >>>>>> >>>>>>> Hi all: >>>>>>> >>>>>>> I have a replicated volume project2 with info: >>>>>>> Volume Name: project2 Type: Distributed-Replicate Volume ID: >>>>>>> 60175b8e-de0e-4409-81ae-7bb5eb5cacbf Status: Started Snapshot >>>>>>> Count: 0 Number of Bricks: 84 x 2 = 168 Transport-type: tcp Bricks: >>>>>>> Brick1: >>>>>>> node20:/data2/bricks/project2 Brick2: node21:/data2/bricks/project2 >>>>>>> Brick3: >>>>>>> node22:/data2/bricks/project2 Brick4: node23:/data2/bricks/project2 >>>>>>> Brick5: >>>>>>> node24:/data2/bricks/project2 Brick6: node25:/data2/bricks/project2 >>>>>>> Brick7: >>>>>>> node26:/data2/bricks/project2 Brick8: node27:/data2/bricks/project2 >>>>>>> Brick9: >>>>>>> node28:/data2/bricks/project2 Brick10: node29:/data2/bricks/project2 >>>>>>> Brick11: node30:/data2/bricks/project2 Brick12: >>>>>>> node31:/data2/bricks/project2 Brick13: node32:/data2/bricks/project2 >>>>>>> Brick14: node33:/data2/bricks/project2 Brick15: >>>>>>> node20:/data3/bricks/project2 Brick16: node21:/data3/bricks/project2 >>>>>>> Brick17: node22:/data3/bricks/project2 Brick18: >>>>>>> node23:/data3/bricks/project2 Brick19: node24:/data3/bricks/project2 >>>>>>> Brick20: node25:/data3/bricks/project2 Brick21: >>>>>>> node26:/data3/bricks/project2 Brick22: node27:/data3/bricks/project2 >>>>>>> Brick23: node28:/data3/bricks/project2 Brick24: >>>>>>> node29:/data3/bricks/project2 Brick25: node30:/data3/bricks/project2 >>>>>>> Brick26: node31:/data3/bricks/project2 Brick27: >>>>>>> node32:/data3/bricks/project2 Brick28: node33:/data3/bricks/project2 >>>>>>> Brick29: node20:/data4/bricks/project2 Brick30: >>>>>>> node21:/data4/bricks/project2 Brick31: node22:/data4/bricks/project2 >>>>>>> Brick32: node23:/data4/bricks/project2 Brick33: >>>>>>> node24:/data4/bricks/project2 Brick34: node25:/data4/bricks/project2 >>>>>>> Brick35: node26:/data4/bricks/project2 Brick36: >>>>>>> node27:/data4/bricks/project2 Brick37: node28:/data4/bricks/project2 >>>>>>> Brick38: node29:/data4/bricks/project2 Brick39: >>>>>>> node30:/data4/bricks/project2 Brick40: node31:/data4/bricks/project2 >>>>>>> Brick41: node32:/data4/bricks/project2 Brick42: >>>>>>> node33:/data4/bricks/project2 Brick43: node20:/data5/bricks/project2 >>>>>>> Brick44: node21:/data5/bricks/project2 Brick45: >>>>>>> node22:/data5/bricks/project2 Brick46: node23:/data5/bricks/project2 >>>>>>> Brick47: node24:/data5/bricks/project2 Brick48: >>>>>>> node25:/data5/bricks/project2 Brick49: node26:/data5/bricks/project2 >>>>>>> Brick50: node27:/data5/bricks/project2 Brick51: >>>>>>> node28:/data5/bricks/project2 Brick52: node29:/data5/bricks/project2 >>>>>>> Brick53: node30:/data5/bricks/project2 Brick54: >>>>>>> node31:/data5/bricks/project2 Brick55: node32:/data5/bricks/project2 >>>>>>> Brick56: node33:/data5/bricks/project2 Brick57: >>>>>>> node20:/data6/bricks/project2 Brick58: node21:/data6/bricks/project2 >>>>>>> Brick59: node22:/data6/bricks/project2 Brick60: >>>>>>> node23:/data6/bricks/project2 Brick61: node24:/data6/bricks/project2 >>>>>>> Brick62: node25:/data6/bricks/project2 Brick63: >>>>>>> node26:/data6/bricks/project2 Brick64: node27:/data6/bricks/project2 >>>>>>> Brick65: node28:/data6/bricks/project2 Brick66: >>>>>>> node29:/data6/bricks/project2 Brick67: node30:/data6/bricks/project2 >>>>>>> Brick68: node31:/data6/bricks/project2 Brick69: >>>>>>> node32:/data6/bricks/project2 Brick70: node33:/data6/bricks/project2 >>>>>>> Brick71: node20:/data7/bricks/project2 Brick72: >>>>>>> node21:/data7/bricks/project2 Brick73: node22:/data7/bricks/project2 >>>>>>> Brick74: node23:/data7/bricks/project2 Brick75: >>>>>>> node24:/data7/bricks/project2 Brick76: node25:/data7/bricks/project2 >>>>>>> Brick77: node26:/data7/bricks/project2 Brick78: >>>>>>> node27:/data7/bricks/project2 Brick79: node28:/data7/bricks/project2 >>>>>>> Brick80: node29:/data7/bricks/project2 Brick81: >>>>>>> node30:/data7/bricks/project2 Brick82: node31:/data7/bricks/project2 >>>>>>> Brick83: node32:/data7/bricks/project2 Brick84: >>>>>>> node33:/data7/bricks/project2 Brick85: node20:/data8/bricks/project2 >>>>>>> Brick86: node21:/data8/bricks/project2 Brick87: >>>>>>> node22:/data8/bricks/project2 Brick88: node23:/data8/bricks/project2 >>>>>>> Brick89: node24:/data8/bricks/project2 Brick90: >>>>>>> node25:/data8/bricks/project2 Brick91: node26:/data8/bricks/project2 >>>>>>> Brick92: node27:/data8/bricks/project2 Brick93: >>>>>>> node28:/data8/bricks/project2 Brick94: node29:/data8/bricks/project2 >>>>>>> Brick95: node30:/data8/bricks/project2 Brick96: >>>>>>> node31:/data8/bricks/project2 Brick97: node32:/data8/bricks/project2 >>>>>>> Brick98: node33:/data8/bricks/project2 Brick99: >>>>>>> node20:/data9/bricks/project2 Brick100: node21:/data9/bricks/project2 >>>>>>> Brick101: node22:/data9/bricks/project2 Brick102: >>>>>>> node23:/data9/bricks/project2 Brick103: node24:/data9/bricks/project2 >>>>>>> Brick104: node25:/data9/bricks/project2 Brick105: >>>>>>> node26:/data9/bricks/project2 Brick106: node27:/data9/bricks/project2 >>>>>>> Brick107: node28:/data9/bricks/project2 Brick108: >>>>>>> node29:/data9/bricks/project2 Brick109: node30:/data9/bricks/project2 >>>>>>> Brick110: node31:/data9/bricks/project2 Brick111: >>>>>>> node32:/data9/bricks/project2 Brick112: node33:/data9/bricks/project2 >>>>>>> Brick113: node20:/data10/bricks/project2 Brick114: >>>>>>> node21:/data10/bricks/project2 Brick115: node22:/data10/bricks/project2 >>>>>>> Brick116: node23:/data10/bricks/project2 Brick117: >>>>>>> node24:/data10/bricks/project2 Brick118: node25:/data10/bricks/project2 >>>>>>> Brick119: node26:/data10/bricks/project2 Brick120: >>>>>>> node27:/data10/bricks/project2 Brick121: node28:/data10/bricks/project2 >>>>>>> Brick122: node29:/data10/bricks/project2 Brick123: >>>>>>> node30:/data10/bricks/project2 Brick124: node31:/data10/bricks/project2 >>>>>>> Brick125: node32:/data10/bricks/project2 Brick126: >>>>>>> node33:/data10/bricks/project2 Brick127: node20:/data11/bricks/project2 >>>>>>> Brick128: node21:/data11/bricks/project2 Brick129: >>>>>>> node22:/data11/bricks/project2 Brick130: node23:/data11/bricks/project2 >>>>>>> Brick131: node24:/data11/bricks/project2 Brick132: >>>>>>> node25:/data11/bricks/project2 Brick133: node26:/data11/bricks/project2 >>>>>>> Brick134: node27:/data11/bricks/project2 Brick135: >>>>>>> node28:/data11/bricks/project2 Brick136: node29:/data11/bricks/project2 >>>>>>> Brick137: node30:/data11/bricks/project2 Brick138: >>>>>>> node31:/data11/bricks/project2 Brick139: node32:/data11/bricks/project2 >>>>>>> Brick140: node33:/data11/bricks/project2 Brick141: >>>>>>> node20:/data12/bricks/project2 Brick142: node21:/data12/bricks/project2 >>>>>>> Brick143: node22:/data12/bricks/project2 Brick144: >>>>>>> node23:/data12/bricks/project2 Brick145: node24:/data12/bricks/project2 >>>>>>> Brick146: node25:/data12/bricks/project2 Brick147: >>>>>>> node26:/data12/bricks/project2 Brick148: node27:/data12/bricks/project2 >>>>>>> Brick149: node28:/data12/bricks/project2 Brick150: >>>>>>> node29:/data12/bricks/project2 Brick151: node30:/data12/bricks/project2 >>>>>>> Brick152: node31:/data12/bricks/project2 Brick153: >>>>>>> node32:/data12/bricks/project2 Brick154: node33:/data12/bricks/project2 >>>>>>> Brick155: node20:/data13/bricks/project2 Brick156: >>>>>>> node21:/data13/bricks/project2 Brick157: node22:/data13/bricks/project2 >>>>>>> Brick158: node23:/data13/bricks/project2 Brick159: >>>>>>> node24:/data13/bricks/project2 Brick160: node25:/data13/bricks/project2 >>>>>>> Brick161: node26:/data13/bricks/project2 Brick162: >>>>>>> node27:/data13/bricks/project2 Brick163: node28:/data13/bricks/project2 >>>>>>> Brick164: node29:/data13/bricks/project2 Brick165: >>>>>>> node30:/data13/bricks/project2 Brick166: node31:/data13/bricks/project2 >>>>>>> Brick167: node32:/data13/bricks/project2 Brick168: >>>>>>> node33:/data13/bricks/project2 Options Reconfigured: >>>>>>> performance.force-readdirp: on performance.write-behind: off >>>>>>> performance.stat-prefetch: on performance.client-io-threads: on >>>>>>> nfs.disable: on transport.address-family: inet features.quota: on >>>>>>> features.inode-quota: on features.quota-deem-statfs: on >>>>>>> cluster.readdir-optimize: on cluster.lookup-optimize: on >>>>>>> dht.force-readdirp: off client.event-threads: 10 server.event-threads: >>>>>>> 10 >>>>>>> performance.readdir-ahead: on performance.io-cache: on >>>>>>> performance.flush-behind: on performance.cache-size: 5GB >>>>>>> performance.cache-max-file-size: 1MB >>>>>>> performance.write-behind-window-size: >>>>>>> 10MB performance.read-ahead: off network.remote-dio: enable >>>>>>> performance.strict-o-direct: disable performance.io-thread-count: 25 >>>>>>> >>>>>>> >>>>>>> the volume looks ok, and I mount this volume on my client machine: >>>>>>> mount -t glusterfs -o oom-score-adj=-999 -o direct-io-mode=disable >>>>>>> -o use-readdirp=no node20:/project2 /mnt/project2 >>>>>>> >>>>>>> I have a directory in /mnt/project2/, but when I mv the directory to >>>>>>> other dirs, files in the dir lost while tree or ls the dir, some files >>>>>>> missing, my operations is list as below: >>>>>>> >>>>>> >>>>>> Looks very similar to: >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1118762 >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1337394 >>>>>> >>>>>> >>>>>> >>>>>>> [email protected]:/mnt/project2/371_37829$ mkdir test-dir >>>>>>> [email protected]:/mnt/project2/371_37829$ tree >>>>>>> . >>>>>>> ├── face_landmarks >>>>>>> │ └── alive >>>>>>> │ └── logs_20180823_28 >>>>>>> │ ├── info_000000.out >>>>>>> │ ├── info_000001.out >>>>>>> │ ├── info_000002.out >>>>>>> │ ├── info_000003.out >>>>>>> │ ├── info_000004.out >>>>>>> │ ├── info_000005.out >>>>>>> │ ├── info_000006.out >>>>>>> │ ├── info_000007.out >>>>>>> │ ├── info_000008.out >>>>>>> │ ├── info_000009.out >>>>>>> │ ├── info_000010.out >>>>>>> │ ├── info_000011.out >>>>>>> │ ├── info_000012.out >>>>>>> │ ├── info_000013.out >>>>>>> │ ├── info_000014.out >>>>>>> │ ├── info_000015.out >>>>>>> │ ├── info_000016.out >>>>>>> │ ├── info_000017.out >>>>>>> │ ├── info_000018.out >>>>>>> │ └── info_000019.out >>>>>>> └── test-dir >>>>>>> >>>>>>> 4 directories, 20 files >>>>>>> [email protected]:/mnt/project2/371_37829$ mv >>>>>>> face_landmarks/ test-dir/ >>>>>>> [email protected]:/mnt/project2/371_37829$ tree >>>>>>> . >>>>>>> └── test-dir >>>>>>> └── face_landmarks >>>>>>> >>>>>>> 2 directories, 0 files >>>>>>> [email protected]:/mnt/project2/371_37829$ cd >>>>>>> test-dir/face_landmarks/ >>>>>>> [email protected]:/mnt/project2/371_37829/test-dir/face_landmarks$ >>>>>>> ls >>>>>>> [email protected]:/mnt/project2/371_37829/test-dir/face_landmarks$ >>>>>>> cd .. >>>>>>> [email protected]:/mnt/project2/371_37829/test-dir$ mv >>>>>>> face_landmarks/ .. >>>>>>> [email protected]:/mnt/project2/371_37829/test-dir$ cd .. >>>>>>> [email protected]:/mnt/project2/371_37829$ tree >>>>>>> . >>>>>>> ├── face_landmarks >>>>>>> │ └── alive >>>>>>> └── test-dir >>>>>>> >>>>>>> 3 directories, 0 files >>>>>>> [email protected]:/mnt/project2/371_37829$ >>>>>>> >>>>>>> I think i make some mistakes with volume option, buti i am not sure, >>>>>>> so how can i find the lost files? the files seems still int the >>>>>>> directory, >>>>>>> because i cant remove the directory and rm tell me "Not empty >>>>>>> directory" >>>>>>> >>>>>> >>>>>> Its likely that src and dst of mv having same gfid and that's causing >>>>>> the issues. Can you look into both src and dst paths on all bricks? Union >>>>>> of contents of both directories should give all the files in the src >>>>>> directory before mv. Once found you can, >>>>>> * keep a backup of contents of src and dst on all bricks >>>>>> * remove trusted.gfid xattr on src and dst from all bricks >>>>>> * remove gfid handle (.glusterfs/<first two characters of >>>>>> gfid>/<second set of two characters of gfid>/<gfid> on each brick) >>>>>> * disable readdirplus in entire stack (maybe you can use a tmp mount >>>>>> for this) [1] >>>>>> * stat src and dst on a mount point with readdirplus disabled. >>>>>> * Now you'll see two directories src and dst on mountpoint. You can >>>>>> copy the contents of both into a new directory >>>>>> >>>>>> [1] https://lists.gluster.org/pipermail/gluster-users/2017-March >>>>>> /030148.html >>>>>> >>>>>> >>>>>>> >>>>>>> Any suggestions is appreciated. >>>>>>> Many Thanks >>>>>>> >>>>>>> Best regards >>>>>>> Yu >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> [email protected] >>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>> >>>>>> >>>>>> >>>> >>> _______________________________________________ >>> Gluster-users mailing list >>> [email protected] >>> https://lists.gluster.org/mailman/listinfo/gluster-users >>> >> >> >
_______________________________________________ Gluster-users mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-users
