Hi Luis, GPFS is 4.2.3 (gpfs.base-4.2.3-6.x86_64), All servers (8 in front of a DDN SFA12K) are RHEL 7.3 (stock DDN setup). All 47 clients are CentOS 7.4.
GPFS mount: # mount | grep gpfs gsfs0 on /srv/gsfs0 type gpfs (rw,relatime) NFS mount: mount | grep $HOME 10.210.15.57:/srv/gsfs0/home/griznog on /home/griznog type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.210.15.57,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.210.15.57) Example script: #!/bin/bash cat pt.txt pt.txt pt.txt pt.txt pt.txt pt.txt pt.txt pt.txt pt.txt > /srv/gsfs0/projects/pipetest.tmp.txt grep L1 /srv/gsfs0/projects/pipetest.tmp.txt | wc -l cat pt.txt pt.txt pt.txt pt.txt pt.txt pt.txt pt.txt pt.txt pt.txt > $HOME/pipetest.tmp.txt grep L1 $HOME/pipetest.tmp.txt | wc -l Example output: # ./pipetest.sh 1 1836 # ls -aln /srv/gsfs0/projects/pipetest.tmp.txt $HOME/pipetest.tmp.txt -rw-r--r-- 1 39073 3953 530721 Feb 14 06:10 /home/griznog/pipetest.tmp.txt -rw-r--r-- 1 39073 3001 530721 Feb 14 06:10 /srv/gsfs0/projects/pipetest.tmp.txt We can "fix" the user case that exposed this by not using a temp file or inserting a sleep, but I'd still like to know why GPFS is behaving this way and make it stop. mmlsconfig below. Thanks, jbh mmlsconfig Configuration data for cluster SCG-GS.scg-gs0: ---------------------------------------------- clusterName SCG-GS.scg-gs0 clusterId 8456032987852400706 dmapiFileHandleSize 32 maxblocksize 4096K cnfsSharedRoot /srv/gsfs0/GS-NFS cnfsMountdPort 597 socketMaxListenConnections 1024 fileHeatPeriodMinutes 1440 fileHeatLossPercent 1 pingPeriod 5 minMissedPingTimeout 30 afmHashVersion 1 minReleaseLevel 4.2.0.1 [scg-gs0,scg-gs1,scg-gs2,scg-gs3,scg-gs4,scg-gs5,scg-gs6,scg-gs7] nsdbufspace 70 [common] healthCheckInterval 20 maxStatCache 512 maxFilesToCache 50000 nsdMinWorkerThreads 512 nsdMaxWorkerThreads 1024 deadlockDetectionThreshold 0 deadlockOverloadThreshold 0 prefetchThreads 288 worker1Threads 320 maxMBpS 2000 [scg-gs0,scg-gs1,scg-gs2,scg-gs3,scg-gs4,scg-gs5,scg-gs6,scg-gs7] maxMBpS 24000 [common] atimeDeferredSeconds 300 pitWorkerThreadsPerNode 2 cipherList AUTHONLY pagepool 1G [scg-gs0,scg-gs1,scg-gs2,scg-gs3,scg-gs4,scg-gs5,scg-gs6,scg-gs7] pagepool 8G [common] cnfsNFSDprocs 256 nfsPrefetchStrategy 1 autoload yes adminMode central File systems in cluster SCG-GS.scg-gs0: --------------------------------------- /dev/gsfs0 On Tue, Feb 13, 2018 at 10:53 PM, Luis Bolinches <[email protected]> wrote: > Sorry > > With cat > > [root@specscale01 IBM_REPO]# cp test a > [root@specscale01 IBM_REPO]# cat a a a a > test && grep ATAG test | wc -l > && sleep 4 && grep ATAG test | wc -l > 0 > 0 > -- > Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations > Luis Bolinches > Consultant IT Specialist > Mobile Phone: +358503112585 <+358%2050%203112585> > https://www.youracclaim.com/user/luis-bolinches > > "If you always give you will always have" -- Anonymous > > > > From: Luis Bolinches <[email protected]> > To: gpfsug main discussion list <[email protected]> > Date: 14/02/2018 08:49 > Subject: Re: [gpfsug-discuss] Odd behavior with cat followed by > grep. > Sent by: [email protected] > ------------------------------ > > > > Hi > > This seems to be setup specific > > Care to explain a bit more of the setup. Number of nodes GPFS versions, > number of FS, Networking, running from admin node, server / client, number > of NSD, separated meta and data, etc? > > I got interested and run a quick test on a gpfs far from powerful cluster > of 3 nodes on KVM > > [root@specscale01 IBM_REPO]# echo "a a a a a a a a a a" > test && grep > ATAG test | wc -l && sleep 4 && grep ATAG test | wc -l > 0 > 0 > [root@specscale01 IBM_REPO]# > > > -- > Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations > Luis Bolinches > Consultant IT Specialist > Mobile Phone: +358503112585 <+358%2050%203112585> > *https://www.youracclaim.com/user/luis-bolinches* > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youracclaim.com_user_luis-2Dbolinches&d=DwMFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=1mZ896psa5caYzBeaugTlc7TtRejJp3uvKYxas3S7Xc&m=HrR-mBJ82ubcbtBin7NGVl2VenLj726Fcah6-3XFvDs&s=hGQKSxR54bjW2i9KVDRpflHxrMcJhY269FZkQGWHtLI&e=> > > "If you always give you will always have" -- Anonymous > > > > From: John Hanks <[email protected]> > To: gpfsug-discuss <[email protected]> > Date: 14/02/2018 07:33 > Subject: [gpfsug-discuss] Odd behavior with cat followed by grep. > Sent by: [email protected] > ------------------------------ > > > > Hi, > > We have a GPFS filesystem mounted on CentOS 7.4 as type gpfs, pretty > straightforward run of the mill stuff. But are seeing this odd behavior. If > I do this in a shell script, given a file called "a" > > cat a a a a a a a a a a > /path/to/gpfs/mount/test > grep ATAG /path/to/gpfs/mount/test | wc -l > sleep 4 > grep ATAG /path/to/gpfs/mount/test | wc -l > > The first grep | wc -l returns 1, because grep outputs "Binary file > /path/to/gpfs/mount/test matches" > > The second grep | wc -l returns the correct count of ATAG in the file. > > Why does it take 4 seconds (3 isn't enough) for that file to be properly > recognized as a text file and/or why is it seen as a binary file in the > first place since a is a plain text file? > > Note that I have the same filesystem mounted via NFS and over an NFS mount > it works as expected. > > Any illumination is appreciated, > > jbh_______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > > *https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=1mZ896psa5caYzBeaugTlc7TtRejJp3uvKYxas3S7Xc&m=ut35qIIMxjZMX3obFJ2xtUMng4MtGtKz4YHxpkgQbak&s=cNt66GjRD6rVhq7nGcvT76l-0_u2C3UTz9SfwzHf1xw&e=* > <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=1mZ896psa5caYzBeaugTlc7TtRejJp3uvKYxas3S7Xc&m=ut35qIIMxjZMX3obFJ2xtUMng4MtGtKz4YHxpkgQbak&s=cNt66GjRD6rVhq7nGcvT76l-0_u2C3UTz9SfwzHf1xw&e=> > > > > > Ellei edellä ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland_______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug. > org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r= > 1mZ896psa5caYzBeaugTlc7TtRejJp3uvKYxas3S7Xc&m=HrR- > mBJ82ubcbtBin7NGVl2VenLj726Fcah6-3XFvDs&s=d5YiAyXz4el9bF0zjGL9gVjnTfbX4z > -qelZodxRqlz0&e= > > > > > > Ellei edellä ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
