[Gluster-users] pNFS
Hi list, I am wondering why do we need Ganesha user-land NFS server in order to get pNFS working? I understand Ganesha is necessary on the MDS, but standard kernel based NFS server should be sufficient on DS bricks (which should bring us additional performance), right? Could someone clarify? Thanks, Ondrej - The information contained in this e-mail and in any attachments is confidential and is designated solely for the attention of the intended recipient(s). If you are not an intended recipient, you must not use, disclose, copy, distribute or retain this e-mail or any part thereof. If you have received this e-mail in error, please notify the sender by return e-mail and delete all copies of this e-mail from your computer system(s). Please direct any additional queries to: communicati...@s3group.com. Thank You. Silicon and Software Systems Limited (S3 Group). Registered in Ireland no. 378073. Registered Office: South County Business Park, Leopardstown, Dublin 18. ___ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] pnfs(glusterfs-3.7.1 + ganesha-2.2) problem for clients layout commit
On 29/06/15 15:49, 莊尚豪 wrote:
Hi Jiffin,
If you have time, could you test multiple client to ganesha for pnfs?
I test a case that two client connect and translate data, and they
will suspend to translate.
Something like CLIENT ID get conflict to each other, and these are
found wireshark in client.
Hi Ben,
I created new pnfs s with latest code for nfs-ganesha(V2.3-dev-9) and
glusterfs (master)
I have three servers and two clients , all running with fedora20 (kernel
3.19.3-100.fc20.x86_64) and having 1G network card in it.
I created a volume test with following configuration in my enviroment
Volume Name: test
Type: Distribute
Volume ID: 77da4906-526c-4f0f-bd81-23eba2053e36
Status: Started
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: 10.19.96.123:/home/brick/b1
Brick2: 10.19.96.125:/home/brick/b2
Brick3: 10.19.96.127:/home/brick/b3
Options Reconfigured:
features.cache-invalidation: on
nfs.disable: on
I used following dd command for 1gb and 10gb file in both clients
echo 3 /proc/sys/vm/drop_caches;dd if=/dev/zero of=mount point/file
bs=1000K count=1000 conv=sync ;
1000+0 records in
1000+0 records out
1024000 bytes (10 GB) copied, 97.0755 s, 105 MB/s
echo 3 /proc/sys/vm/drop_caches;dd if=/dev/zero of=mount_pointfile
bs=1K count=1000 conv=sync ;
1000+0 records in
1000+0 records out
102400 bytes (1.0 GB) copied, 10.013 s, 102 MB/s
It didn't hang for me. Also i can read file from client1 which is
written from client2.
Also i tried to run distributed iozone sequential write test in which
four files{iozone.DUMMY.[1-4] } of size 2G is written to two clients
concurrently.
Still it didn't hang for me.
I used following command for that.
iozone -+m ./clients.ioz -+h 10.19.96.131 -i 0 -w -+n -c -C -e -s 8G -r
64k -t 4 (clients.ioz -- iozone configuration file.)
It seems to be i can't reproduce your issue
Can u send ur packet trace in a file to me, so that i can analyze same
in a wireshark?
And also send your test case in step by step, so that i can try the same
in my environment.
I am attaching my iozone result with this mail
With regards,
Jiffin
Thanks,
Ben
*From:*Jiffin Tony Thottan [mailto:jthot...@redhat.com]
*Sent:* Friday, June 26, 2015 2:34 PM
*To:* 莊尚豪; gluster-users@gluster.org
*Subject:* Re: [Gluster-users] pnfs(glusterfs-3.7.1 + ganesha-2.2)
problem for clients layout commit
On 26/06/15 08:25, 莊尚豪wrote:
Hi Jiffin,
Thanks your advice, but I still have these errors.
I suspect the pnfs clients causes errors because client can pass
successfully sometimes.
Can you tell me what the linux or nfs-utils version you test? Or
the requirement for client?
As I said before, I ran perf tests (using dd, iozone) for large files
upto 10G two or three months back. I used fedora 20 clients and done
basic testing in RHEL6.5 clients
Initially I had some issues with rhel clients, with new updates i
don't anything. Anyway I will run again same with latest sources and
give my updates on the same.
Just make sure your all ganesha server is up and exporting the
required volume. And more thing , every servers should be resolvable
from m.d.s (there is should information about all other server in
/etc/host).
Regards,
Jiffin
Many thanks,
Ben
Iozone: Performance Test of File I/O
Version $Revision: 3.429 $
Compiled for 64 bit mode.
Build: linux-AMD64
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
Vangel Bojaxhi, Ben England, Vikentsi Lapa.
Run began: Tue Jun 30 02:16:42 2015
Network distribution mode enabled.
Hostname = 10.19.96.131
Setting no_unlink
No retest option selected
Include close in write timing
Include fsync in write timing
File size set to 8388608 kB
Record Size 64 kB
Command line used: iozone -+m ./clients.ioz -+h 10.19.96.131 -i 0 -w
-+n -c -C -e -s 8G -r 64k -t 4
Output is in kBytes/sec
Time Resolution = 0.01 seconds.
Processor cache size set to 1024 kBytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 4 processes
Each process writes a 8388608 kByte file in 64 kByte records
Test running:
Children see throughput for 4 initial writers = 214398.12 kB/sec
Min throughput per process = 53125.08
Re: [Gluster-users] pnfs(glusterfs-3.7.1 + ganesha-2.2) problem for clients layout commit
Hi Jiffin, If you have time, could you test multiple client to ganesha for pnfs? I test a case that two client connect and translate data, and they will suspend to translate. Something like CLIENT ID get conflict to each other, and these are found wireshark in client. Thanks, Ben From: Jiffin Tony Thottan [mailto:jthot...@redhat.com] Sent: Friday, June 26, 2015 2:34 PM To: 莊尚豪; gluster-users@gluster.org Subject: Re: [Gluster-users] pnfs(glusterfs-3.7.1 + ganesha-2.2) problem for clients layout commit On 26/06/15 08:25, 莊尚豪 wrote: Hi Jiffin, Thanks your advice, but I still have these errors. I suspect the pnfs clients causes errors because client can pass successfully sometimes. Can you tell me what the linux or nfs-utils version you test? Or the requirement for client? As I said before, I ran perf tests (using dd, iozone) for large files upto 10G two or three months back. I used fedora 20 clients and done basic testing in RHEL6.5 clients Initially I had some issues with rhel clients, with new updates i don't anything. Anyway I will run again same with latest sources and give my updates on the same. Just make sure your all ganesha server is up and exporting the required volume. And more thing , every servers should be resolvable from m.d.s (there is should information about all other server in /etc/host). Regards, Jiffin Many thanks, Ben ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] pnfs(glusterfs-3.7.1 + ganesha-2.2) problem for clients layout commit
On 26/06/15 08:25, 莊尚豪 wrote: Hi Jiffin, Thanks your advice, but I still have these errors. I suspect the pnfs clients causes errors because client can pass successfully sometimes. Can you tell me what the linux or nfs-utils version you test? Or the requirement for client? As I said before, I ran perf tests (using dd, iozone) for large files upto 10G two or three months back. I used fedora 20 clients and done basic testing in RHEL6.5 clients Initially I had some issues with rhel clients, with new updates i don't anything. Anyway I will run again same with latest sources and give my updates on the same. Just make sure your all ganesha server is up and exporting the required volume. And more thing , every servers should be resolvable from m.d.s (there is should information about all other server in /etc/host). Regards, Jiffin Many thanks, Ben ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] pnfs(glusterfs-3.7.1 + ganesha-2.2) problem for clients layout commit
Hi Jiffin, Thanks your advice, but I still have these errors. I suspect the pnfs clients causes errors because client can pass successfully sometimes. Can you tell me what the linux or nfs-utils version you test? Or the requirement for client? Many thanks, Ben ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] pnfs(glusterfs-3.7.1 + ganesha-2.2) problem for clients layout commit
Hi all,
I test some perfomance from pnfs (gluster-3.7.1 + ganesha-2.2) in Fedora 22.
There are 4 glusterfs nodes with ganesha.
I reference from
https://gluster.readthedocs.org/en/latest/Features/mount_gluster_volume_usin
g_pnfs/
The clients(Fedora 21) are fine to mount and commit some small files to
gluster.
However, when dd the bigger files(dd 600MB file), client will be suspend on
layout commit protocol.
There are some tshark information from client
--
331036 68.713945 192.168.100.12 - 192.168.100.16 NFS 182 V4 Reply (Call In
331012) WRITE
331037 68.718067 192.168.100.16 - 192.168.100.12 NFS 286 V4 Call COMMIT FH:
0x67571bfb Offset: 0 Len: 0
331038 68.718999 192.168.100.12 - 192.168.100.16 NFS 174 V4 Reply (Call In
331037) COMMIT
331039 68.740898 192.168.100.16 - 192.168.100.10 NFS 334 V4 Call
LAYOUTCOMMIT
331040 68.741619 192.168.100.10 - 192.168.100.16 NFS 114 V4 Reply (Call In
331039) SEQUENCE Status: NFS4ERR_BADSESSION
331041 68.741684 192.168.100.16 - 192.168.100.10 TCP 66 908→2049 [ACK]
Seq=8561 Ack=6417 Win=942 Len=0 TSval=8509746 TSecr=108629619
331042 68.742060 192.168.100.16 - 192.168.100.10 NFS 186 V4 Call
DESTROY_SESSION
331043 68.742686 192.168.100.10 - 192.168.100.16 NFS 114 V4 Reply (Call In
331042) DESTROY_SESSION Status: NFS4ERR_BADSESSION
331044 68.742819 192.168.100.16 - 192.168.100.10 NFS 298 V4 Call
CREATE_SESSION
331045 68.743371 192.168.100.10 - 192.168.100.16 NFS 114 V4 Reply (Call In
331044) CREATE_SESSION Status: NFS4ERR_STALE_CLIENTID
331046 68.743529 192.168.100.16 - 192.168.100.10 NFS 334 V4 Call
EXCHANGE_ID
331047 68.744174 192.168.100.10 - 192.168.100.16 NFS 182 V4 Reply (Call In
331046) EXCHANGE_ID
331048 68.744317 192.168.100.16 - 192.168.100.10 NFS 298 V4 Call
CREATE_SESSION
331049 68.756698 192.168.100.10 - 192.168.100.16 NFS 154 V1 CB_NULL Call
331050 68.756825 192.168.100.16 - 192.168.100.10 NFS 94 V1 CB_NULL Reply
(Call In 331049)
331051 68.757543 192.168.100.10 - 192.168.100.16 NFS 194 V4 Reply (Call In
331048) CREATE_SESSION
331052 68.757655 192.168.100.16 - 192.168.100.10 NFS 218 V4 Call PUTROOTFH
| GETATTR
331053 68.758289 192.168.100.10 - 192.168.100.16 NFS 182 V4 Reply (Call In
331052) PUTROOTFH | GETATTR
331054 68.758329 192.168.100.16 - 192.168.100.12 TCP 66 980→2049 [ACK]
Seq=1574831169 Ack=697393 Win=31360 Len=0 TSval=8509763 TSecr=109327597
331055 68.761509 192.168.100.16 - 192.168.100.10 NFS 338 V4 Call OPEN DH:
0x1dfddbb4/
331056 68.762148 192.168.100.10 - 192.168.100.16 NFS 166 V4 Reply (Call In
331055) OPEN Status: NFS4ERR_NO_GRACE
331057 68.762324 192.168.100.16 - 192.168.100.10 NFS 210 V4 Call
RECLAIM_COMPLETE
331058 68.762969 192.168.100.10 - 192.168.100.16 NFS 158 V4 Reply (Call In
331057) RECLAIM_COMPLETE
331059 68.763135 192.168.100.16 - 192.168.100.10 NFS 338 V4 Call OPEN DH:
0x1dfddbb4/
331060 68.763844 192.168.100.10 - 192.168.100.16 NFS 398 V4 Reply (Call In
331059) OPEN StateID: 0x9d75
331061 68.764080 192.168.100.16 - 192.168.100.10 NFS 334 V4 Call
LAYOUTCOMMIT
331062 68.764720 192.168.100.10 - 192.168.100.16 NFS 166 V4 Reply (Call In
331061) LAYOUTCOMMIT Status: NFS4ERR_EXPIRED
331063 68.765075 192.168.100.16 - 192.168.100.10 NFS 202 V4 Call SEQUENCE
331064 68.765699 192.168.100.10 - 192.168.100.16 NFS 150 V4 Reply (Call In
331063) SEQUENCE
331065 68.765906 192.168.100.16 - 192.168.100.10 NFS 334 V4 Call
LAYOUTCOMMIT
331066 68.766555 192.168.100.10 - 192.168.100.16 NFS 166 V4 Reply (Call In
331065) LAYOUTCOMMIT Status: NFS4ERR_EXPIRED
331067 68.766855 192.168.100.16 - 192.168.100.10 NFS 202 V4 Call SEQUENCE
331068 68.767493 192.168.100.10 - 192.168.100.16 NFS 150 V4 Reply (Call In
331067) SEQUENCE
331069 68.767697 192.168.100.16 - 192.168.100.10 NFS 334 V4 Call
LAYOUTCOMMIT
--
Meanwhile, the ganesha server appears the log like these.
There are some server logs when client are failed to commit.
24/06/2015 16:08:49 : epoch 558a6555 : gluster1 : nfs-ganesha-20876[reaper]
nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE
24/06/2015 16:09:56 : epoch 558a6555 : gluster1 : nfs-ganesha-20876[work-10]
nfs4_op_lookup :EXPORT :MAJ :PSEUDO FS JUNCTION TRAVERSAL: Failed to get
FSAL credentials for /ganesha, id=1
24/06/2015 16:09:56 : epoch 558a6555 : gluster1 : nfs-ganesha-20876[work-13]
nfs4_op_lookup :EXPORT :MAJ :PSEUDO FS JUNCTION TRAVERSAL: Failed to get
FSAL credentials for /ganesha, id=1
The following is my glusterfs nodes configuration. There are the same
confiuration for all node.
/etc/ganesha/gluster.conf
--
EXPORT{
Export_Id = 1;
Path = /ganesha; #Is the path attribute useless in the configuration?
FSAL {
name = GLUSTER;
Re: [Gluster-users] pnfs(glusterfs-3.7.1 + ganesha-2.2) problem for clients layout commit
Hi, Comments inline. On 24/06/15 14:39, 莊尚豪 wrote: Hi all, I test some perfomance from pnfs (gluster-3.7.1 + ganesha-2.2) in Fedora 22. There are 4 glusterfs nodes with ganesha. I reference from https://gluster.readthedocs.org/en/latest/Features/mount_gluster_volume_using_pnfs/ Can u check whether ganesha is running on every nodes(M.D.S and D.Ses) , # service nfs-ganesha status or try #ps ax | grep ganesha and also checks whether volume is exported in every node # showmount -e local host The clients(Fedora 21) are fine to mount and commit some small files to gluster. However, when dd the bigger files(dd 600MB file), client will be suspend on layout commit protocol. I had tested dd command (two to three months back) upto file size 10GB , I didn't notice this issue There are some tshark information from client -- 331036 68.713945 192.168.100.12 - 192.168.100.16 NFS 182 V4 Reply (Call In 331012) WRITE 331037 68.718067 192.168.100.16 - 192.168.100.12 NFS 286 V4 Call COMMIT FH: 0x67571bfb Offset: 0 Len: 0 331038 68.718999 192.168.100.12 - 192.168.100.16 NFS 174 V4 Reply (Call In 331037) COMMIT 331039 68.740898 192.168.100.16 - 192.168.100.10 NFS 334 V4 Call LAYOUTCOMMIT 331040 68.741619 192.168.100.10 - 192.168.100.16 NFS 114 V4 Reply (Call In 331039) SEQUENCE Status: NFS4ERR_BADSESSION 331041 68.741684 192.168.100.16 - 192.168.100.10 TCP 66 908→2049 [ACK] Seq=8561 Ack=6417 Win=942 Len=0 TSval=8509746 TSecr=108629619 331042 68.742060 192.168.100.16 - 192.168.100.10 NFS 186 V4 Call DESTROY_SESSION 331043 68.742686 192.168.100.10 - 192.168.100.16 NFS 114 V4 Reply (Call In 331042) DESTROY_SESSION Status: NFS4ERR_BADSESSION 331044 68.742819 192.168.100.16 - 192.168.100.10 NFS 298 V4 Call CREATE_SESSION 331045 68.743371 192.168.100.10 - 192.168.100.16 NFS 114 V4 Reply (Call In 331044) CREATE_SESSION Status: NFS4ERR_STALE_CLIENTID 331046 68.743529 192.168.100.16 - 192.168.100.10 NFS 334 V4 Call EXCHANGE_ID 331047 68.744174 192.168.100.10 - 192.168.100.16 NFS 182 V4 Reply (Call In 331046) EXCHANGE_ID 331048 68.744317 192.168.100.16 - 192.168.100.10 NFS 298 V4 Call CREATE_SESSION 331049 68.756698 192.168.100.10 - 192.168.100.16 NFS 154 V1 CB_NULL Call 331050 68.756825 192.168.100.16 - 192.168.100.10 NFS 94 V1 CB_NULL Reply (Call In 331049) 331051 68.757543 192.168.100.10 - 192.168.100.16 NFS 194 V4 Reply (Call In 331048) CREATE_SESSION 331052 68.757655 192.168.100.16 - 192.168.100.10 NFS 218 V4 Call PUTROOTFH | GETATTR 331053 68.758289 192.168.100.10 - 192.168.100.16 NFS 182 V4 Reply (Call In 331052) PUTROOTFH | GETATTR 331054 68.758329 192.168.100.16 - 192.168.100.12 TCP 66 980→2049 [ACK] Seq=1574831169 Ack=697393 Win=31360 Len=0 TSval=8509763 TSecr=109327597 331055 68.761509 192.168.100.16 - 192.168.100.10 NFS 338 V4 Call OPEN DH: 0x1dfddbb4/ 331056 68.762148 192.168.100.10 - 192.168.100.16 NFS 166 V4 Reply (Call In 331055) OPEN Status: NFS4ERR_NO_GRACE 331057 68.762324 192.168.100.16 - 192.168.100.10 NFS 210 V4 Call RECLAIM_COMPLETE 331058 68.762969 192.168.100.10 - 192.168.100.16 NFS 158 V4 Reply (Call In 331057) RECLAIM_COMPLETE 331059 68.763135 192.168.100.16 - 192.168.100.10 NFS 338 V4 Call OPEN DH: 0x1dfddbb4/ 331060 68.763844 192.168.100.10 - 192.168.100.16 NFS 398 V4 Reply (Call In 331059) OPEN StateID: 0x9d75 331061 68.764080 192.168.100.16 - 192.168.100.10 NFS 334 V4 Call LAYOUTCOMMIT 331062 68.764720 192.168.100.10 - 192.168.100.16 NFS 166 V4 Reply (Call In 331061) LAYOUTCOMMIT Status: NFS4ERR_EXPIRED 331063 68.765075 192.168.100.16 - 192.168.100.10 NFS 202 V4 Call SEQUENCE 331064 68.765699 192.168.100.10 - 192.168.100.16 NFS 150 V4 Reply (Call In 331063) SEQUENCE 331065 68.765906 192.168.100.16 - 192.168.100.10 NFS 334 V4 Call LAYOUTCOMMIT 331066 68.766555 192.168.100.10 - 192.168.100.16 NFS 166 V4 Reply (Call In 331065) LAYOUTCOMMIT Status: NFS4ERR_EXPIRED 331067 68.766855 192.168.100.16 - 192.168.100.10 NFS 202 V4 Call SEQUENCE 331068 68.767493 192.168.100.10 - 192.168.100.16 NFS 150 V4 Reply (Call In 331067) SEQUENCE 331069 68.767697 192.168.100.16 - 192.168.100.10 NFS 334 V4 Call LAYOUTCOMMIT -- Meanwhile, the ganesha server appears the log like these. There are some server logs when client are failed to commit. 24/06/2015 16:08:49 : epoch 558a6555 : gluster1 : nfs-ganesha-20876[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE 24/06/2015 16:09:56 : epoch 558a6555 : gluster1 : nfs-ganesha-20876[work-10] nfs4_op_lookup :EXPORT :MAJ :PSEUDO FS JUNCTION TRAVERSAL: Failed to get FSAL credentials for /ganesha, id=1 24/06/2015 16:09:56 : epoch 558a6555 : gluster1 : nfs-ganesha-20876[work-13] nfs4_op_lookup :EXPORT :MAJ :PSEUDO FS JUNCTION TRAVERSAL: Failed
