It depends upon the memory available and the workload. In this case, the size of the files being copied are huge. So more I/O happens to completely copy the file.
Can you please give the o/p of "gluster volume info <volume name>"? Regards, Raghavendra On Wed, Feb 3, 2016 at 4:54 PM, Taste-Of-IT <[email protected]> wrote: > Am 2016-02-03 21:24, schrieb Raghavendra Bhat: > >> I think this is what is happening. Someone please correct me if I am >> wrong. >> >> I think this is happening because nfs client, nfs server and bricks >> being in the same machine. What happens is, when the large write >> comes, nfs client sends the request to the nfs server and the nfs >> server sends it to the brick. The brick process tries to write it via >> making the write system call and the call enters the kernel. Kernel >> might not find memory available for performing the operation and thus >> wants to free some memory. NFS client does heavy caching. It might >> have saved many things in its memory. So, it has to free some memory. >> But nfs client is stuck with the write operation. It is still waiting >> for a response from the server. So it will not be able to free the >> memory till it gets a response from the nfs server (which in turn is >> waiting for a response from the brick) for the write operation it >> sent. But brick cannot get a response from kernel until kernel is able >> to get some memory for the operation and perform write. >> >> Thus it is stuck in this deadlock. Thats why you see your setup >> blocked. >> >> Can you please mount your volume via nfs on a different node other >> than the gluster server, and see if the issue happens again? >> >> Regards, >> Raghavendra >> >> On Wed, Feb 3, 2016 at 2:32 PM, Taste-Of-IT <[email protected]> >> wrote: >> >> Am 2016-02-03 20:09, schrieb Raghavendra Bhat: >>> >>> Hi, >>> >>> Is your nfs client mounted on one of the gluster serves? >>> >>> Regards, >>> Raghavendra >>> >>> On Wed, Feb 3, 2016 at 10:08 AM, Taste-Of-IT >>> <[email protected]> >>> wrote: >>> >>> Hello, >>> >>> hope some expert can help. I have a 2 Brick 1 Volume Distributed >>> GlusterFS in Version 3.7.6 on Debian. The volume is shared via nfs. >>> If i copy via midnight commander large files (>30GB), i got >>> following messages. I replace sata cable, checked memory but i >>> didnt >>> find an error. SMART Values on all disks seems ok. After 30-40 >>> minutes i can copy again. Any Idea? >>> >>> Feb 3 12:46:31 gluster01 kernel: [11186.588367] [sched_delayed] >>> sched: RT throttling activated >>> Feb 3 12:56:09 gluster01 kernel: [11764.932749] glusterfsd >>> D ffff88040ca6d788 0 1150 1 0x00000000 >>> Feb 3 12:56:09 gluster01 kernel: [11764.932759] >>> ffff88040ca6d330 0000000000000082 0000000000012f00 ffff88040ad1bfd8 >>> Feb 3 12:56:09 gluster01 kernel: [11764.932767] >>> 0000000000012f00 ffff88040ca6d330 ffff88040ca6d330 ffff88040ad1be88 >>> Feb 3 12:56:09 gluster01 kernel: [11764.932773] >>> ffff88040e18d4b8 ffff88040e18d4a0 ffffffff00000000 ffff88040e18d4a8 >>> Feb 3 12:56:09 gluster01 kernel: [11764.932780] Call Trace: >>> Feb 3 12:56:09 gluster01 kernel: [11764.932796] >>> [<ffffffff81512cd5>] ? rwsem_down_write_failed+0x1d5/0x320 >>> Feb 3 12:56:09 gluster01 kernel: [11764.932807] >>> [<ffffffff812b7d13>] ? call_rwsem_down_write_failed+0x13/0x20 >>> Feb 3 12:56:09 gluster01 kernel: [11764.932816] >>> [<ffffffff812325b0>] ? proc_keys_show+0x3f0/0x3f0 >>> Feb 3 12:56:09 gluster01 kernel: [11764.932823] >>> [<ffffffff81512649>] ? down_write+0x29/0x40 >>> Feb 3 12:56:09 gluster01 kernel: [11764.932830] >>> [<ffffffff811592bc>] ? vm_mmap_pgoff+0x6c/0xc0 >>> Feb 3 12:56:09 gluster01 kernel: [11764.932838] >>> [<ffffffff8116ea4e>] ? SyS_mmap_pgoff+0x10e/0x250 >>> Feb 3 12:56:09 gluster01 kernel: [11764.932844] >>> [<ffffffff811a969a>] ? SyS_readv+0x6a/0xd0 >>> Feb 3 12:56:09 gluster01 kernel: [11764.932853] >>> [<ffffffff81513ccd>] ? system_call_fast_compare_end+0x10/0x15 >>> Feb 3 12:58:09 gluster01 kernel: [11884.979935] glusterfsd >>> D ffff88040ca6d788 0 1150 1 0x00000000 >>> Feb 3 12:58:09 gluster01 kernel: [11884.979945] >>> ffff88040ca6d330 0000000000000082 0000000000012f00 ffff88040ad1bfd8 >>> Feb 3 12:58:09 gluster01 kernel: [11884.979952] >>> 0000000000012f00 ffff88040ca6d330 ffff88040ca6d330 ffff88040ad1be88 >>> Feb 3 12:58:09 gluster01 kernel: [11884.979959] >>> ffff88040e18d4b8 ffff88040e18d4a0 ffffffff00000000 ffff88040e18d4a8 >>> Feb 3 12:58:09 gluster01 kernel: [11884.979966] Call Trace: >>> Feb 3 12:58:09 gluster01 kernel: [11884.979982] >>> [<ffffffff81512cd5>] ? rwsem_down_write_failed+0x1d5/0x320 >>> Feb 3 12:58:09 gluster01 kernel: [11884.979993] >>> [<ffffffff812b7d13>] ? call_rwsem_down_write_failed+0x13/0x20 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980001] >>> [<ffffffff812325b0>] ? proc_keys_show+0x3f0/0x3f0 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980008] >>> [<ffffffff81512649>] ? down_write+0x29/0x40 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980015] >>> [<ffffffff811592bc>] ? vm_mmap_pgoff+0x6c/0xc0 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980023] >>> [<ffffffff8116ea4e>] ? SyS_mmap_pgoff+0x10e/0x250 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980030] >>> [<ffffffff811a969a>] ? SyS_readv+0x6a/0xd0 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980038] >>> [<ffffffff81513ccd>] ? system_call_fast_compare_end+0x10/0x15 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980351] mc >>> D ffff88040e6d8fb8 0 5119 1447 0x00000000 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980358] >>> ffff88040e6d8b60 0000000000000082 0000000000012f00 ffff88040d5dbfd8 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980365] >>> 0000000000012f00 ffff88040e6d8b60 ffff88041ec937b0 ffff88041efcc9e8 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980371] >>> 0000000000000002 ffffffff8113ce00 ffff88040d5dbcb0 ffff88040d5dbd98 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980377] Call Trace: >>> Feb 3 12:58:09 gluster01 kernel: [11884.980385] >>> [<ffffffff8113ce00>] ? wait_on_page_read+0x60/0x60 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980392] >>> [<ffffffff81510759>] ? io_schedule+0x99/0x120 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980399] >>> [<ffffffff8113ce0a>] ? sleep_on_page+0xa/0x10 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980405] >>> [<ffffffff81510adc>] ? __wait_on_bit+0x5c/0x90 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980412] >>> [<ffffffff8113cbff>] ? wait_on_page_bit+0x7f/0x90 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980420] >>> [<ffffffff810a7bd0>] ? autoremove_wake_function+0x30/0x30 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980426] >>> [<ffffffff8114a17d>] ? pagevec_lookup_tag+0x1d/0x30 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980433] >>> [<ffffffff8113cce0>] ? filemap_fdatawait_range+0xd0/0x160 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980442] >>> [<ffffffff8113e7ca>] ? filemap_write_and_wait_range+0x3a/0x60 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980461] >>> [<ffffffffa072363f>] ? nfs_file_fsync+0x7f/0x100 [nfs] >>> Feb 3 12:58:09 gluster01 kernel: [11884.980476] >>> [<ffffffffa0723a2a>] ? nfs_file_write+0xda/0x1a0 [nfs] >>> Feb 3 12:58:09 gluster01 kernel: [11884.980484] >>> [<ffffffff811a7e24>] ? new_sync_write+0x74/0xa0 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980492] >>> [<ffffffff811a8562>] ? vfs_write+0xb2/0x1f0 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980500] >>> [<ffffffff811a842d>] ? vfs_read+0xed/0x170 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980505] >>> [<ffffffff811a90a2>] ? SyS_write+0x42/0xa0 >>> Feb 3 12:58:09 gluster01 kernel: [11884.980513] >>> [<ffffffff81513ccd>] ? system_call_fast_compare_end+0x10/0x15 >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> [email protected] >>> http://www.gluster.org/mailman/listinfo/gluster-users [1] [1] >>> >>> Links: >>> ------ >>> [1] http://www.gluster.org/mailman/listinfo/gluster-users [1] >>> >> Hi Raghavendra, >> yes in this case i have to mount on one of the gluster server, but it >> doesnt matter on which i mount and its only a question of time when >> the trace came. >> Taste >> >> _______________________________________________ >> Gluster-users mailing list >> [email protected] >> http://www.gluster.org/mailman/listinfo/gluster-users [1] >> >> >> Links: >> ------ >> [1] http://www.gluster.org/mailman/listinfo/gluster-users >> > > Hi, > sounds logical. Is that a normal behavior? I tested it from a client and > it looks fine, without trace. I tried 4 files about 30GB. The only thing i > notice is, that the first file was copied with nearly full bandwidth, over > both server, but the second was only with 20-30 Percent of possible > bandwith. are there any perforamnce / stable option which i can use for nfs > or glusterfs mount? > > _______________________________________________ > Gluster-users mailing list > [email protected] > http://www.gluster.org/mailman/listinfo/gluster-users >
_______________________________________________ Gluster-users mailing list [email protected] http://www.gluster.org/mailman/listinfo/gluster-users
