On 07/17/2011 11:19 PM, Ken Randall wrote:
Joe,
Thank you for your response. After seeing what you wrote, I bumped up
the performance.cache-size up to 4096MB, the max allowed, and ran into
the same wall.
Hmmm ...
I wouldn't think that any SMB caching would help in this case, since the
same Samba server on top of the raw Gluster data wasn't exhibiting any
trouble, or am I deceived?
Samba could cache better so it didn't have to hit Gluster so hard.
I haven't used strace before, but I ran it on the glusterfs process, and
saw a lot of:
epoll_wait(3, {{EPOLLIN, {u32=9, u64=9}}}, 257, 4294967295) = 1
readv(9, [{"\200\0\16,", 4}], 1) = 4
readv(9, [{"\0\n;\227\0\0\0\1", 8}], 1) = 8
readv(9,
[{"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\31\0\0\0\0\0\0\0\1\0\0\0\0"...,
3620}],
1) = 1436
readv(9, 0xa90b1b8, 1) = -1 EAGAIN (Resource
temporarily unavailable)
Interesting ... I am not sure why its reporting an EAGAIN for readv,
other than it can't fill the vector from the read.
And when I ran it on smbd, I saw a constant stream of this kind of activity:
getdents(29, /* 25 entries */, 32768) = 840
getdents(29, /* 25 entries */, 32768) = 856
getdents(29, /* 25 entries */, 32768) = 848
getdents(29, /* 24 entries */, 32768) = 856
getdents(29, /* 25 entries */, 32768) = 864
getdents(29, /* 24 entries */, 32768) = 832
getdents(29, /* 25 entries */, 32768) = 832
getdents(29, /* 24 entries */, 32768) = 856
getdents(29, /* 25 entries */, 32768) = 840
getdents(29, /* 24 entries */, 32768) = 832
getdents(29, /* 25 entries */, 32768) = 784
getdents(29, /* 25 entries */, 32768) = 824
getdents(29, /* 25 entries */, 32768) = 808
getdents(29, /* 25 entries */, 32768) = 840
getdents(29, /* 25 entries */, 32768) = 864
getdents(29, /* 25 entries */, 32768) = 872
getdents(29, /* 25 entries */, 32768) = 832
getdents(29, /* 24 entries */, 32768) = 832
getdents(29, /* 25 entries */, 32768) = 840
getdents(29, /* 25 entries */, 32768) = 824
getdents(29, /* 25 entries */, 32768) = 824
getdents(29, /* 24 entries */, 32768) = 864
getdents(29, /* 25 entries */, 32768) = 848
getdents(29, /* 24 entries */, 32768) = 840
Get directory entries. This is the stuff that NTFS is caching for its
web server, and it appears Samba is not.
Try
aio read size = 32768
csc policy = documents
dfree cache time = 60
directory name cache size = 100000
fake oplocks = yes
getwd cache = yes
level2 oplocks = yes
max stat cache size = 16384
That chunk would get repeated over and over and over again as fast as
the screen could go, with the occasional (every 5-10 seconds or so),
would you see anything that you'd normally expect to see, such as:
close(29) = 0
stat("Storage/01", 0x7fff07dae870) = -1 ENOENT (No such file or directory)
write(23,
"\0\0\0#\377SMB24\0\0\300\210A\310\0\0\0\0\0\0\0\0\0\0\0\0\1\0d\233"...,
39) = 39
select(38, [5 20 23 27 30 31 35 36 37], [], NULL, {60, 0}) = 1 (in [23],
left {60, 0})
read(23, "\0\0\0x", 4) = 4
read(23,
"\377SMB2\0\0\0\0\30\7\310\0\0\0\0\0\0\0\0\0\0\0\0\1\0\250P\273\0[8"...,
120) = 120
stat("Storage", {st_mode=S_IFDIR|0755, st_size=1581056, ...}) = 0
stat("Storage/011235", 0x7fff07dad470) = -1 ENOENT (No such file or
directory)
stat("Storage/011235", 0x7fff07dad470) = -1 ENOENT (No such file or
directory)
open("Storage", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 29
fcntl(29, F_SETFD, FD_CLOEXEC) = 0
(The no such file or directory part is expected since some of the image
references don't exist.)
Ok. It looks like Samba is pounding on GlusterFS metadata (getdents).
GlusterFS doesn't really do a great job in this case ... you have to
give it help and cache pretty aggressively here. Samba can do this
caching to some extent. You might want to enable stat-cache and fast
lookups. These have been problematic for us in the past though, and I'd
recommend caution.
Ken
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: [email protected]
web : http://scalableinformatics.com
http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
