MY issue is related on reading bunch of 20KB slices inside a bigger
200GB side.
I found out it is not related to Lustre but to ZFS.
So I set up ZFS with proper record size and the problem looks like to be
mitigated.
Thanks for your hints.
Riccardo
On 03/08/16 08:32, Mohr Jr, Richard Frank (Rick Mohr) wrote:
Do you have the Lustre read caching feature enabled? I think it should be on
by default, but you might want to check. If the files are only 20 KB, then I
would think the Lustre OSS nodes could keep them in memory most of the time to
speed up access (unless of course this is a metadata bottleneck as Oliver
suggested.) Do your OSS nodes have a lot of memory? Do you know what your
typical memory usage is on the OSS nodes?
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
On Jul 28, 2016, at 10:19 PM, Riccardo Veraldi <[email protected]>
wrote:
Hello,
I have a lustre cluster on rhel7, 6 OSS each of them has 3 OSTs and 1 MDS.
I am using lustre on ZFS.
While write performances are excellent also on smaller files, I find there is a
drop down in performance
on reading 20KB files. Performance can go as low as 200MB/sec or even less.
I am talking about random reads and random stride reads.
I did the following to try to improve things:
• disabled lnet debug messages:
• sysctl -w lnet.debug=0
• increased dirty cache
• lctl set_param osc.lutrefs\*.max_dirty_mb=256
• increased number of RPC in flight
• for i in `ls
/proc/fs/lustre/osc/lustrefs-OST00*/max_rpcs_in_flight`; do echo 32 > $i; done
it did not improve reading 20KB file performances.
I have to say in advance I did not set up any striping because I will have no
more than 6 concurrent reads and writes,
so striping is not that much important for me.
Here the problem is that one single random read of a 20KB file is around
190MB/s and this is really disappointing.
I am open to any suggestion.
I made sure it is not a ZFS problem, on the OSSs ZFS is performing like a charm
locally.
thank you
Riccardo
_______________________________________________
lustre-discuss mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
_______________________________________________
lustre-discuss mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
