Re: [lustre-discuss] poor performance on reading small files

[Dilger, Andreas]

On Aug 3, 2016, at 19:28, Riccardo Veraldi  
wrote:
> 
> On 03/08/16 10:57, Dilger, Andreas wrote:
>> On Jul 29, 2016, at 03:33, Oliver Mangold  
>> wrote:
>>> On 29.07.2016 04:19, Riccardo Veraldi wrote:
 I am using lustre on ZFS.
 
 While write performances are excellent also on smaller files, I find
 there is a drop down in performance
 on reading 20KB files. Performance can go as low as 200MB/sec or even
 less.
>>> 
>>> Getting 200 MB/s with 20kB files means you have to do 1 metadata
>>> ops/s. Don't want to say it is impossible to get more than that, but at
>>> least with MDT on ZFS this doesn't sound bad either. Did you run an
>>> mdtest on your system? Maybe some serious tuning of MD performance is in
>>> order.
>> 
>> I'd agree with Oliver that getting 200MB/s with 20KB files is not too bad.
>> Are you using HDDs or SSDs for the MDT and OST devices?  If using HDDs,
>> are you using SSD L2ARC to allow the metadata and file data be cached in
>> L2ARC, and allowing enough time for L2ARC to be warmed up?
>> 
>> Are you using TCP or IB networking?  If using TCP then there is a lower
>> limit on the number of RPCs that can be handled compared to IB.
> 
> Yes Andreas perhaps is not too bad and in my particular situation I am 
> reading bunch of 20KB chunks inside a bigger 200GB file.
> I found benefits reducing the ZFS record size that was set up at the 
> beginning to a quite large value.

For large streaming writes recordsize=1024k will give the best performance,
but this is not very good for small random IO.  It is not currently possible
to explicitly change the blocksize on a per-file basis.  However, there is
some interest to be able to change this in the future with the "lfs ladvise"
tunable.  See https://jira.hpdd.intel.com/browse/LU-7225 if you are interested
to contribute to this work.

> I am using SSD disks and I did not set up a L2ARC because I do not think I'd 
> have much benefit in my siutation.  So it is not a Lustre problem at all.
> 
> thank you I did not know about LU-4865

The LU-4865 patch is mostly useful for starting with smaller blocksize for
small files, or files written with random IO patterns.  It will not
necessarily help if the file is written sequentially and then read in a
random pattern.

Cheers, Andreas
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] poor performance on reading small files

[Dilger, Andreas]

On Aug 3, 2016, at 19:28, Riccardo Veraldi  
wrote:
> 
> On 03/08/16 10:57, Dilger, Andreas wrote:
>> On Jul 29, 2016, at 03:33, Oliver Mangold  
>> wrote:
>>> On 29.07.2016 04:19, Riccardo Veraldi wrote:
 I am using lustre on ZFS.
 
 While write performances are excellent also on smaller files, I find
 there is a drop down in performance
 on reading 20KB files. Performance can go as low as 200MB/sec or even
 less.
>>> 
>>> Getting 200 MB/s with 20kB files means you have to do 1 metadata
>>> ops/s. Don't want to say it is impossible to get more than that, but at
>>> least with MDT on ZFS this doesn't sound bad either. Did you run an
>>> mdtest on your system? Maybe some serious tuning of MD performance is in
>>> order.
>> 
>> I'd agree with Oliver that getting 200MB/s with 20KB files is not too bad.
>> Are you using HDDs or SSDs for the MDT and OST devices?  If using HDDs,
>> are you using SSD L2ARC to allow the metadata and file data be cached in
>> L2ARC, and allowing enough time for L2ARC to be warmed up?
>> 
>> Are you using TCP or IB networking?  If using TCP then there is a lower
>> limit on the number of RPCs that can be handled compared to IB.
> 
> Yes Andreas perhaps is not too bad and in my particular situation I am 
> reading bunch of 20KB chunks inside a bigger 200GB file.
> I found benefits reducing the ZFS record size that was set up at the 
> beginning to a quite large value.

For large streaming writes recordsize=1024k will give the best performance,
but this is not very good for small random IO.  It is not currently possible
to explicitly change the blocksize on a per-file basis.  However, there is
some interest to be able to change this in the future with the "lfs ladvise"
tunable.  See https://jira.hpdd.intel.com/browse/LU-7225 if you are interested
to contribute to this work.

> I am using SSD disks and I did not set up a L2ARC because I do not think I'd 
> have much benefit in my siutation.  So it is not a Lustre problem at all.
> 
> thank you I did not know about LU-4865

The LU-4865 patch is mostly useful for starting with smaller blocksize for
small files, or files written with random IO patterns.  It will not
necessarily help if the file is written sequentially and then read in a
random pattern.

Cheers, Andreas
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] poor performance on reading small files

[Riccardo Veraldi]

thank you I did not know about LU-4865
On 03/08/16 17:01, Dilger, Andreas wrote:

On Aug 3, 2016, at 12:32, Jeff Johnson  wrote:

On 8/3/16 10:57 AM, Dilger, Andreas wrote:

On Jul 29, 2016, at 03:33, Oliver Mangold  wrote:

On 29.07.2016 04:19, Riccardo Veraldi wrote:

I am using lustre on ZFS.

While write performances are excellent also on smaller files, I find
there is a drop down in performance
on reading 20KB files. Performance can go as low as 200MB/sec or even
less.

Getting 200 MB/s with 20kB files means you have to do 1 metadata
ops/s. Don't want to say it is impossible to get more than that, but at
least with MDT on ZFS this doesn't sound bad either. Did you run an
mdtest on your system? Maybe some serious tuning of MD performance is in
order.

I'd agree with Oliver that getting 200MB/s with 20KB files is not too bad.
Are you using HDDs or SSDs for the MDT and OST devices?  If using HDDs,
are you using SSD L2ARC to allow the metadata and file data be cached in
L2ARC, and allowing enough time for L2ARC to be warmed up?

Are you using TCP or IB networking?  If using TCP then there is a lower
limit on the number of RPCs that can be handled compared to IB.

Also consider that 20KB of data per lnet RPC, assuming a 1MB RPC, to move 20KB 
files at 200MB/sec into a non-striped LFS directory you are using EDR for lnet? 
100GB Ethernet?

It should be clarified that even if the maximum RPC size is 1MB, Lustre will
not send more data than actually contained in the file (subject to the page
size granularity of 4KB).  However, one caveat below for ZFS...

One potential issue if using ZFS with recordsize=1024k is used on the OSTs
then without patch http://review.whamcloud.com/18441 "LU-4865 zfs: grow
block size by write pattern" the blocksize will always be 1MB on the OSTs.
If you are storing a large number of small files then this is probably not
the most efficient use of space, and it will inflate the amount of data sent
over the network as well.  Better to either apply that patch locally (and
provide feedback on how it is working), or select a recordsize that better
matches your file size (e.g. 64KB or 128KB).

Cheers, Andreas




___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] poor performance on reading small files

[Riccardo Veraldi]

On 03/08/16 10:57, Dilger, Andreas wrote:

On Jul 29, 2016, at 03:33, Oliver Mangold  wrote:

On 29.07.2016 04:19, Riccardo Veraldi wrote:

I am using lustre on ZFS.

While write performances are excellent also on smaller files, I find
there is a drop down in performance
on reading 20KB files. Performance can go as low as 200MB/sec or even
less.

Getting 200 MB/s with 20kB files means you have to do 1 metadata
ops/s. Don't want to say it is impossible to get more than that, but at
least with MDT on ZFS this doesn't sound bad either. Did you run an
mdtest on your system? Maybe some serious tuning of MD performance is in
order.

I'd agree with Oliver that getting 200MB/s with 20KB files is not too bad.
Are you using HDDs or SSDs for the MDT and OST devices?  If using HDDs,
are you using SSD L2ARC to allow the metadata and file data be cached in
L2ARC, and allowing enough time for L2ARC to be warmed up?

Are you using TCP or IB networking?  If using TCP then there is a lower
limit on the number of RPCs that can be handled compared to IB.

Cheers, Andreas
Yes Andreas perhaps is not too bad and in my particular situation I am 
reading bunch of 20KB chunks inside a bigger 200GB file.
I found benefits reducing the ZFS record size that was set up at the 
beginning to a quite large value.
I am using SSD disks and I did not set up a L2ARC because I do not think 
I'd have much benefit in my siutation.

So ti is not a Lustre problem at all.
thank you!

Riccardo


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] poor performance on reading small files

[Riccardo Veraldi]

MY issue is related on reading bunch of 20KB slices inside a bigger 
200GB side.

I found out it is not related to Lustre but to ZFS.
So I set up ZFS with proper record size and the problem looks like to be 
mitigated.

Thanks for your hints.

Riccardo


On 03/08/16 08:32, Mohr Jr, Richard Frank (Rick Mohr) wrote:

Do you have the Lustre read caching feature enabled?  I think it should be on 
by default, but you might want to check.  If the files are only 20 KB, then I 
would think the Lustre OSS nodes could keep them in memory most of the time to 
speed up access (unless of course this is a metadata bottleneck as Oliver 
suggested.)  Do your OSS nodes have a lot of memory?  Do you know what your 
typical memory usage is on the OSS nodes?

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu



On Jul 28, 2016, at 10:19 PM, Riccardo Veraldi  
wrote:

Hello,

I have a lustre cluster on rhel7, 6 OSS each of them has 3 OSTs and 1 MDS.

I am using lustre on ZFS.
While write performances are excellent also on smaller files, I find there is a 
drop down in performance
on reading 20KB files. Performance can go as low as 200MB/sec or even less.
I am talking about random reads and random stride reads.
I did the following to try to improve things:
• disabled lnet debug messages:
• sysctl -w lnet.debug=0
• increased dirty cache
• lctl set_param osc.lutrefs\*.max_dirty_mb=256
• increased number of RPC in flight
• for i in `ls  
/proc/fs/lustre/osc/lustrefs-OST00*/max_rpcs_in_flight`; do echo 32 > $i; done
it did not improve reading 20KB file performances.
I have to say in advance I did not set up any striping because I will have no 
more than 6 concurrent reads and writes,
so striping is not that much important for me.
Here the problem is that one single random read  of a 20KB file is around 
190MB/s and this is really disappointing.
I am open to any suggestion.
I made sure it is not a ZFS problem, on the OSSs ZFS is performing like a charm 
locally.
thank you


Riccardo


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org





___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] poor performance on reading small files

[Dilger, Andreas]

On Aug 3, 2016, at 12:32, Jeff Johnson  wrote:
> 
> On 8/3/16 10:57 AM, Dilger, Andreas wrote:
>> On Jul 29, 2016, at 03:33, Oliver Mangold  
>> wrote:
>>> On 29.07.2016 04:19, Riccardo Veraldi wrote:
 I am using lustre on ZFS.
 
 While write performances are excellent also on smaller files, I find
 there is a drop down in performance
 on reading 20KB files. Performance can go as low as 200MB/sec or even
 less.
>>> Getting 200 MB/s with 20kB files means you have to do 1 metadata
>>> ops/s. Don't want to say it is impossible to get more than that, but at
>>> least with MDT on ZFS this doesn't sound bad either. Did you run an
>>> mdtest on your system? Maybe some serious tuning of MD performance is in
>>> order.
>> I'd agree with Oliver that getting 200MB/s with 20KB files is not too bad.
>> Are you using HDDs or SSDs for the MDT and OST devices?  If using HDDs,
>> are you using SSD L2ARC to allow the metadata and file data be cached in
>> L2ARC, and allowing enough time for L2ARC to be warmed up?
>> 
>> Are you using TCP or IB networking?  If using TCP then there is a lower
>> limit on the number of RPCs that can be handled compared to IB.
> 
> Also consider that 20KB of data per lnet RPC, assuming a 1MB RPC, to move 
> 20KB files at 200MB/sec into a non-striped LFS directory you are using EDR 
> for lnet? 100GB Ethernet?

It should be clarified that even if the maximum RPC size is 1MB, Lustre will
not send more data than actually contained in the file (subject to the page
size granularity of 4KB).  However, one caveat below for ZFS...

One potential issue if using ZFS with recordsize=1024k is used on the OSTs
then without patch http://review.whamcloud.com/18441 "LU-4865 zfs: grow
block size by write pattern" the blocksize will always be 1MB on the OSTs.
If you are storing a large number of small files then this is probably not
the most efficient use of space, and it will inflate the amount of data sent
over the network as well.  Better to either apply that patch locally (and
provide feedback on how it is working), or select a recordsize that better
matches your file size (e.g. 64KB or 128KB).

Cheers, Andreas

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] poor performance on reading small files

[Mohr Jr, Richard Frank (Rick Mohr)]

> On Aug 3, 2016, at 1:30 PM, Ben Evans  wrote:
> 
> I thought read caching was disabled by default, as the kernel's default
> handling of pages was better.  

You might be right.  It has been a while since I have set up a Lustre file 
system from scratch, and I haven’t done so for newer versions of Lustre.  So 
it’s possible I am mistaken about the defaults.

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] poor performance on reading small files

[Oucharek, Doug S]

Also note: If you are using IB, these small reads will make use of RDMA.  LNet 
only uses rdma_writes (historical reasons for this) so the client has to use IB 
immediate messages to tell the server to write the 20kb file to the client.  
The extra round-trip handshake involved with this will add latency to each file 
read.  That could be why writes, which don’t need this extra handshake, perform 
better than the reads.

The bigger the files (i.e. the more data moved per rdma_write) the less the 
additional overhead of the handshake will be noticed.

Doug

> On Aug 3, 2016, at 11:32 AM, Jeff Johnson  
> wrote:
> 
> On 8/3/16 10:57 AM, Dilger, Andreas wrote:
>> On Jul 29, 2016, at 03:33, Oliver Mangold  
>> wrote:
>>> On 29.07.2016 04:19, Riccardo Veraldi wrote:
 I am using lustre on ZFS.
 
 While write performances are excellent also on smaller files, I find
 there is a drop down in performance
 on reading 20KB files. Performance can go as low as 200MB/sec or even
 less.
>>> Getting 200 MB/s with 20kB files means you have to do 1 metadata
>>> ops/s. Don't want to say it is impossible to get more than that, but at
>>> least with MDT on ZFS this doesn't sound bad either. Did you run an
>>> mdtest on your system? Maybe some serious tuning of MD performance is in
>>> order.
>> I'd agree with Oliver that getting 200MB/s with 20KB files is not too bad.
>> Are you using HDDs or SSDs for the MDT and OST devices?  If using HDDs,
>> are you using SSD L2ARC to allow the metadata and file data be cached in
>> L2ARC, and allowing enough time for L2ARC to be warmed up?
>> 
>> Are you using TCP or IB networking?  If using TCP then there is a lower
>> limit on the number of RPCs that can be handled compared to IB.
>> 
>> Cheers, Andreas
>> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> 
> Also consider that 20KB of data per lnet RPC, assuming a 1MB RPC, to move 
> 20KB files at 200MB/sec into a non-striped LFS directory you are using EDR 
> for lnet? 100GB Ethernet?
> 
> --Jeff
> 
> 
> -- 
> --
> Jeff Johnson
> Co-Founder
> Aeon Computing
> 
> jeff.john...@aeoncomputing.com
> www.aeoncomputing.com
> t: 858-412-3810 x1001   f: 858-412-3845
> m: 619-204-9061
> 
> 4170 Morena Boulevard, Suite D - San Diego, CA 92117
> 
> High-performance Computing / Lustre Filesystems / Scale-out Storage
> 
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] poor performance on reading small files

[Jeff Johnson]

On 8/3/16 10:57 AM, Dilger, Andreas wrote:

On Jul 29, 2016, at 03:33, Oliver Mangold  wrote:

On 29.07.2016 04:19, Riccardo Veraldi wrote:

I am using lustre on ZFS.

While write performances are excellent also on smaller files, I find
there is a drop down in performance
on reading 20KB files. Performance can go as low as 200MB/sec or even
less.

Getting 200 MB/s with 20kB files means you have to do 1 metadata
ops/s. Don't want to say it is impossible to get more than that, but at
least with MDT on ZFS this doesn't sound bad either. Did you run an
mdtest on your system? Maybe some serious tuning of MD performance is in
order.

I'd agree with Oliver that getting 200MB/s with 20KB files is not too bad.
Are you using HDDs or SSDs for the MDT and OST devices?  If using HDDs,
are you using SSD L2ARC to allow the metadata and file data be cached in
L2ARC, and allowing enough time for L2ARC to be warmed up?

Are you using TCP or IB networking?  If using TCP then there is a lower
limit on the number of RPCs that can be handled compared to IB.

Cheers, Andreas
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


Also consider that 20KB of data per lnet RPC, assuming a 1MB RPC, to 
move 20KB files at 200MB/sec into a non-striped LFS directory you are 
using EDR for lnet? 100GB Ethernet?


--Jeff


--
--
Jeff Johnson
Co-Founder
Aeon Computing

jeff.john...@aeoncomputing.com
www.aeoncomputing.com
t: 858-412-3810 x1001   f: 858-412-3845
m: 619-204-9061

4170 Morena Boulevard, Suite D - San Diego, CA 92117

High-performance Computing / Lustre Filesystems / Scale-out Storage

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] poor performance on reading small files

[Dilger, Andreas]

On Jul 29, 2016, at 03:33, Oliver Mangold  wrote:
> 
> On 29.07.2016 04:19, Riccardo Veraldi wrote:
>> 
>> I am using lustre on ZFS.
>> 
>> While write performances are excellent also on smaller files, I find 
>> there is a drop down in performance
>> on reading 20KB files. Performance can go as low as 200MB/sec or even 
>> less.
> 
> Getting 200 MB/s with 20kB files means you have to do 1 metadata 
> ops/s. Don't want to say it is impossible to get more than that, but at 
> least with MDT on ZFS this doesn't sound bad either. Did you run an 
> mdtest on your system? Maybe some serious tuning of MD performance is in 
> order.

I'd agree with Oliver that getting 200MB/s with 20KB files is not too bad.
Are you using HDDs or SSDs for the MDT and OST devices?  If using HDDs,
are you using SSD L2ARC to allow the metadata and file data be cached in
L2ARC, and allowing enough time for L2ARC to be warmed up?

Are you using TCP or IB networking?  If using TCP then there is a lower
limit on the number of RPCs that can be handled compared to IB.

Cheers, Andreas
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] poor performance on reading small files

[Ben Evans]

I thought read caching was disabled by default, as the kernel's default
handling of pages was better.  It's been awhile since I looked at those
test results.

On 8/3/16, 11:32 AM, "lustre-discuss on behalf of Mohr Jr, Richard Frank
(Rick Mohr)"  wrote:

>Do you have the Lustre read caching feature enabled?  I think it should
>be on by default, but you might want to check.  If the files are only 20
>KB, then I would think the Lustre OSS nodes could keep them in memory
>most of the time to speed up access (unless of course this is a metadata
>bottleneck as Oliver suggested.)  Do your OSS nodes have a lot of memory?
> Do you know what your typical memory usage is on the OSS nodes?
>
>--
>Rick Mohr
>Senior HPC System Administrator
>National Institute for Computational Sciences
>http://www.nics.tennessee.edu
>
>
>> On Jul 28, 2016, at 10:19 PM, Riccardo Veraldi
>> wrote:
>> 
>> Hello,
>> 
>> I have a lustre cluster on rhel7, 6 OSS each of them has 3 OSTs and 1
>>MDS.
>> 
>> I am using lustre on ZFS.
>> While write performances are excellent also on smaller files, I find
>>there is a drop down in performance
>> on reading 20KB files. Performance can go as low as 200MB/sec or even
>>less.
>> I am talking about random reads and random stride reads.
>> I did the following to try to improve things:
>>  € disabled lnet debug messages:
>>  € sysctl -w lnet.debug=0
>>  € increased dirty cache
>>  € lctl set_param osc.lutrefs\*.max_dirty_mb=256
>>  € increased number of RPC in flight
>>  € for i in `ls
>>/proc/fs/lustre/osc/lustrefs-OST00*/max_rpcs_in_flight`; do echo 32 >
>>$i; done
>> it did not improve reading 20KB file performances.
>> I have to say in advance I did not set up any striping because I will
>>have no more than 6 concurrent reads and writes,
>> so striping is not that much important for me.
>> Here the problem is that one single random read  of a 20KB file is
>>around 190MB/s and this is really disappointing.
>> I am open to any suggestion.
>> I made sure it is not a ZFS problem, on the OSSs ZFS is performing like
>>a charm locally.
>> thank you
>> 
>> 
>> Riccardo
>> 
>> 
>> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
>
>___
>lustre-discuss mailing list
>lustre-discuss@lists.lustre.org
>http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] poor performance on reading small files

[Mohr Jr, Richard Frank (Rick Mohr)]

Do you have the Lustre read caching feature enabled?  I think it should be on 
by default, but you might want to check.  If the files are only 20 KB, then I 
would think the Lustre OSS nodes could keep them in memory most of the time to 
speed up access (unless of course this is a metadata bottleneck as Oliver 
suggested.)  Do your OSS nodes have a lot of memory?  Do you know what your 
typical memory usage is on the OSS nodes?

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu


> On Jul 28, 2016, at 10:19 PM, Riccardo Veraldi 
>  wrote:
> 
> Hello,
> 
> I have a lustre cluster on rhel7, 6 OSS each of them has 3 OSTs and 1 MDS.
> 
> I am using lustre on ZFS.
> While write performances are excellent also on smaller files, I find there is 
> a drop down in performance
> on reading 20KB files. Performance can go as low as 200MB/sec or even less.
> I am talking about random reads and random stride reads.
> I did the following to try to improve things:
>   • disabled lnet debug messages:
>   • sysctl -w lnet.debug=0 
>   • increased dirty cache
>   • lctl set_param osc.lutrefs\*.max_dirty_mb=256
>   • increased number of RPC in flight
>   • for i in `ls  
> /proc/fs/lustre/osc/lustrefs-OST00*/max_rpcs_in_flight`; do echo 32 > $i; done
> it did not improve reading 20KB file performances.
> I have to say in advance I did not set up any striping because I will have no 
> more than 6 concurrent reads and writes,
> so striping is not that much important for me.
> Here the problem is that one single random read  of a 20KB file is around 
> 190MB/s and this is really disappointing.
> I am open to any suggestion.
> I made sure it is not a ZFS problem, on the OSSs ZFS is performing like a 
> charm locally.
> thank you
> 
> 
> Riccardo
> 
> 
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] poor performance on reading small files

[Riccardo Veraldi]

Hello,

I have a lustre cluster on rhel7, 6 OSS each of them has 3 OSTs and 1 MDS.

I am using lustre on ZFS.

While write performances are excellent also on smaller files, I find 
there is a drop down in performance

on reading 20KB files. Performance can go as low as 200MB/sec or even less.
I am talking about random reads and random stride reads.
I did the following to try to improve things:

 * disabled lnet debug messages:
 o sysctl -w lnet.debug=0
 * increased dirty cache
 o lctl set_param osc.lutrefs\*.max_dirty_mb=256
 * increased number of RPC in flight
 o for i in `ls
   /proc/fs/lustre/osc/lustrefs-OST00*/max_rpcs_in_flight`; do echo
   32 > $i; done

it did not improve reading 20KB file performances.
I have to say in advance I did not set up any striping because I will 
have no more than 6 concurrent reads and writes,

so striping is not that much important for me.
Here the problem is that one single random read  of a 20KB file is 
around 190MB/s and this is really disappointing.

I am open to any suggestion.
I made sure it is not a ZFS problem, on the OSSs ZFS is performing like 
a charm locally.


thank you


Riccardo


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org