Re: [lustre-discuss] poor performance on reading small files
On Aug 3, 2016, at 19:28, Riccardo Veraldiwrote: > > On 03/08/16 10:57, Dilger, Andreas wrote: >> On Jul 29, 2016, at 03:33, Oliver Mangold >> wrote: >>> On 29.07.2016 04:19, Riccardo Veraldi wrote: I am using lustre on ZFS. While write performances are excellent also on smaller files, I find there is a drop down in performance on reading 20KB files. Performance can go as low as 200MB/sec or even less. >>> >>> Getting 200 MB/s with 20kB files means you have to do 1 metadata >>> ops/s. Don't want to say it is impossible to get more than that, but at >>> least with MDT on ZFS this doesn't sound bad either. Did you run an >>> mdtest on your system? Maybe some serious tuning of MD performance is in >>> order. >> >> I'd agree with Oliver that getting 200MB/s with 20KB files is not too bad. >> Are you using HDDs or SSDs for the MDT and OST devices? If using HDDs, >> are you using SSD L2ARC to allow the metadata and file data be cached in >> L2ARC, and allowing enough time for L2ARC to be warmed up? >> >> Are you using TCP or IB networking? If using TCP then there is a lower >> limit on the number of RPCs that can be handled compared to IB. > > Yes Andreas perhaps is not too bad and in my particular situation I am > reading bunch of 20KB chunks inside a bigger 200GB file. > I found benefits reducing the ZFS record size that was set up at the > beginning to a quite large value. For large streaming writes recordsize=1024k will give the best performance, but this is not very good for small random IO. It is not currently possible to explicitly change the blocksize on a per-file basis. However, there is some interest to be able to change this in the future with the "lfs ladvise" tunable. See https://jira.hpdd.intel.com/browse/LU-7225 if you are interested to contribute to this work. > I am using SSD disks and I did not set up a L2ARC because I do not think I'd > have much benefit in my siutation. So it is not a Lustre problem at all. > > thank you I did not know about LU-4865 The LU-4865 patch is mostly useful for starting with smaller blocksize for small files, or files written with random IO patterns. It will not necessarily help if the file is written sequentially and then read in a random pattern. Cheers, Andreas ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] poor performance on reading small files
On Aug 3, 2016, at 19:28, Riccardo Veraldiwrote: > > On 03/08/16 10:57, Dilger, Andreas wrote: >> On Jul 29, 2016, at 03:33, Oliver Mangold >> wrote: >>> On 29.07.2016 04:19, Riccardo Veraldi wrote: I am using lustre on ZFS. While write performances are excellent also on smaller files, I find there is a drop down in performance on reading 20KB files. Performance can go as low as 200MB/sec or even less. >>> >>> Getting 200 MB/s with 20kB files means you have to do 1 metadata >>> ops/s. Don't want to say it is impossible to get more than that, but at >>> least with MDT on ZFS this doesn't sound bad either. Did you run an >>> mdtest on your system? Maybe some serious tuning of MD performance is in >>> order. >> >> I'd agree with Oliver that getting 200MB/s with 20KB files is not too bad. >> Are you using HDDs or SSDs for the MDT and OST devices? If using HDDs, >> are you using SSD L2ARC to allow the metadata and file data be cached in >> L2ARC, and allowing enough time for L2ARC to be warmed up? >> >> Are you using TCP or IB networking? If using TCP then there is a lower >> limit on the number of RPCs that can be handled compared to IB. > > Yes Andreas perhaps is not too bad and in my particular situation I am > reading bunch of 20KB chunks inside a bigger 200GB file. > I found benefits reducing the ZFS record size that was set up at the > beginning to a quite large value. For large streaming writes recordsize=1024k will give the best performance, but this is not very good for small random IO. It is not currently possible to explicitly change the blocksize on a per-file basis. However, there is some interest to be able to change this in the future with the "lfs ladvise" tunable. See https://jira.hpdd.intel.com/browse/LU-7225 if you are interested to contribute to this work. > I am using SSD disks and I did not set up a L2ARC because I do not think I'd > have much benefit in my siutation. So it is not a Lustre problem at all. > > thank you I did not know about LU-4865 The LU-4865 patch is mostly useful for starting with smaller blocksize for small files, or files written with random IO patterns. It will not necessarily help if the file is written sequentially and then read in a random pattern. Cheers, Andreas ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] poor performance on reading small files
thank you I did not know about LU-4865 On 03/08/16 17:01, Dilger, Andreas wrote: On Aug 3, 2016, at 12:32, Jeff Johnsonwrote: On 8/3/16 10:57 AM, Dilger, Andreas wrote: On Jul 29, 2016, at 03:33, Oliver Mangold wrote: On 29.07.2016 04:19, Riccardo Veraldi wrote: I am using lustre on ZFS. While write performances are excellent also on smaller files, I find there is a drop down in performance on reading 20KB files. Performance can go as low as 200MB/sec or even less. Getting 200 MB/s with 20kB files means you have to do 1 metadata ops/s. Don't want to say it is impossible to get more than that, but at least with MDT on ZFS this doesn't sound bad either. Did you run an mdtest on your system? Maybe some serious tuning of MD performance is in order. I'd agree with Oliver that getting 200MB/s with 20KB files is not too bad. Are you using HDDs or SSDs for the MDT and OST devices? If using HDDs, are you using SSD L2ARC to allow the metadata and file data be cached in L2ARC, and allowing enough time for L2ARC to be warmed up? Are you using TCP or IB networking? If using TCP then there is a lower limit on the number of RPCs that can be handled compared to IB. Also consider that 20KB of data per lnet RPC, assuming a 1MB RPC, to move 20KB files at 200MB/sec into a non-striped LFS directory you are using EDR for lnet? 100GB Ethernet? It should be clarified that even if the maximum RPC size is 1MB, Lustre will not send more data than actually contained in the file (subject to the page size granularity of 4KB). However, one caveat below for ZFS... One potential issue if using ZFS with recordsize=1024k is used on the OSTs then without patch http://review.whamcloud.com/18441 "LU-4865 zfs: grow block size by write pattern" the blocksize will always be 1MB on the OSTs. If you are storing a large number of small files then this is probably not the most efficient use of space, and it will inflate the amount of data sent over the network as well. Better to either apply that patch locally (and provide feedback on how it is working), or select a recordsize that better matches your file size (e.g. 64KB or 128KB). Cheers, Andreas ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] poor performance on reading small files
On 03/08/16 10:57, Dilger, Andreas wrote: On Jul 29, 2016, at 03:33, Oliver Mangoldwrote: On 29.07.2016 04:19, Riccardo Veraldi wrote: I am using lustre on ZFS. While write performances are excellent also on smaller files, I find there is a drop down in performance on reading 20KB files. Performance can go as low as 200MB/sec or even less. Getting 200 MB/s with 20kB files means you have to do 1 metadata ops/s. Don't want to say it is impossible to get more than that, but at least with MDT on ZFS this doesn't sound bad either. Did you run an mdtest on your system? Maybe some serious tuning of MD performance is in order. I'd agree with Oliver that getting 200MB/s with 20KB files is not too bad. Are you using HDDs or SSDs for the MDT and OST devices? If using HDDs, are you using SSD L2ARC to allow the metadata and file data be cached in L2ARC, and allowing enough time for L2ARC to be warmed up? Are you using TCP or IB networking? If using TCP then there is a lower limit on the number of RPCs that can be handled compared to IB. Cheers, Andreas Yes Andreas perhaps is not too bad and in my particular situation I am reading bunch of 20KB chunks inside a bigger 200GB file. I found benefits reducing the ZFS record size that was set up at the beginning to a quite large value. I am using SSD disks and I did not set up a L2ARC because I do not think I'd have much benefit in my siutation. So ti is not a Lustre problem at all. thank you! Riccardo ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] poor performance on reading small files
MY issue is related on reading bunch of 20KB slices inside a bigger 200GB side. I found out it is not related to Lustre but to ZFS. So I set up ZFS with proper record size and the problem looks like to be mitigated. Thanks for your hints. Riccardo On 03/08/16 08:32, Mohr Jr, Richard Frank (Rick Mohr) wrote: Do you have the Lustre read caching feature enabled? I think it should be on by default, but you might want to check. If the files are only 20 KB, then I would think the Lustre OSS nodes could keep them in memory most of the time to speed up access (unless of course this is a metadata bottleneck as Oliver suggested.) Do your OSS nodes have a lot of memory? Do you know what your typical memory usage is on the OSS nodes? -- Rick Mohr Senior HPC System Administrator National Institute for Computational Sciences http://www.nics.tennessee.edu On Jul 28, 2016, at 10:19 PM, Riccardo Veraldiwrote: Hello, I have a lustre cluster on rhel7, 6 OSS each of them has 3 OSTs and 1 MDS. I am using lustre on ZFS. While write performances are excellent also on smaller files, I find there is a drop down in performance on reading 20KB files. Performance can go as low as 200MB/sec or even less. I am talking about random reads and random stride reads. I did the following to try to improve things: • disabled lnet debug messages: • sysctl -w lnet.debug=0 • increased dirty cache • lctl set_param osc.lutrefs\*.max_dirty_mb=256 • increased number of RPC in flight • for i in `ls /proc/fs/lustre/osc/lustrefs-OST00*/max_rpcs_in_flight`; do echo 32 > $i; done it did not improve reading 20KB file performances. I have to say in advance I did not set up any striping because I will have no more than 6 concurrent reads and writes, so striping is not that much important for me. Here the problem is that one single random read of a 20KB file is around 190MB/s and this is really disappointing. I am open to any suggestion. I made sure it is not a ZFS problem, on the OSSs ZFS is performing like a charm locally. thank you Riccardo ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] poor performance on reading small files
On Aug 3, 2016, at 12:32, Jeff Johnsonwrote: > > On 8/3/16 10:57 AM, Dilger, Andreas wrote: >> On Jul 29, 2016, at 03:33, Oliver Mangold >> wrote: >>> On 29.07.2016 04:19, Riccardo Veraldi wrote: I am using lustre on ZFS. While write performances are excellent also on smaller files, I find there is a drop down in performance on reading 20KB files. Performance can go as low as 200MB/sec or even less. >>> Getting 200 MB/s with 20kB files means you have to do 1 metadata >>> ops/s. Don't want to say it is impossible to get more than that, but at >>> least with MDT on ZFS this doesn't sound bad either. Did you run an >>> mdtest on your system? Maybe some serious tuning of MD performance is in >>> order. >> I'd agree with Oliver that getting 200MB/s with 20KB files is not too bad. >> Are you using HDDs or SSDs for the MDT and OST devices? If using HDDs, >> are you using SSD L2ARC to allow the metadata and file data be cached in >> L2ARC, and allowing enough time for L2ARC to be warmed up? >> >> Are you using TCP or IB networking? If using TCP then there is a lower >> limit on the number of RPCs that can be handled compared to IB. > > Also consider that 20KB of data per lnet RPC, assuming a 1MB RPC, to move > 20KB files at 200MB/sec into a non-striped LFS directory you are using EDR > for lnet? 100GB Ethernet? It should be clarified that even if the maximum RPC size is 1MB, Lustre will not send more data than actually contained in the file (subject to the page size granularity of 4KB). However, one caveat below for ZFS... One potential issue if using ZFS with recordsize=1024k is used on the OSTs then without patch http://review.whamcloud.com/18441 "LU-4865 zfs: grow block size by write pattern" the blocksize will always be 1MB on the OSTs. If you are storing a large number of small files then this is probably not the most efficient use of space, and it will inflate the amount of data sent over the network as well. Better to either apply that patch locally (and provide feedback on how it is working), or select a recordsize that better matches your file size (e.g. 64KB or 128KB). Cheers, Andreas ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] two lustre fs on same lnet was: Re: lustre clients cannot access different OSS groups on TCP and infiniband at same time
Hi Andreas, the network names need to be unique if the same clients are connecting to both filesystems. What are complication having two lustre filesystems on the same lnet on the same IB? Does it have performance impact (broadcasts, credits, buffers)? We have two (three) lustre fs facing clusters on the same lnet. I'm thinking if I need to change that - we have service window right now. Initially I set up separate lnets for each lustre but as we were doing ethernet 'bridge" lnet1(ib)-rtr-eth-rtr-lnet2(ib) to remote cluster between IB networks the routing get kind of complicated. As a practical matter we were able to move 1PB of data between two lustres plus IO from/to compute cluster in this configuration. We have one lnet per IB fabric: router(eth) -- tcp11...tcp14 -- (eth)routers - o2ib2 -- cluster2 | +-- lustre1 + | | cluster0 -{o2ib0} -- lustre2 --{o2ib1} - cluster1 | | +-- lustre3 + Right now we are merging clusters 1 and 2 and retire lustre1; It can be good time to reconsider and split lnets like o2ib0 -> (o2ib20,oi2ib30) and o2ib1->(o2ib30,o2ib31). What would be a reason for such lnet split? Alex. On Jul 13, 2016, at 8:15 PM, Dilger, Andreas> wrote: It sounds like you have two different filesystems, each using the same LNet networks "tcp0" and "o2ib0". While "tcp" is a shorthand for network "tcp0", the network names need to be unique if the same clients are connecting to both filesystems. One of the filesystems will need to regenerate the configuration to use "tcp1" and "o2ib1" (or whatever) to allow the clients to distinguish between the different networks. Cheers, Andreas ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] poor performance on reading small files
> On Aug 3, 2016, at 1:30 PM, Ben Evanswrote: > > I thought read caching was disabled by default, as the kernel's default > handling of pages was better. You might be right. It has been a while since I have set up a Lustre file system from scratch, and I haven’t done so for newer versions of Lustre. So it’s possible I am mistaken about the defaults. -- Rick Mohr Senior HPC System Administrator National Institute for Computational Sciences http://www.nics.tennessee.edu ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] poor performance on reading small files
Also note: If you are using IB, these small reads will make use of RDMA. LNet only uses rdma_writes (historical reasons for this) so the client has to use IB immediate messages to tell the server to write the 20kb file to the client. The extra round-trip handshake involved with this will add latency to each file read. That could be why writes, which don’t need this extra handshake, perform better than the reads. The bigger the files (i.e. the more data moved per rdma_write) the less the additional overhead of the handshake will be noticed. Doug > On Aug 3, 2016, at 11:32 AM, Jeff Johnson> wrote: > > On 8/3/16 10:57 AM, Dilger, Andreas wrote: >> On Jul 29, 2016, at 03:33, Oliver Mangold >> wrote: >>> On 29.07.2016 04:19, Riccardo Veraldi wrote: I am using lustre on ZFS. While write performances are excellent also on smaller files, I find there is a drop down in performance on reading 20KB files. Performance can go as low as 200MB/sec or even less. >>> Getting 200 MB/s with 20kB files means you have to do 1 metadata >>> ops/s. Don't want to say it is impossible to get more than that, but at >>> least with MDT on ZFS this doesn't sound bad either. Did you run an >>> mdtest on your system? Maybe some serious tuning of MD performance is in >>> order. >> I'd agree with Oliver that getting 200MB/s with 20KB files is not too bad. >> Are you using HDDs or SSDs for the MDT and OST devices? If using HDDs, >> are you using SSD L2ARC to allow the metadata and file data be cached in >> L2ARC, and allowing enough time for L2ARC to be warmed up? >> >> Are you using TCP or IB networking? If using TCP then there is a lower >> limit on the number of RPCs that can be handled compared to IB. >> >> Cheers, Andreas >> ___ >> lustre-discuss mailing list >> lustre-discuss@lists.lustre.org >> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > > Also consider that 20KB of data per lnet RPC, assuming a 1MB RPC, to move > 20KB files at 200MB/sec into a non-striped LFS directory you are using EDR > for lnet? 100GB Ethernet? > > --Jeff > > > -- > -- > Jeff Johnson > Co-Founder > Aeon Computing > > jeff.john...@aeoncomputing.com > www.aeoncomputing.com > t: 858-412-3810 x1001 f: 858-412-3845 > m: 619-204-9061 > > 4170 Morena Boulevard, Suite D - San Diego, CA 92117 > > High-performance Computing / Lustre Filesystems / Scale-out Storage > > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] poor performance on reading small files
On 8/3/16 10:57 AM, Dilger, Andreas wrote: On Jul 29, 2016, at 03:33, Oliver Mangoldwrote: On 29.07.2016 04:19, Riccardo Veraldi wrote: I am using lustre on ZFS. While write performances are excellent also on smaller files, I find there is a drop down in performance on reading 20KB files. Performance can go as low as 200MB/sec or even less. Getting 200 MB/s with 20kB files means you have to do 1 metadata ops/s. Don't want to say it is impossible to get more than that, but at least with MDT on ZFS this doesn't sound bad either. Did you run an mdtest on your system? Maybe some serious tuning of MD performance is in order. I'd agree with Oliver that getting 200MB/s with 20KB files is not too bad. Are you using HDDs or SSDs for the MDT and OST devices? If using HDDs, are you using SSD L2ARC to allow the metadata and file data be cached in L2ARC, and allowing enough time for L2ARC to be warmed up? Are you using TCP or IB networking? If using TCP then there is a lower limit on the number of RPCs that can be handled compared to IB. Cheers, Andreas ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org Also consider that 20KB of data per lnet RPC, assuming a 1MB RPC, to move 20KB files at 200MB/sec into a non-striped LFS directory you are using EDR for lnet? 100GB Ethernet? --Jeff -- -- Jeff Johnson Co-Founder Aeon Computing jeff.john...@aeoncomputing.com www.aeoncomputing.com t: 858-412-3810 x1001 f: 858-412-3845 m: 619-204-9061 4170 Morena Boulevard, Suite D - San Diego, CA 92117 High-performance Computing / Lustre Filesystems / Scale-out Storage ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] poor performance on reading small files
On Jul 29, 2016, at 03:33, Oliver Mangoldwrote: > > On 29.07.2016 04:19, Riccardo Veraldi wrote: >> >> I am using lustre on ZFS. >> >> While write performances are excellent also on smaller files, I find >> there is a drop down in performance >> on reading 20KB files. Performance can go as low as 200MB/sec or even >> less. > > Getting 200 MB/s with 20kB files means you have to do 1 metadata > ops/s. Don't want to say it is impossible to get more than that, but at > least with MDT on ZFS this doesn't sound bad either. Did you run an > mdtest on your system? Maybe some serious tuning of MD performance is in > order. I'd agree with Oliver that getting 200MB/s with 20KB files is not too bad. Are you using HDDs or SSDs for the MDT and OST devices? If using HDDs, are you using SSD L2ARC to allow the metadata and file data be cached in L2ARC, and allowing enough time for L2ARC to be warmed up? Are you using TCP or IB networking? If using TCP then there is a lower limit on the number of RPCs that can be handled compared to IB. Cheers, Andreas ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] poor performance on reading small files
I thought read caching was disabled by default, as the kernel's default handling of pages was better. It's been awhile since I looked at those test results. On 8/3/16, 11:32 AM, "lustre-discuss on behalf of Mohr Jr, Richard Frank (Rick Mohr)"wrote: >Do you have the Lustre read caching feature enabled? I think it should >be on by default, but you might want to check. If the files are only 20 >KB, then I would think the Lustre OSS nodes could keep them in memory >most of the time to speed up access (unless of course this is a metadata >bottleneck as Oliver suggested.) Do your OSS nodes have a lot of memory? > Do you know what your typical memory usage is on the OSS nodes? > >-- >Rick Mohr >Senior HPC System Administrator >National Institute for Computational Sciences >http://www.nics.tennessee.edu > > >> On Jul 28, 2016, at 10:19 PM, Riccardo Veraldi >> wrote: >> >> Hello, >> >> I have a lustre cluster on rhel7, 6 OSS each of them has 3 OSTs and 1 >>MDS. >> >> I am using lustre on ZFS. >> While write performances are excellent also on smaller files, I find >>there is a drop down in performance >> on reading 20KB files. Performance can go as low as 200MB/sec or even >>less. >> I am talking about random reads and random stride reads. >> I did the following to try to improve things: >> € disabled lnet debug messages: >> € sysctl -w lnet.debug=0 >> € increased dirty cache >> € lctl set_param osc.lutrefs\*.max_dirty_mb=256 >> € increased number of RPC in flight >> € for i in `ls >>/proc/fs/lustre/osc/lustrefs-OST00*/max_rpcs_in_flight`; do echo 32 > >>$i; done >> it did not improve reading 20KB file performances. >> I have to say in advance I did not set up any striping because I will >>have no more than 6 concurrent reads and writes, >> so striping is not that much important for me. >> Here the problem is that one single random read of a 20KB file is >>around 190MB/s and this is really disappointing. >> I am open to any suggestion. >> I made sure it is not a ZFS problem, on the OSSs ZFS is performing like >>a charm locally. >> thank you >> >> >> Riccardo >> >> >> ___ >> lustre-discuss mailing list >> lustre-discuss@lists.lustre.org >> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > > >___ >lustre-discuss mailing list >lustre-discuss@lists.lustre.org >http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] poor performance on reading small files
Do you have the Lustre read caching feature enabled? I think it should be on by default, but you might want to check. If the files are only 20 KB, then I would think the Lustre OSS nodes could keep them in memory most of the time to speed up access (unless of course this is a metadata bottleneck as Oliver suggested.) Do your OSS nodes have a lot of memory? Do you know what your typical memory usage is on the OSS nodes? -- Rick Mohr Senior HPC System Administrator National Institute for Computational Sciences http://www.nics.tennessee.edu > On Jul 28, 2016, at 10:19 PM, Riccardo Veraldi >wrote: > > Hello, > > I have a lustre cluster on rhel7, 6 OSS each of them has 3 OSTs and 1 MDS. > > I am using lustre on ZFS. > While write performances are excellent also on smaller files, I find there is > a drop down in performance > on reading 20KB files. Performance can go as low as 200MB/sec or even less. > I am talking about random reads and random stride reads. > I did the following to try to improve things: > • disabled lnet debug messages: > • sysctl -w lnet.debug=0 > • increased dirty cache > • lctl set_param osc.lutrefs\*.max_dirty_mb=256 > • increased number of RPC in flight > • for i in `ls > /proc/fs/lustre/osc/lustrefs-OST00*/max_rpcs_in_flight`; do echo 32 > $i; done > it did not improve reading 20KB file performances. > I have to say in advance I did not set up any striping because I will have no > more than 6 concurrent reads and writes, > so striping is not that much important for me. > Here the problem is that one single random read of a 20KB file is around > 190MB/s and this is really disappointing. > I am open to any suggestion. > I made sure it is not a ZFS problem, on the OSSs ZFS is performing like a > charm locally. > thank you > > > Riccardo > > > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org