[lustre-discuss] parallel write/reads problem

2018-05-10 Thread Riccardo Veraldi 
Hello,
So far I am not able to solve this problem on my Lustre setup.
I can reach very good performance with multi threaded writes or reads,
that are sequential writes and sequential reads at different times.
I can saturate Infiniband FDR capabilities reaching 6GB/s.
The problem rises when while writing I start also reading the same file
or even a different file.
In our I/O model there are writers and readers which start reading files
after a while they're being written. In this case read
performances drops dramatically. Writes go up to 6GB/s but reads have a
barrier and won't go more than 3GB/s.
I Tried all kind of optimizations. ZFS is performing very well itself,
but when Lustre is on top of it I have this problem.
Infiniband is working at full speed and Lnet test also is at full speed.
So I do not understand while I have concurrent writes/reads the reading
performances go down.

I also tweaked the ko2blnd parameters to gain more parallelism:

options ko2iblnd timeout=100 peer_credits=63 credits=2560
concurrent_sends=63 ntx=2048 fmr_pool_size=1280 fmr_flush_trigger=1024
ntx=5120

then on OSS side:

lctl set_param timeout=600
lctl set_param ldlm_timeout=200
lctl set_param at_min=250
lctl set_param at_max=600

on client side:

lctl set_param osc.*.checksums=0
lctl set_param timeout=600
lctl set_param at_min=250
lctl set_param at_max=600
lctl set_param ldlm.namespaces.*.lru_size=2000
lctl set_param osc.*.max_rpcs_in_flight=64
lctl set_param osc.*.max_dirty_mb=1024
lctl set_param llite.*.max_read_ahead_mb=1024
lctl set_param llite.*.max_cached_mb=81920
lctl set_param llite.*.max_read_ahead_per_file_mb=1024
lctl set_param subsystem_debug=0

I tried to set

lctl set_param osc.*.max_pages_per_rpc=1024

but it is not allowed...

 lctl set_param osc.*.max_pages_per_rpc=1024
error: set_param: setting
/proc/fs/lustre/osc/drplu-OST0001-osc-881ed6b05800/max_pages_per_rpc=1024:
Numerical result out of range


any other idea on what I may work on to get better performance on
concurrent writes/reads ?

thank you


Rick



___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lustre project quotas

2018-05-10 Thread Einar Næss Jensen 
Ok. I probably misunderstood this part:
Introduced in Lustre 2.10
Lustre filesystems formatted with a Lustre release prior to 2.10 can be still 
safely upgraded to release 2.10, but will not have project quota usage 
reporting functional until tune2fs -O project is run against all ldiskfs 
backend targets. This command sets the PROJECT feature flag in the superblock 
and runs e2fsck (as a result, the target must be offline). See Section 24.5, “ 
Quotas and Version Interoperability” for further important considerations.

I assumed this refered to project quotas being enabled as default when 
installing (not upgrading) version 2.10.x

also I was tricked by this output:
root@lustre-mds1 ~]# lctl get_param osd-*.*.quota_slave.info
osd-ldiskfs.LUSTRE1-MDT.quota_slave.info=
target name:LUSTRE1-MDT
pool ID:0
type:   md
quota enabled:  ugp
conn to master: setup
space acct: ug
user uptodate:  glb[1],slv[1],reint[0]
group uptodate: glb[1],slv[1],reint[0]
project uptodate: glb[0],slv[0],reint[0]

which, I thought, showed that quota was enabled for users, groups and projects.

but apparantly not:
Filesystem features:  has_journal ext_attr dir_index filetype 
needs_recovery extent 64bit flex_bg sparse_super large_file huge_file uninit_bg 
dir_nlink quota


Thanks for clarification. I will schedule downtime and activate it.


Best Regards
Einar Næss Jensen
 

-- 
Einar Næss Jensen
NTNU HPC Section
Norwegian University of Science and Technoloy
Address: Høgskoleringen 7i
 N-7491 Trondheim, NORWAY
tlf: +47 90990249
email:   einar.nass.jen...@ntnu.no


From: Shuichi Ihara 
Sent: Thursday, May 10, 2018 9:37 PM
To: Einar Næss Jensen
Cc: Alexander I Kulyavtsev; lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] lustre project quotas

please check 'dumpe2fs -h /dev/xxx | grep 'Filesystem features' if 'project' 
flag is enabled.
If not, you need to add it to all MDT/OSTs. e.g. tune2fs -O project /dev/xxx. 
described in Lustre manual in detail.

Thanks
Ihara

> On May 11, 2018, at 1:02, Einar Næss Jensen  wrote:
>
> ldiskfs
>
> Sent from my iPhone
>
> On 10 May 2018, at 17:26, Alexander I Kulyavtsev  wrote:
>
>> Do you use zfs or ldiskfs on OST?
>> Zfs does not have project quota yet. Alex.
>>
>> From: lustre-discuss  on behalf of 
>> Einar Næss Jensen 
>> Date: Thursday, May 10, 2018 at 7:47 AM
>> To: "lustre-discuss@lists.lustre.org" 
>> Subject: Re: [lustre-discuss] lustre project quotas
>>
>> ​Lustre server is 2.10.1
>> lustre client is 2.10.3
>>
>> --
>> Einar Næss Jensen
>> NTNU HPC Section
>> Norwegian University of Science and Technoloy
>> Address: Høgskoleringen 7i
>>  N-7491 Trondheim, NORWAY
>> tlf: +47 90990249
>> email:   einar.nass.jen...@ntnu.no
>> From: lustre-discuss  on behalf of 
>> Einar Næss Jensen 
>> Sent: Thursday, May 10, 2018 2:45 PM
>> To: lustre-discuss@lists.lustre.org
>> Subject: [lustre-discuss] lustre project quotas
>>
>> Hello.
>>
>> I have sucessfully installed lustre and it works well, but I'm having 
>> trouble figuring out how to enable / set project quotas.
>> How can I verify that project quotas are enabled, and how do I set up 
>> projects and assign directories and users to the projects?
>>
>>
>> Best Regards
>> Einar Næss Jensen
>>
>> --
>> Einar Næss Jensen
>> NTNU HPC Section
>> Norwegian University of Science and Technoloy
>> Address: Høgskoleringen 7i
>>  N-7491 Trondheim, NORWAY
>> tlf: +47 90990249
>> email:   einar.nass.jen...@ntnu.no
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lustre project quotas

2018-05-10 Thread Shuichi Ihara 
please check 'dumpe2fs -h /dev/xxx | grep 'Filesystem features' if 'project' 
flag is enabled.
If not, you need to add it to all MDT/OSTs. e.g. tune2fs -O project /dev/xxx. 
described in Lustre manual in detail.

Thanks
Ihara

> On May 11, 2018, at 1:02, Einar Næss Jensen  wrote:
> 
> ldiskfs
> 
> Sent from my iPhone
> 
> On 10 May 2018, at 17:26, Alexander I Kulyavtsev  wrote:
> 
>> Do you use zfs or ldiskfs on OST?
>> Zfs does not have project quota yet. Alex.
>>  
>> From: lustre-discuss  on behalf of 
>> Einar Næss Jensen 
>> Date: Thursday, May 10, 2018 at 7:47 AM
>> To: "lustre-discuss@lists.lustre.org" 
>> Subject: Re: [lustre-discuss] lustre project quotas
>>  
>> ​Lustre server is 2.10.1
>> lustre client is 2.10.3
>>  
>> -- 
>> Einar Næss Jensen
>> NTNU HPC Section
>> Norwegian University of Science and Technoloy
>> Address: Høgskoleringen 7i
>>  N-7491 Trondheim, NORWAY
>> tlf: +47 90990249
>> email:   einar.nass.jen...@ntnu.no
>> From: lustre-discuss  on behalf of 
>> Einar Næss Jensen 
>> Sent: Thursday, May 10, 2018 2:45 PM
>> To: lustre-discuss@lists.lustre.org
>> Subject: [lustre-discuss] lustre project quotas
>>  
>> Hello.
>>  
>> I have sucessfully installed lustre and it works well, but I'm having 
>> trouble figuring out how to enable / set project quotas.
>> How can I verify that project quotas are enabled, and how do I set up 
>> projects and assign directories and users to the projects?
>>  
>>  
>> Best Regards
>> Einar Næss Jensen
>>  
>> -- 
>> Einar Næss Jensen
>> NTNU HPC Section
>> Norwegian University of Science and Technoloy
>> Address: Høgskoleringen 7i
>>  N-7491 Trondheim, NORWAY
>> tlf: +47 90990249
>> email:   einar.nass.jen...@ntnu.no
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lustre project quotas

2018-05-10 Thread Einar Næss Jensen 
ldiskfs

Sent from my iPhone

On 10 May 2018, at 17:26, Alexander I Kulyavtsev 
mailto:a...@fnal.gov>> wrote:

Do you use zfs or ldiskfs on OST?
Zfs does not have project quota yet. Alex.

From: lustre-discuss 
mailto:lustre-discuss-boun...@lists.lustre.org>>
 on behalf of Einar Næss Jensen 
mailto:einar.nass.jen...@ntnu.no>>
Date: Thursday, May 10, 2018 at 7:47 AM
To: "lustre-discuss@lists.lustre.org" 
mailto:lustre-discuss@lists.lustre.org>>
Subject: Re: [lustre-discuss] lustre project quotas


​Lustre server is 2.10.1

lustre client is 2.10.3


--
Einar Næss Jensen
NTNU HPC Section
Norwegian University of Science and Technoloy
Address: Høgskoleringen 7i
 N-7491 Trondheim, NORWAY
tlf: +47 90990249
email:   einar.nass.jen...@ntnu.no

From: lustre-discuss 
mailto:lustre-discuss-boun...@lists.lustre.org>>
 on behalf of Einar Næss Jensen 
mailto:einar.nass.jen...@ntnu.no>>
Sent: Thursday, May 10, 2018 2:45 PM
To: lustre-discuss@lists.lustre.org
Subject: [lustre-discuss] lustre project quotas


Hello.



I have sucessfully installed lustre and it works well, but I'm having trouble 
figuring out how to enable / set project quotas.

How can I verify that project quotas are enabled, and how do I set up projects 
and assign directories and users to the projects?





Best Regards

Einar Næss Jensen


--
Einar Næss Jensen
NTNU HPC Section
Norwegian University of Science and Technoloy
Address: Høgskoleringen 7i
 N-7491 Trondheim, NORWAY
tlf: +47 90990249
email:   einar.nass.jen...@ntnu.no
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lustre project quotas

2018-05-10 Thread Alexander I Kulyavtsev 
Do you use zfs or ldiskfs on OST?
Zfs does not have project quota yet. Alex.

From: lustre-discuss  on behalf of 
Einar Næss Jensen 
Date: Thursday, May 10, 2018 at 7:47 AM
To: "lustre-discuss@lists.lustre.org" 
Subject: Re: [lustre-discuss] lustre project quotas


​Lustre server is 2.10.1

lustre client is 2.10.3


--
Einar Næss Jensen
NTNU HPC Section
Norwegian University of Science and Technoloy
Address: Høgskoleringen 7i
 N-7491 Trondheim, NORWAY
tlf: +47 90990249
email:   einar.nass.jen...@ntnu.no

From: lustre-discuss  on behalf of 
Einar Næss Jensen 
Sent: Thursday, May 10, 2018 2:45 PM
To: lustre-discuss@lists.lustre.org
Subject: [lustre-discuss] lustre project quotas


Hello.



I have sucessfully installed lustre and it works well, but I'm having trouble 
figuring out how to enable / set project quotas.

How can I verify that project quotas are enabled, and how do I set up projects 
and assign directories and users to the projects?





Best Regards

Einar Næss Jensen


--
Einar Næss Jensen
NTNU HPC Section
Norwegian University of Science and Technoloy
Address: Høgskoleringen 7i
 N-7491 Trondheim, NORWAY
tlf: +47 90990249
email:   einar.nass.jen...@ntnu.no
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lustre project quotas

2018-05-10 Thread Einar Næss Jensen 
?Well, I tried to, at least.:

?

[root@lustre-mds1 ~]# lctl get_param osd-*.*.quota_slave.info
osd-ldiskfs.LUSTRE1-MDT.quota_slave.info=
target name:LUSTRE1-MDT
pool ID:0
type:   md
quota enabled:  ugp
conn to master: setup
space acct: ug
user uptodate:  glb[1],slv[1],reint[0]
group uptodate: glb[1],slv[1],reint[0]
project uptodate: glb[0],slv[0],reint[0]



on client:


projects]# lfs quota -u user /lustre1/
Disk quotas for usr user (uid 2278):
 Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
  /lustre1/ 777802552  1073741824 1073741824   -  182433   0   
0   -
[projects]# lfs quota -p 1 /lustre1/
Unexpected quotactl error: Operation not supported
Disk quotas for prj 1 (pid 1):
 Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
  /lustre1/ [0] [0] [0]   - [0] [0] [0]   -
Some errors happened when getting quota info. Some devices may be not working 
or deactivated. The data in "[]" is inaccurate.





--
Einar Næss Jensen
NTNU HPC Section
Norwegian University of Science and Technoloy
Address: Høgskoleringen 7i
 N-7491 Trondheim, NORWAY
tlf: +47 90990249
email:   einar.nass.jen...@ntnu.no

From: Jones, Peter A 
Sent: Thursday, May 10, 2018 2:51 PM
To: Einar Næss Jensen; lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] lustre project quotas

At the risk of stating the obvious - did you try following the documentation in 
chapter 24 of the manual? http://doc.lustre.org/lustre_manual.xhtml


On 2018-05-10, 5:47 AM, "lustre-discuss on behalf of Einar Næss Jensen" 
mailto:lustre-discuss-boun...@lists.lustre.org>
 on behalf of einar.nass.jen...@ntnu.no> 
wrote:


?Lustre server is 2.10.1

lustre client is 2.10.3


--
Einar Næss Jensen
NTNU HPC Section
Norwegian University of Science and Technoloy
Address: Høgskoleringen 7i
 N-7491 Trondheim, NORWAY
tlf: +47 90990249
email:   einar.nass.jen...@ntnu.no

From: lustre-discuss 
mailto:lustre-discuss-boun...@lists.lustre.org>>
 on behalf of Einar Næss Jensen 
mailto:einar.nass.jen...@ntnu.no>>
Sent: Thursday, May 10, 2018 2:45 PM
To: lustre-discuss@lists.lustre.org
Subject: [lustre-discuss] lustre project quotas


Hello.


I have sucessfully installed lustre and it works well, but I'm having trouble 
figuring out how to enable / set project quotas.

How can I verify that project quotas are enabled, and how do I set up projects 
and assign directories and users to the projects?



Best Regards

Einar Næss Jensen


--
Einar Næss Jensen
NTNU HPC Section
Norwegian University of Science and Technoloy
Address: Høgskoleringen 7i
 N-7491 Trondheim, NORWAY
tlf: +47 90990249
email:   einar.nass.jen...@ntnu.no
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lustre project quotas

2018-05-10 Thread Jones, Peter A 
At the risk of stating the obvious – did you try following the documentation in 
chapter 24 of the manual? http://doc.lustre.org/lustre_manual.xhtml


On 2018-05-10, 5:47 AM, "lustre-discuss on behalf of Einar Næss Jensen" 
mailto:lustre-discuss-boun...@lists.lustre.org>
 on behalf of einar.nass.jen...@ntnu.no> 
wrote:


​Lustre server is 2.10.1

lustre client is 2.10.3


--
Einar Næss Jensen
NTNU HPC Section
Norwegian University of Science and Technoloy
Address: Høgskoleringen 7i
 N-7491 Trondheim, NORWAY
tlf: +47 90990249
email:   einar.nass.jen...@ntnu.no

From: lustre-discuss 
mailto:lustre-discuss-boun...@lists.lustre.org>>
 on behalf of Einar Næss Jensen 
mailto:einar.nass.jen...@ntnu.no>>
Sent: Thursday, May 10, 2018 2:45 PM
To: lustre-discuss@lists.lustre.org
Subject: [lustre-discuss] lustre project quotas


Hello.


I have sucessfully installed lustre and it works well, but I'm having trouble 
figuring out how to enable / set project quotas.

How can I verify that project quotas are enabled, and how do I set up projects 
and assign directories and users to the projects?



Best Regards

Einar Næss Jensen


--
Einar Næss Jensen
NTNU HPC Section
Norwegian University of Science and Technoloy
Address: Høgskoleringen 7i
 N-7491 Trondheim, NORWAY
tlf: +47 90990249
email:   einar.nass.jen...@ntnu.no
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] lustre project quotas

2018-05-10 Thread Einar Næss Jensen 
?Lustre server is 2.10.1

lustre client is 2.10.3


--
Einar Næss Jensen
NTNU HPC Section
Norwegian University of Science and Technoloy
Address: Høgskoleringen 7i
 N-7491 Trondheim, NORWAY
tlf: +47 90990249
email:   einar.nass.jen...@ntnu.no

From: lustre-discuss  on behalf of 
Einar Næss Jensen 
Sent: Thursday, May 10, 2018 2:45 PM
To: lustre-discuss@lists.lustre.org
Subject: [lustre-discuss] lustre project quotas


Hello.


I have sucessfully installed lustre and it works well, but I'm having trouble 
figuring out how to enable / set project quotas.

How can I verify that project quotas are enabled, and how do I set up projects 
and assign directories and users to the projects?



Best Regards

Einar Næss Jensen


--
Einar Næss Jensen
NTNU HPC Section
Norwegian University of Science and Technoloy
Address: Høgskoleringen 7i
 N-7491 Trondheim, NORWAY
tlf: +47 90990249
email:   einar.nass.jen...@ntnu.no
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] lustre project quotas

2018-05-10 Thread Einar Næss Jensen 
Hello.


I have sucessfully installed lustre and it works well, but I'm having trouble 
figuring out how to enable / set project quotas.

How can I verify that project quotas are enabled, and how do I set up projects 
and assign directories and users to the projects?



Best Regards

Einar Næss Jensen


--
Einar Næss Jensen
NTNU HPC Section
Norwegian University of Science and Technoloy
Address: Høgskoleringen 7i
 N-7491 Trondheim, NORWAY
tlf: +47 90990249
email:   einar.nass.jen...@ntnu.no
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org