Re: [Lustre-discuss] High CPU load, only on 1 OSS
All of our OSS's are configured the same. However on the first OSS which is experiencing the high CPU load the number of io threads being started is much higher than the other OSS servers. The server that is experiencing high CPU ost.OSS.ost_io.threads_started=175 Other OSS servers ost.OSS.ost_io.threads_started=128 All off the OSS servers are configured like this. ost.OSS.ost_io.threads_min=128 ost.OSS.ost_io.threads_max=512 Any direction/information to resolve this issue is greatly appreciated. If any other info is needed please let me know. Rocky From: Wang Yibin wang.yi...@oracle.com To: Ronald K Long rkl...@usgs.gov Cc: lustre-discuss@lists.lustre.org Date: 11/16/2010 09:54 AM Subject: Re: [Lustre-discuss] High CPU load, only on 1 OSS Hello, Normally when stripe_offset is set to -1, MDS will do load/space balancing automatically. What is your use pattern of the filesystem? It sounds like that your applications are doing extensive I/O on that particular OSS. To find out why the load on the OSS is so high, please - find what processes are hogging the CPUs using top(1). - get the stripe info of your in-use files to see whether most of them reside on the same OSS. If the files in use are not distributed among the OSS servers, or your file usage pattern is one-OSS bound, you may want to consider tuning the stripe_count/stripe_size. 在 2010-11-16，下午10:38， Ronald K Long 写道： We recently setup a lustre config. 1 MDS 4 OSS's. Everything is running fine except on the first OSS we are experiencing very high cpu load. The first OSS is running a CPU load in the high 50's. The other 3 OSS's are steady at around 8. Everything is the same between all of the OSS's. The stripe is setup stripe_count: 1 stripe_offset: -1 Red Hat 5 64bit kernel-2.6.18-194.3.1.el5_lustre.1.8.4 kernel-devel-2.6.18-194.3.1.el5_lustre.1.8.4 lustre-ldiskfs-3.1.3-2.6.18_194.3.1.el5_lustre.1.8.4 lustre-1.8.4-2.6.18_194.3.1.el5_lustre.1.8.4 lustre-modules-1.8.4-2.6.18_194.3.1.el5_lustre.1.8.4 Any thing I can check on the problem OSS to rectify this issue. Thank you in advance Rocky ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] High CPU load, only on 1 OSS
We recently setup a lustre config. 1 MDS 4 OSS's. Everything is running fine except on the first OSS we are experiencing very high cpu load. The first OSS is running a CPU load in the high 50's. The other 3 OSS's are steady at around 8. Everything is the same between all of the OSS's. The stripe is setup stripe_count: 1 stripe_offset: -1 Red Hat 5 64bit kernel-2.6.18-194.3.1.el5_lustre.1.8.4 kernel-devel-2.6.18-194.3.1.el5_lustre.1.8.4 lustre-ldiskfs-3.1.3-2.6.18_194.3.1.el5_lustre.1.8.4 lustre-1.8.4-2.6.18_194.3.1.el5_lustre.1.8.4 lustre-modules-1.8.4-2.6.18_194.3.1.el5_lustre.1.8.4 Any thing I can check on the problem OSS to rectify this issue. Thank you in advance Rocky ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] High CPU load, only on 1 OSS
Hello, Normally when stripe_offset is set to -1, MDS will do load/space balancing automatically. What is your use pattern of the filesystem? It sounds like that your applications are doing extensive I/O on that particular OSS. To find out why the load on the OSS is so high, please - find what processes are hogging the CPUs using top(1). - get the stripe info of your in-use files to see whether most of them reside on the same OSS. If the files in use are not distributed among the OSS servers, or your file usage pattern is one-OSS bound, you may want to consider tuning the stripe_count/stripe_size. 在 2010-11-16，下午10:38， Ronald K Long 写道： We recently setup a lustre config. 1 MDS 4 OSS's. Everything is running fine except on the first OSS we are experiencing very high cpu load. The first OSS is running a CPU load in the high 50's. The other 3 OSS's are steady at around 8. Everything is the same between all of the OSS's. The stripe is setup stripe_count: 1 stripe_offset: -1 Red Hat 5 64bit kernel-2.6.18-194.3.1.el5_lustre.1.8.4 kernel-devel-2.6.18-194.3.1.el5_lustre.1.8.4 lustre-ldiskfs-3.1.3-2.6.18_194.3.1.el5_lustre.1.8.4 lustre-1.8.4-2.6.18_194.3.1.el5_lustre.1.8.4 lustre-modules-1.8.4-2.6.18_194.3.1.el5_lustre.1.8.4 Any thing I can check on the problem OSS to rectify this issue. Thank you in advance Rocky ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
