Mike:
There is something going on with your system, as I am able to touch 500
files in 12.5 seconds and delete them in 8.8 seconds on our cluster.
Did you remove all of ATTR entries from your conf file and restart the
servers?
If not, please do so and then capture the output from the following and
send it to me:
for i in `seq 1 500`; do time touch myfile${i}; done
and then
for i in myfile*; do time rm -f ${i}; done.
Thanks,
Becky
On Fri, May 31, 2013 at 12:02 PM, Michael Robbert <[email protected]>wrote:
> top - 09:54:53 up 6 days, 19:11, 1 user, load average: 0.00, 0.00, 0.00
> Tasks: 156 total, 1 running, 155 sleeping, 0 stopped, 0 zombie
> Cpu(s): 0.1%us, 0.2%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Mem: 12289220k total, 1322196k used, 10967024k free, 85820k buffers
> Swap: 2104432k total, 232k used, 2104200k free, 965636k cached
>
> They all look very similar to this. 232k swap used on all of them
> throughout a touch/rm of 100 files. Ganglia doesn't show any change over
> time with cache on or off.
>
> Mike
>
>
> On 5/31/13 9:30 AM, Becky Ligon wrote:
>
>> Michael:
>>
>> Can you send me a screen shot of "top" from your servers when the
>> metadata is running on the local disk? I'd like to see how much memory
>> is available. I'm wondering if 1GB for your DB cache is too high,
>> possibly causing excessive swapping.
>>
>> Becky
>>
>>
>> On Fri, May 24, 2013 at 6:06 PM, Michael Robbert <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> We recently noticed a performance problem with our OrangeFS server.
>>
>> Here are the server stats:
>> 3 servers, built identically with identical hardware
>>
>> [root@orangefs02 ~]# /usr/sbin/pvfs2-server --version
>> 2.8.7-orangefs (mode: aio-threaded)
>>
>> [root@orangefs02 ~]# uname -r
>> 2.6.18-308.16.1.el5.584g0000
>>
>> 4 core E5603 1.60GHz
>> 12GB of RAM
>>
>> OrangeFS is being served to clients using bmi_tcp over DDR Infiniband.
>> Backend storage is PanFS with 2x10Gig connections on the servers.
>> Performance to the backend looks fine using bonnie++. >100MB/sec
>> write and ~250MB/s read to each stack. ~300 creates/sec.
>>
>> On the OrangeFS clients are running kernel version
>> 2.6.18-238.19.1.el5.
>>
>> The biggest problem I have right now is that delete are taking a
>> long time. Almost 1 sec per file.
>>
>> [root@fatcompute-11-32 L_10_V0.2_eta0.3_wRes___**truncerr1e-11]# find
>> N2/|wc -l
>> 137
>> [root@fatcompute-11-32 L_10_V0.2_eta0.3_wRes___**truncerr1e-11]# time
>>
>> rm -rf N2
>>
>> real 1m31.096s
>> user 0m0.000s
>> sys 0m0.015s
>>
>> Similar results for file creates:
>>
>> [root@fatcompute-11-32 ]# date;for i in `seq 1 50`;do touch
>> file${i};done;date
>> Fri May 24 16:04:17 MDT 2013
>> Fri May 24 16:05:05 MDT 2013
>>
>> What else do you need to know? Which debug flags? What should we be
>> looking at?
>> I don't see any load on the servers and I've restarted server and
>> rebooted server nodes.
>>
>> Thanks for any pointers,
>> Mike Robbert
>> Colorado School of Mines
>>
>>
>> ______________________________**_________________
>> Pvfs2-users mailing list
>>
>> Pvfs2-users@beowulf-**underground.org<[email protected]>
>>
>> <mailto:Pvfs2-users@beowulf-**underground.org<[email protected]>
>> >
>>
>>
>> http://www.beowulf-**underground.org/mailman/**listinfo/pvfs2-users<http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users>
>>
>>
>>
>>
>> --
>> Becky Ligon
>> OrangeFS Support and Development
>> Omnibond Systems
>> Anderson, South Carolina
>>
>>
>
--
Becky Ligon
OrangeFS Support and Development
Omnibond Systems
Anderson, South Carolina
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
