Henry, The benchmark, any benchmark is as good as the profile of the workload it wants to represent as close as possible as many applications behavior. It is obvious that there are applications running on Lustre that are not representative to typical usage of Lustre FS. The art here is to find the best workload generator that will represent the majority of the apps using Lustre. Yet when a users selects a Lustre FS server he needs to give them a clear picture if Lustre is the right FS as well as which Lustre vendor has a better implementation. It is almost impossible for a user to use a benchmark for a single type of application. What majority of users do is build their own micro-benchmark specific to the application they us, of course if they have a real workload characterization of the application. In my experience many apps do not have a definition of the workload and prefer to select servers using a more general benchmark such as sfs, SPC, and others. This is the value of published benchmark data that are general enough to allow to compare different implementations. As a disclaimer a benchmark cannot ever represent all the behavior of all the apps using a FS but can represent accurately the average goodness of a FS implementation assuming a typical average usage of the FS. That being said I need the help of the LUG community to build the best most representative average workload definition for sfs2013. I hope this explains the reason behind using general benchmarks and sfs2013 in particular
Sorin Faibish Distinguished Engineer EMC - Office of CTO -----Original Message----- From: Henry Newman [mailto:[email protected]] Sent: Saturday, March 24, 2012 10:39 AM To: faibish, sorin; [email protected]; [email protected]; [email protected]; [email protected]; [email protected] Subject: RE: Benchmarking Working Group Have some SPEC data is interesting in my opinion only if the workload maps to user application. Much of Lustre usage from what I have seen is defensive I/O and therefore is write intensive. How does that map is the question at hand. Thanks hsn ________________________________________ Henry Newman | CEO/CTO Instrumental, Inc | High Performance Innovation 1450 Energy Park Drive Suite 375 St. Paul, MN 55108-5274 Direct: 651-280-4801 Fax: 651-280-4839 Toll-free: 1.800.886.6188 E-Mail: [email protected] Web: http://www.instrumental.com STRICTLY PERSONAL AND CONFIDENTIAL. This email may contain confidential and proprietary material for the sole use of the intended recipient. Any review or distribution by others is strictly prohibited. If you are not the intended recipient please contact the sender and delete all copies. -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of [email protected] Sent: Friday, March 23, 2012 4:35 PM To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected] Subject: Re: [Openbenchmark] Benchmarking Working Group -----Original Message----- From: [email protected] [mailto:[email protected]] Sent: Friday, March 23, 2012 4:55 PM To: faibish, sorin; [email protected]; [email protected]; [email protected]; [email protected] Subject: RE: Benchmarking Working Group Sorin, Is the intent to add some benchmarks that utilize the ideas and concepts of sfs2013 to the list? Or is the intent to possible add sfs2013 itself to the list of benchmarks? [SF] My intention was to include Lustre as one of the workloads of sfs2013 as a special profile to the benchmark. Regardless, I was approved by sfs committee to add a Lustre specific profile to sfs2013. I hope with the help of the Lustre community to create a realistic and representative profile to be included in sfs2013 additional to other benchmarks that this group will decide to do. The advantage of such a benchmark will be that vendors can publish results officially to the SPEC.org reviewed by a peer audit group. Hope this makes sense Sorin Faibish Distinguished Engineer EMC - Office of CTO Thanks! Dr. Jeff Layton HPC Enterprise Technologist Dell | Research Computing mobile + 1 678 427-5819 [email protected] -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of [email protected] Sent: Friday, March 23, 2012 1:12 PM To: [email protected]; [email protected]; [email protected]; [email protected] Subject: Re: [Openbenchmark] Benchmarking Working Group Per the discussion during the call, attached please find the documentation of sfs2013 benchmark (currently called netmist). Please take a look and let me know if it is of interest to continue on this path of action for the Lustre benchmark. I will be attending the LUG and plan to introduce sfs2013 benchmark. Thank you for your understanding and patience Sorin Faibish Distinguished Engineer EMC - Office of CTO _____________________________________________ From: Vanderbilt, Richard [mailto:[email protected]] Sent: Monday, March 19, 2012 12:25 PM To: [email protected]; [email protected]; [email protected]; faibish, sorin Subject: [Openbenchmark] Benchmarking Working Group When: Friday, March 23, 2012 11:30 AM-12:30 PM (GMT-05:00) Eastern Time (US & Canada). Where: 1(866)844-9418 id 4088226016 The OpenSFS Benchmarking Workgroup aims to provide an I/O benchmark suite to satisfy the benchmarking requirements of the scalable parallel file system users and facilities. Towards this end, as a first step, the workgroup aims to characterize the I/O workloads from small- to very large-scale scalable parallel file systems deployed at various high-performance computing and parallel computing facilities and institutions. Using these gathered characteristics, the workgroup will then collect and build the required I/O benchmarks to emulate these workloads and provide required documentation about the benchmark suite. The workgroup meeting will reoccur every Friday at 8:30am Pacific time. 1 hour each. The dial-in is as follows: 1(866)844-9418 id# 4088226016. Also note: we will be having our first face to face meeting on Sunday April 22nd from 2:00-5:00pm at the LUG. Details to come. On Tuesday April 24 4:10-5:00pm there will be a Benchmark panel discussion moderated by Sarp Oral. Panelist will be Jeremy Filizetti (DOD), Andrew Uselton (NERSC), Eric Barton (Whamcloud), Nathan Rutman (Xyratex), Stephane Thiell (CEA), and Stephen Simms (IU). If you are interested in contributing to this work group, please email Sarp Oral at [email protected]<mailto:[email protected]> or myself at [email protected]<mailto:[email protected]>. Sincerely, Sarp Oral, PhD National Center for Computational Sciences Oak Ridge National Laboratory [email protected]<mailto:[email protected]> 865-574-2173 Richard Vanderbilt Co-chair Netapp Inc. (408)822-6016 << File: ATT00001.txt >> _______________________________________________ Openbenchmark mailing list [email protected] http://lists.opensfs.org/listinfo.cgi/openbenchmark-opensfs.org STRICTLY PERSONAL AND CONFIDENTIAL. This email may contain confidential and proprietary material for the sole use of the intended recipient. Any review or distribution by others is strictly prohibited. If you are not the intended recipient please contact the sender and delete all copies. _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
