Re: [SLUG] Petabytes on a budget

[Peter Chubb]

> "Jake" == Jake Anderson  writes:

Jake> On 03/09/09 16:09, Martin Visser wrote:
>> Jake,
>> 
>> Most of thenumbers you are showing are just the clocked speeds on
>> busses and cables.
Jake> actually no they are benchmarked sustained and average transfer
Jake> rates for the devices.  sata "2" line speed is 3gbit, with
Jake> overhead it will saturate at 250mbyte/sec the hdd's min transfer
Jake> rate is 60mbyte/sec and max is 160.

>> Certainly the components can clock at those speeds, but the biggest
>> issue with lower-end components is whether they can actually feed
>> and sustain data at that rate, and how well they handle contention
>> for resources. Generally this comes down to size of buffers, and
>> whether have fast or wide enough processors at those interface
>> points.
>> 
Jake> They aren't really "lower end components" There is little
Jake> difference in performance between SATA and SAS these days.
Jake> PCI/PCI-X will sustain those transfer rates by design, the cards
Jake> themselves are being little more than a slightly bent pipe so
Jake> there should be no real bottleneck there.

We've seen some problems with some SATA port multipliere, that don't
allow more than one drive at a time to be working --- effectively
slowing the transfer rate to that of a single spindle.  I don't know
if the ones these people are using have that problem.

--
Dr Peter Chubb  http://www.gelato.unsw.edu.au  peterc AT gelato.unsw.edu.au
http://www.ertos.nicta.com.au   ERTOS within National ICT Australia
-- 
SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/
Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html

Re: [SLUG] Petabytes on a budget

[Jake Anderson]

On 03/09/09 16:09, Martin Visser wrote:

Jake,

Most of thenumbers you are showing are just the clocked speeds on busses and
cables.
actually no they are benchmarked sustained and average transfer rates 
for the devices.

sata "2" line speed is 3gbit, with overhead it will saturate at 250mbyte/sec
the hdd's min transfer rate is 60mbyte/sec and max is 160.


  Certainly the components can clock at those speeds, but the biggest
issue with lower-end components is whether they can actually feed and
sustain data at that rate, and how well they handle contention for
resources. Generally this comes down to size of buffers, and whether have
fast or wide enough processors at those interface points.
   

They aren't really "lower end components"
There is little difference in performance between SATA and SAS these days.
PCI/PCI-X will sustain those transfer rates by design, the cards 
themselves are being little more than a slightly bent pipe so there 
should be no real bottleneck there.



But it sounds like for the business Backblaze is in they are building
something that is big rather than fast (or at least something that is fast
enough). And certainly if you can parallelize the system enough (and
maintain reliablity) then you probably even achieve cheap and fast.
   

Their biggest issue would be seek times.

Regards, Martin

martinvisse...@gmail.com


On Thu, Sep 3, 2009 at 12:40 PM, Jake Andersonwrote:

   

On 03/09/09 10:37, Mark Walkom wrote:

 

I was thinking the same, but I reckon because they are just backing
up/archiving data it wouldn't be too bad.
ie They aren't looking for huge performance, just huge, cheap storage.


2009/9/3 Morgan Storey



   

I know I am a geek but that is hot.
I am wondering if they see any throughput issues with the sata backplanes
and pci sata cards.


 

The backplanes are probably fine, each sata cable is good for
   

~300mbytes/sec most physical disks couldn't hope to hit that.
lesse what their maximum xfer rate is.
each drive can hit 103mb/sec average (better than I thought)
each sata channel will max out at 250mbyte/sec

so they are going to be loosing some bandwidth there.
their backplanes take 5 disks, so a potential bandwidth of well call it
500mbyte/second
so 50% is out the window there actual bandwidth per 5 disks is going to be
250mbyte

they have 9 of these channels for a total bandwith available of
2250mbyte/sec (2 gigabytes a second, that'll rip some dvds fast)

standard PCI tops out at 133mbyte/sec so thats out ;->

it looks like they are using PCI-E SATA cards
the mbo they have and the cards they are using indicate they have 3x of
something like this
http://www.syba.com/index.php?controller=Product&action=Info&Id=861
which maxes out at 250mbyte/sec per (1x PCI-E 1x lane)
and one 4 port card which if it comes from that mob must be a PCI by the
look of things.
but I'll assume that its PCI-E and at least 4 lanes.

so the total xfer rate is 1750mbyte/sec
(or 883 if they are using the PCI card)

Vs the total possible xfer rate of 4500
they aren't doing *too* badly

given that on a gigabit ethernet connection you are lucky to push
30mbyte/sec (or 60 if you tweak it), I think its not going to be a big issue
;->


If they wanted more oomph their best bet would be to put 2x 16 port PCI-E
16x cards into a motherboard that supported it (most decent SLI motherboards
will do that)

better still one with 4x pci-E 16 slots so that you can put some 10gigE
cards in as well something like

http://www.aria.co.uk/Products/Components/Motherboards/Socket+AM3+(AMD)/MSI+790FX-GD70+AM3+Motherboard+with+4+x+PCIe+x+16+?productId=36604
say (but with intel of course ;->)

That should net you (assuming you use dual port 10gig-E nics) an xfer rate
out of the box of around 2400mbytes/sec
almost fast enough to spy on teh entirez intarwebz!!

--
SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/
Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html

 



On 03/09/09 16:09, Martin Visser wrote:

Jake,

Most of thenumbers you are showing are just the clocked speeds on busses and
cables. Certainly the components can clock at those speeds, but the biggest
issue with lower-end components is whether they can actually feed and
sustain data at that rate, and how well they handle contention for
resources. Generally this comes down to size of buffers, and whether have
fast or wide enough processors at those interface points.

But it sounds like for the business Backblaze is in they are building
something that is big rather than fast (or at least something that is fast
enough). And certainly if you can parallelize the system enough (and
maintain reliablity) then you probably even achieve cheap and fast.

Regards, Martin

martinvisse...@gmail.com


On Thu, Sep 3, 2009 at 12:40 PM, Jake Andersonwrote: