On 05/08/2014 01:43 PM, Lockwood, Glenn wrote:
On May 8, 2014, at 10:30 AM, Ellis H. Wilson III <[email protected]>
wrote:
On 05/08/2014 10:29 AM, John Hearns wrote:
Forget building compute clusters - soon we will be building
Beowulfs with disk drives!
Color me dubious. I highly doubt there will be any entire clusters
of just HDDs anytime soon. The cpu/ram you can fit on them will be
far lower than a full machine, even if you consider 16 of them or
so.
This is the end goal of Hadoop clusters. Not everything needs fast
CPUs and a ton of RAM.
Maybe for some uses of Hadoop. Not all, from my experience at least.
Simpler things will benefit enormously. High-shuffling MR applications
(e.g., sort) will kick and scream when they have a factor or more
machines to swap data with that are now slower. Note that in the paper
I shared the researchers look at MR specifically for on-SSD computation.
But you're right -- for very straightforward, I/O-heavy and CPU-berift
workloads, a cluster of these could exist. I guess I continue to
believe the usability of a traditional cluster with SATA/SCSI attached
versions of these (who can still filter) inside would be better. The
key here is being able to do simple queries/filters/etc on-drive, and
just push needed data off-drive.
Is this still a beowulf cluster? Probably not, but these sorts of
devices have a lot of utility in HPC. We make heavy use of iSCSI on
our largest machine, and we'd free up a fair amount of resources if
we didn't have to wrap our iSCSI targets in iSCSI servers.
While they would unarguably be great for Big Data, generally stating
they would be great for HPC is uncertain to me at least (with the
exception of checkpointing). These devices, for better or worse, make
the I/O stack more, not less, indirect. You are just issuing queries
against a KV store instead of plain ol' blocks. Not ideal conditions
for complex I/O patterns and enabling richer locking semantics available
in many robust parallel file systems.
Best,
ellis
--
Ph.D. Candidate
Department of Computer Science and Engineering
The Pennsylvania State University
www.ellisv3.com
_______________________________________________
Beowulf mailing list, [email protected] sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
