Xu, Jerry wrote:
Hello,
Currently I generate nearly one TB data every few days and I need to pass it
along enterprise network to the storage center attached to my HPC system, I am
thinking about compressing it (most tiff format image data) as much as I can, as
fast as I can before I send it crossing network ... So, I am wondering whether
anyone is familiar with any hardware based accelerator, which can dramatically
improve the compressing procedure.. suggestion for any file system architecture
will be appreciated too.. I have couple of contacts from some vendors but not
sure whether it works as I expected, so if anyone has experience about it and
want to share, it will be really appreciated !
Hi Jerry:
Sounds like a bunch of sequencers or arrays going, and dumping data
to a file system. I am not sure you can get deterministic compression
run times from a compressor, or even deterministic compression ratios on
random (binary) data. I have heard of some "xml accelerators" in the
past (back when XML was considered a good buzzword) that did on-the-fly
compression.
I guess it boils down to if
T(compression) + T(comppressed transfer) <<
T(uncompressed_transfer)
And the cost benefit analysis would focus upon the cost of
T(compression) as compared to faster networks. That is, if you spent
$1000/node more to get a faster fabric, which dropped your transfer time
to 20%, is this better/more cost effective than spending $10,000 or so
on an accelerate that may get 70% file compression, and double the
overall time?
Obviously the above numbers are made up, but you get the idea.
Will look around. If you want to talk to a group doing FPGA stuff in
other markets, let me know and I can hook you up. Just be aware that
this might not be cost/time effective.
Joe
Thanks,
Jerry
Jerry Xu PhD
HPC Scientific Computing Specialist
Enterprise Research Infrastructure Systems (ERIS)
Partners Healthcare, Harvard Medical School
http://www.partners.org
The information transmitted in this electronic communication is intended only
for the person or entity to whom it is addressed and may contain confidential
and/or privileged material. Any review, retransmission, dissemination or other
use of or taking of any action in reliance upon this information by persons or
entities other than the intended recipient is prohibited. If you received this
information in error, please contact the Compliance HelpLine at 800-856-1983 and
properly dispose of this information.
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: [EMAIL PROTECTED]
web : http://www.scalableinformatics.com
http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
