[
https://issues.apache.org/jira/browse/HADOOP-8800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yankay updated HADOOP-8800:
---------------------------
Description:
h2. Introduce
This is a patch to provide a feature that a big file can be transferred in
different compress algorithm and level dynamically. So the performance would be
better.
h2. Why
Compression is important in transfer big file. I found that we should need
different compress algorithm in different cases. I have tested some cases.
| |compression ratio| compression throughput(MB/s) |Throughput in 100MB/s
network |Throughput in 10MB/s network|
|ZLIB |35.80% |9.6 |9.6 |9.6|
|LZO |54.40% |101.7 |101.7 |18.38235294|
|LIBLZF |54.60% |134.3 |134.3 |18.3150183|
|QUICKLZ |54.90% |183.4 |182.1493625 |18.21493625|
|FASTLZ |56.20% |134.4 |134.4 |17.79359431|
|SNAPPY |59.80% |189 |167.2240803 |16.72240803|
|NONE |100% |300 |100 |10|
So there are no “perfect compress algorithm” can suit all cases. I want to
build a dynamic “CompressOutputStream” to change “compress algorithm” in
runtime.
In a busy cluster, The CPU and the network is changing all the time. There are
some cases:
* The cluster is very huge; the network between each node is not the same. Some
are 10MB/s, some are 100MB/s. We cannot choose a perfect compress algorithm in
a MapReduce Job.
* The CPU can be use up when there is a “big job”, but the network is free. The
job should not compress any more.
* Some node in a cluster have high load, but the others not. We should use
different compress algorithm for them.
h2. What
In my idea, I can transfer a file with blocks. First block may use a compress
algorithm such as LZO. After transfer it, we can get some information, and
decide what the compress algorithm in the next block. Like TCP, it starts up
slowly, and try run faster and faster. It can make the io faster by choose a
more suitable compress algorithm.
h2. Design
In a big file transfer, compress and network all takes time. Consider to
transfer a fixed size file:
{code}
T = c/p + r/s
{code}
Define:
* T: the total time used
* C:the CPU cycle to compress the file
* P:the available CPU GHz for compressing
* R:compress radio
* S:the available speed(throughput) of network (including decompress)
The variable are not fixed, the all can be change mainly by some reason.
* C:decide by different content , algorithm and algorithm level
* P: decide by CPU , processes at the machine
* R: decide by different content , algorithm and algorithm level
* S: decide by network, the pairs’ network and processes at the machine.
The file is transferred by block. After a block transferred, there is some
information we can get:
* C/P : the time takes for compress
* R : the compress radio for the block
* R/S : the time takes for network
With the information and some reasonable assumptions, we can forecast each
compress algorithm’s performance. The reasonable assumptions are:
* In one transfer, the content is similar
* P, S is continuous, we can assume that the next P, S is the same as the
current one.
* C, R is proportional totally. For example, LZO is always faster than ZIP.
With the information and reasonable assumptions, we can forecast the next one
by:
* C2/P2 = (last C1/P1 ) * (avg C2/C1)
* R2/S2 = F(R1) / S1 (S1 is known)
* F(R1)=R1*(avg R2/R1)
Then we can know the time each compress algorithm needs. And choose the best
one to compress next block.
To optimize the avg values, we must log some statistics information. To
statistic the avg, we can set an N = 3, 5…, and change the avg value after each
information comes.
{code}
Avg V = (n-1) / n * Avg V+ V / n
{code}
h2. Next Work
I would try to submit a patch later. And is there anyone interested about that?
was:
h2. Introduce
This is a patch to provide a feature that a big file can be transferred in
different compress algorithm and level dynamically. So the performance would be
better.
h2. Why
Compression is important in transfer big file. I found that we should need
different compress algorithm in different cases. I have tested some cases.
||compression ratio| compression |throughput(MB/s) |Throughput in 100MB/s
network |Throughput in 10MB/s network|
|ZLIB |35.80% |9.6 |9.6 |9.6|
|LZO |54.40% |101.7 |101.7 |18.38235294|
|LIBLZF |54.60% |134.3 |134.3 |18.3150183|
|QUICKLZ |54.90% |183.4 |182.1493625 |18.21493625|
|FASTLZ |56.20% |134.4 |134.4 |17.79359431|
|SNAPPY |59.80% |189 |167.2240803 |16.72240803|
|NONE |100% |300 |100 |10|
So there are no “perfect compress algorithm” can suit all cases. I want to
build a dynamic “CompressOutputStream” to change “compress algorithm” in
runtime.
In a busy cluster, The CPU and the network is changing all the time. There are
some cases:
* The cluster is very huge; the network between each node is not the same. Some
are 10MB/s, some are 100MB/s. We cannot choose a perfect compress algorithm in
a MapReduce Job.
* The CPU can be use up when there is a “big job”, but the network is free. The
job should not compress any more.
* Some node in a cluster have high load, but the others not. We should use
different compress algorithm for them.
h2. What
In my idea, I can transfer a file with blocks. First block may use a compress
algorithm such as LZO. After transfer it, we can get some information, and
decide what the compress algorithm in the next block. Like TCP, it starts up
slowly, and try run faster and faster. It can make the io faster by choose a
more suitable compress algorithm.
h2. Design
In a big file transfer, compress and network all takes time. Consider to
transfer a fixed size file:
{code}
T = c/p + r/s
{code}
Define:
* T: the total time used
* C:the CPU cycle to compress the file
* P:the available CPU GHz for compressing
* R:compress radio
* S:the available speed(throughput) of network (including decompress)
The variable are not fixed, the all can be change mainly by some reason.
* C:decide by different content , algorithm and algorithm level
* P: decide by CPU , processes at the machine
* R: decide by different content , algorithm and algorithm level
* S: decide by network, the pairs’ network and processes at the machine.
The file is transferred by block. After a block transferred, there is some
information we can get:
* C/P : the time takes for compress
* R : the compress radio for the block
* R/S : the time takes for network
With the information and some reasonable assumptions, we can forecast each
compress algorithm’s performance. The reasonable assumptions are:
* In one transfer, the content is similar
* P, S is continuous, we can assume that the next P, S is the same as the
current one.
* C, R is proportional totally. For example, LZO is always faster than ZIP.
With the information and reasonable assumptions, we can forecast the next one
by:
* C2/P2 = (last C1/P1 ) * (avg C2/C1)
* R2/S2 = F(R1) / S1 (S1 is known)
* F(R1)=R1*(avg R2/R1)
Then we can know the time each compress algorithm needs. And choose the best
one to compress next block.
To optimize the avg values, we must log some statistics information. To
statistic the avg, we can set an N = 3, 5…, and change the avg value after each
information comes.
{code}
Avg V = (n-1) / n * Avg V+ V / n
{code}
h2. Next Work
I would try to submit a patch later. And is there anyone interested about that?
> Dynamic Compress Stream
> -----------------------
>
> Key: HADOOP-8800
> URL: https://issues.apache.org/jira/browse/HADOOP-8800
> Project: Hadoop Common
> Issue Type: New Feature
> Components: io
> Affects Versions: 2.0.1-alpha
> Reporter: yankay
> Labels: patch
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> h2. Introduce
> This is a patch to provide a feature that a big file can be transferred in
> different compress algorithm and level dynamically. So the performance would
> be better.
> h2. Why
> Compression is important in transfer big file. I found that we should need
> different compress algorithm in different cases. I have tested some cases.
>
> | |compression ratio| compression throughput(MB/s) |Throughput in 100MB/s
> network |Throughput in 10MB/s network|
> |ZLIB |35.80% |9.6 |9.6 |9.6|
> |LZO |54.40% |101.7 |101.7 |18.38235294|
> |LIBLZF |54.60% |134.3 |134.3 |18.3150183|
> |QUICKLZ |54.90% |183.4 |182.1493625 |18.21493625|
> |FASTLZ |56.20% |134.4 |134.4 |17.79359431|
> |SNAPPY |59.80% |189 |167.2240803 |16.72240803|
> |NONE |100% |300 |100 |10|
> So there are no “perfect compress algorithm” can suit all cases. I want to
> build a dynamic “CompressOutputStream” to change “compress algorithm” in
> runtime.
> In a busy cluster, The CPU and the network is changing all the time. There
> are some cases:
> * The cluster is very huge; the network between each node is not the same.
> Some are 10MB/s, some are 100MB/s. We cannot choose a perfect compress
> algorithm in a MapReduce Job.
> * The CPU can be use up when there is a “big job”, but the network is free.
> The job should not compress any more.
> * Some node in a cluster have high load, but the others not. We should use
> different compress algorithm for them.
> h2. What
> In my idea, I can transfer a file with blocks. First block may use a compress
> algorithm such as LZO. After transfer it, we can get some information, and
> decide what the compress algorithm in the next block. Like TCP, it starts up
> slowly, and try run faster and faster. It can make the io faster by choose a
> more suitable compress algorithm.
> h2. Design
> In a big file transfer, compress and network all takes time. Consider to
> transfer a fixed size file:
> {code}
> T = c/p + r/s
> {code}
> Define:
> * T: the total time used
> * C:the CPU cycle to compress the file
> * P:the available CPU GHz for compressing
> * R:compress radio
> * S:the available speed(throughput) of network (including decompress)
> The variable are not fixed, the all can be change mainly by some reason.
> * C:decide by different content , algorithm and algorithm level
> * P: decide by CPU , processes at the machine
> * R: decide by different content , algorithm and algorithm level
> * S: decide by network, the pairs’ network and processes at the machine.
> The file is transferred by block. After a block transferred, there is some
> information we can get:
> * C/P : the time takes for compress
> * R : the compress radio for the block
> * R/S : the time takes for network
> With the information and some reasonable assumptions, we can forecast each
> compress algorithm’s performance. The reasonable assumptions are:
> * In one transfer, the content is similar
> * P, S is continuous, we can assume that the next P, S is the same as the
> current one.
> * C, R is proportional totally. For example, LZO is always faster than ZIP.
> With the information and reasonable assumptions, we can forecast the next one
> by:
> * C2/P2 = (last C1/P1 ) * (avg C2/C1)
> * R2/S2 = F(R1) / S1 (S1 is known)
> * F(R1)=R1*(avg R2/R1)
> Then we can know the time each compress algorithm needs. And choose the best
> one to compress next block.
> To optimize the avg values, we must log some statistics information. To
> statistic the avg, we can set an N = 3, 5…, and change the avg value after
> each information comes.
> {code}
> Avg V = (n-1) / n * Avg V+ V / n
> {code}
> h2. Next Work
> I would try to submit a patch later. And is there anyone interested about
> that?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
