[
https://issues.apache.org/jira/browse/HADOOP-8800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yankay updated HADOOP-8800:
---------------------------
Description:
We use compress in MapReduce in some case because It use CPU to improve IO
throughput.
But we can only set one compress algorithm in configure file. The hadoop
cluster is changing every time. So a compress algorithm may not work well in
all case.
Why not provide an algorithm named dynamic. It can change compress level and
algorithm dynamicly based on performance. Like tcp, it starts up slowly, and
try run faster and faster. It can make the io faster by choose a more suitable
compress algorithm.
I would write a detail design here, and try to submit a patch.
was:
We use compress in MapReduce in some case because It use CPU to improve IO
throughput.
But we can only set one compress algorithm in configure file. The hadoop
cluster is changing every time. So a compress algorithm may not work well in
all case.
Why not provide a algorithm named dynamic. It can change compress level and
algorithm dynamic based on performance. Like tcp, it starts up slowly, and try
run faster and faster.
I would write a detail design here, and try to submit a patch.
> Dynamic Compress Stream
> -----------------------
>
> Key: HADOOP-8800
> URL: https://issues.apache.org/jira/browse/HADOOP-8800
> Project: Hadoop Common
> Issue Type: New Feature
> Components: io
> Affects Versions: 2.0.1-alpha
> Reporter: yankay
> Labels: patch
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> We use compress in MapReduce in some case because It use CPU to improve IO
> throughput.
> But we can only set one compress algorithm in configure file. The hadoop
> cluster is changing every time. So a compress algorithm may not work well in
> all case.
> Why not provide an algorithm named dynamic. It can change compress level and
> algorithm dynamicly based on performance. Like tcp, it starts up slowly, and
> try run faster and faster. It can make the io faster by choose a more
> suitable compress algorithm.
> I would write a detail design here, and try to submit a patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
