[
https://issues.apache.org/jira/browse/AVRO-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987973#comment-15987973
]
Hadoop QA commented on AVRO-1976:
---------------------------------
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 0m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} buildtest {color} | {color:green} 6m
52s{color} | {color:green} the patch passed {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 7m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.13.1 Server=1.13.1 Image:yetus/avro:793178a |
| JIRA Issue | AVRO-1976 |
| GITHUB PR | https://github.com/apache/avro/pull/182 |
| Optional Tests | buildtest javac |
| uname | Linux 05582b3c96e1 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 GNU/Linux |
| Build tool | build |
| git revision | master / 793178a |
| Default Java | 1.7.0_111 |
| modules | C: lang/java U: lang/java |
| Console output |
https://builds.apache.org/job/PreCommit-AVRO-Build-TEST/11/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> Add Input/OutputFormat to read/write encoded objects
> ----------------------------------------------------
>
> Key: AVRO-1976
> URL: https://issues.apache.org/jira/browse/AVRO-1976
> Project: Avro
> Issue Type: Improvement
> Components: java
> Environment: hadoop
> Reporter: Marius Posta
> Priority: Minor
> Labels: newbie
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> In certain cases, performance of some Avro map-reduce jobs can be
> considerably improved by de-coupling Avro encoding from actual Avro container
> file IO.
> In my case, a complex schema (100+ record fields) and large HDFS blocks
> resulted in Spark jobs where a lot of workers were idling while a couple of
> them were busy decoding their input splits.Furthermore, the objects then
> needed to be re-encoded in order to be shuffled about, which crippled
> performance further.
> I propose the addition of an InputFormat which reads a container file input
> split as key-value pairs in which the key is the file header and the value is
> the decompressed file data block. Also, an OutputFormat which follows the
> same logic for writing.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
