[
https://issues.apache.org/jira/browse/TEZ-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044893#comment-14044893
]
Gopal V commented on TEZ-1228:
------------------------------
[~hitesh]: TezMerger has no way of returning an iterator with a pre-knowledge
of sizes.
I could potentially do this if the entire data set was already present in
memory or back patch it in if all my writes were also going into memory - this
is what ORC does (hold upto Stripe size in memory, with multiple compression
streams).
In re: to the in-memory details, I know that all of Hive is fairly bounded on
its tuple sizes for each key (this is what every SQL engine does, pretty much),
but decided to not go down that path to support pig's large bag entries.
A single compressed stream destroys my ability to back-patch the total value
count in, even in-memory - but that is the most efficient way of reading
hundreds of IFiles at the same time for the merge. This is because compression
buffers are direct byte buffers & are not always GC'd immediately (see
TEZ-1162).
All those design decisions aside, in the prototype Rajesh built, V_END_MARKER
is a vInt, which is exactly as big.
> Prototype IFile : Define a memory & merge optimized vertex-intermediate file
> format for Tez
> -------------------------------------------------------------------------------------------
>
> Key: TEZ-1228
> URL: https://issues.apache.org/jira/browse/TEZ-1228
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Labels: perfomance
> Attachments: TEZ-1228-IFile.pdf
>
>
> The current vertex-intermediate format used all across Tez is a flat file of
> variable length k,v pairs. For a significant number of use-cases, in
> particular the sorted output phase, a large number of consecutive identical
> keys are found within the same stream. The IFile format ends up writing each
> key out fully into the stream to generate (K,V) pairs instead of ordering it
> into a more efficient K, {V1, .. Vn} list.
> This duplication of key data needs larger buffers to hold in memory and
> requires comparison between keys known to be identical while doing a merge
> sort.
> This bug tracks the building of a prototype IFile format which is optimized
> for lower uncompressed sizes within memory buffers and less compute intensive
> to perform merge sorts during the reducer phase.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
