[
https://issues.apache.org/jira/browse/TEZ-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606376#comment-14606376
]
Saikat edited comment on TEZ-2574 at 6/29/15 8:51 PM:
------------------------------------------------------
Marking this JIRA as "depends on" TEZ-2575, since then we can handle scenario
where KV doesnt fit into the entire buffer.
for example:
1. initially allocate a span with kvmeta = 16 and rest kvbuffer.(i.e just 1
item worth meta)
Now this can result in three case:
case 1: the KV pair that is written, fits into kvbuffer, so we can update out
estimate of optimal kvmeta and kvbuffer split.
case2: the KV pair written did not fit into the kvbuffer(which means the kv is
so large it doesnot even fit into a 16bytes metasize split), in that case we
should just spill the single record to disk.
Another case:
case 3: we have updated our estimate of optimal split, and written a number of
KV pairs in a span. Now a KV pair arrives which didnot fit into the remaining
space of the span. Thus the current span would be spilled to disk, a new span
will be allocated with meta size of 16bytes, and we try to write the KV pair in
this new span.
case 3.1 If this fails, that means KV pair is large to fit into one block,
so spill the single record to disk.
case 3.2 if kv fits into the new span kvbuffer, we update the optimal size
split again.
was (Author: saikatr):
Marking this JIRA as "depends on" TEZ-2575, since then we can handle scenario
where KV doesnt fit into the entire buffer.
for example:
1. initially allocate a span with kvmeta = 16 and rest kvbuffer.(i.e just 1
item worth meta)
Now this can result in two case:
case 1: the KV pair that is written, fits into kvbuffer, so we can update out
estimate of optimal kvmeta and kvbuffer split.
case2: the KV pair written did not fit into the kvbuffer(which means the kv is
so large it doesnot even fit into a 16bytes metasize split), in that case we
should just spill the single record to disk.
case 3: we have updated our estimate of optimal split, and written a number of
KV pairs in a span. Now a KV pair arrives which didnot fit into the remaining
space of the span. Thus the current span would be spilled to disk, a new span
will be allocated with meta size of 16bytes, and we try to write the KV pair in
this new span.
case 3.1 If this fails, that means KV pair is large to fit into one block,
so spill the single record to disk.
case 3.2 if kv fits into the new span kvbuffer, we update the optimal size
split again.
> Make a better Metadata Value split choice in Pipeline sort
> ----------------------------------------------------------
>
> Key: TEZ-2574
> URL: https://issues.apache.org/jira/browse/TEZ-2574
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Saikat
> Assignee: Saikat
> Attachments: TEZ-2574.patch
>
>
> In the current implementation of pipeline sort, when a new sort span object
> is created with a hard coded value of 1M items and 16 bytes per item.
> According to the present code logic,
> int metasize = METASIZE*maxItems;
> int dataSize = maxItems * perItem;
> if(capacity < (metasize+dataSize)) {
> // try to allocate less meta space, because we have sample data
> metasize = METASIZE*(capacity/(perItem+METASIZE));
> }
> if capacity is less than 32mb, the buffer will be halved into meta and value
> buffers, which is not efficient.
> We need a more generic split, based on the KV pair size written to the buffer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
