[
https://issues.apache.org/jira/browse/TEZ-3701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated TEZ-3701:
----------------------------------
Attachment: TEZ-3701.6.patch
Attaching the revised patch with review comments addressed.
Tried the patch with Hive/LLAP. In LLAP, it would be desirable to allocate
memory lazily to avoid mem pressures.
Observed 8% improvement in the following query in "Map 1", which used
"UnorderedPartitionedKVWriter" and had lots of spills at 10 TB TPC-DS dataset.
10 nodes were used for this benchmark with tez-autobuild. I had
"tez.runtime.unordered.output.max-per-buffer.size-bytes=10000000" and
"tez.runtime.unordered.output.buffer.size-mb=512". Ideally we would like to set
"unordered.output.max-per-buffer" to around 30MB in this config so that it is
closer to the one region in G1GC. But 10 MB was used for this benchmark given
lesser num of nodes.
{noformat}
create table testdata as select distinct customer.c_customer_sk,
ws_sold_time_sk,
d_date as d3 from
tpcds_bin_partitioned_orc_10000.web_sales JOIN
tpcds_bin_partitioned_orc_10000.date_dim
ON web_sales.ws_sold_date_sk = date_dim.d_date_sk
JOIN tpcds_bin_partitioned_orc_10000.customer ON web_sales.ws_bill_customer_sk
= customer.c_customer_sk
ORDER BY customer.c_customer_sk limit 10;
Without Patch:
Task Execution Summary
----------------------------------------------------------------------------------------------
VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
OUTPUT_RECORDS
----------------------------------------------------------------------------------------------
Map 1 261082.00 0 0 7,198,162,681
7,197,265,278
Map 5 8096.00 0 0 65,000,000
65,000,000
Map 6 1015.00 0 0 73,049
73,049
Reducer 2 295271.00 0 0 7,262,338,327
10,090
Reducer 3 18985.00 0 0 10,090
2,528
Reducer 4 851.00 0 0 2,528
0
----------------------------------------------------------------------------------------------
With Patch:
Task Execution Summary
----------------------------------------------------------------------------------------------
VERTICES DURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS
OUTPUT_RECORDS
----------------------------------------------------------------------------------------------
Map 1 240025.00 0 0 7,198,162,681
7,197,265,278
Map 5 12145.00 0 0 65,000,000
65,000,000
Map 6 1008.00 0 0 73,049
73,049
Reducer 2 267080.00 0 0 7,262,338,327
10,090
Reducer 3 16148.00 0 0 10,090
2,528
Reducer 4 334.00 0 0 2,528
0
----------------------------------------------------------------------------------------------
{noformat}
> UnorderedPartitionedKVWriter - issues with parallel Deflater usage,
> synchronousqueue in threadpool
> --------------------------------------------------------------------------------------------------
>
> Key: TEZ-3701
> URL: https://issues.apache.org/jira/browse/TEZ-3701
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Harish Jaiprakash
> Assignee: Rajesh Balamohan
> Priority: Blocker
> Attachments: TEZ-3701.2.patch, TEZ-3701.3.patch, TEZ-3701.4.patch,
> TEZ-3701.5.patch, TEZ-3701.6.patch
>
>
> UnorderedPartitionedKVWriter add task to the executor, but does not wait for
> them to finish before starting the final merge. This can cause finalMerge to
> fail or write incorrect data.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
