[
https://issues.apache.org/jira/browse/DRILL-5502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053476#comment-16053476
]
Paul Rogers commented on DRILL-5502:
------------------------------------
With columns above 256 characters, and the current version of the readers,
Drill will create vectors in excess of 16 MB. When it does, allocation comes
directly from the direct memory pool instead of the Netty free lists. Tests
have shown that direct memory allocation is 1000x slower than allocation from
the Netty free list. We could be hitting that problem here.
> Parallelized external sort is slower compared to the single fragment scenario
> on some data sets
> -----------------------------------------------------------------------------------------------
>
> Key: DRILL-5502
> URL: https://issues.apache.org/jira/browse/DRILL-5502
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
> Affects Versions: 1.10.0
> Reporter: Rahul Challapalli
> Assignee: Paul Rogers
> Attachments: multiple_fragments.log, multiple_fragments.sys.drill,
> single_fragment.log, single_fragment.sys.drill
>
>
> git.commit.id.abbrev=1e0a14c
> The below query runs in a single fragment and completes in ~13 minutes
> {code}
> ALTER SESSION SET `exec.sort.disable_managed` = false;
> alter session set `planner.width.max_per_node` = 1;
> alter session set `planner.memory.max_query_memory_per_node` = 62600000;
> alter session set `planner.width.max_per_query` = 17;
> select count(*) from (select * from
> dfs.`/drill/testdata/resource-manager/5kwidecolumns_500k.tbl` order by
> columns[0]) d where d.columns[0] = '4041054511';
> +---------+
> | EXPR$0 |
> +---------+
> | 0 |
> +---------+
> 1 row selected (832.705 seconds)
> {code}
> Now I increased the parallelization to 10 and also increased the memory
> allocated to the sort by 10 times, so that each individual fragments still
> ends up getting the similar amount of memory. In this case however the query
> takes ~30 minutes to complete which is strange
> {code}
> ALTER SESSION SET `exec.sort.disable_managed` = false;
> alter session set `planner.width.max_per_node` = 10;
> alter session set `planner.memory.max_query_memory_per_node` = 626000000;
> alter session set `planner.width.max_per_query` = 17;
> select count(*) from (select * from
> dfs.`/drill/testdata/resource-manager/5kwidecolumns_500k.tbl` order by
> columns[0]) d where d.columns[0] = '4041054511';
> +---------+
> | EXPR$0 |
> +---------+
> | 0 |
> +---------+
> 1 row selected (1845.508 seconds)
> {code}
> My data set contains wide columns (5k chars wide). I will try to reproduce
> this with a data set where the column width is < 256 bytes.
> Attached the data profile and log file from both the scenarios. The data set
> is too large to attach to a jira
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
