[
https://issues.apache.org/jira/browse/DRILL-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053483#comment-16053483
]
Paul Rogers commented on DRILL-5493:
------------------------------------
Basics:
{code}
Config: memory limit = 63,161,283, spill file size = 268435456, spill batch
size = 8388608, merge limit = 2147483647, merge batch size = 16777216
...
Actual batch schema & sizes {
col1(type: VARCHAR, std col. size: 54, actual col. size: 7404, total size:
268730368, data size: 242606878, row capacity: 32768, density: -5)
Records: 32768, Total size: 268,795,904, Gross row width:8203, Net row
width:7404, Density:0}
{code}
The two critical numbers are:
{code}
memory limit = 63,161,283
Total size: 268,795,904
{code}
The sort is running in frag:0:0, which is the screen fragment. A downstream
batch has created a batch of size 270 MB. But, the sort has been given only 63
MB. Clearly, the sort cannot make progress.
So, this is an expected error given the casual way that Drill currently creates
batches of sizes that are completely independent of memory limits applied to
operators. This is a bug in the Drill execution engine in general, not a bug in
the external sort in particular.
> Managed External Sort + CTAS partition by results in "Unable to allocate sv2
> vector" error
> ------------------------------------------------------------------------------------------
>
> Key: DRILL-5493
> URL: https://issues.apache.org/jira/browse/DRILL-5493
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
> Affects Versions: 1.10.0
> Reporter: Rahul Challapalli
> Assignee: Paul Rogers
> Attachments: 26ee07bb-81ff-1c10-9003-90510f4b8e1d.sys.drill,
> drillbit.log
>
>
> Config :
> {code}
> git.commit.id.abbrev=1e0a14c
> No of nodes : 1
> DRILL_MAX_DIRECT_MEMORY="32G"
> DRILL_MAX_HEAP="4G"
> Assertions Enabled : true
> {code}
> The below query fails during the CTAS phase (the explicit order by in the
> query runs fine)
> {code}
> ALTER SESSION SET `exec.sort.disable_managed` = false;
> alter session set `planner.width.max_per_query` = 17;
> create table dfs.drillTestDir.xsort_ctas4 partition by (col1) as select
> columns[0] as col1 from (select * from
> dfs.`/drill/testdata/resource-manager/wide-to-zero` order by columns[0]);
> Error: RESOURCE ERROR: Unable to allocate sv2 buffer
> Fragment 0:0
> [Error Id: 24ae2ec8-ac2a-45c3-b550-43c12764165d on qa-node190.qa.lab:31010]
> (state=,code=0)
> {code}
> I attached the logs and profiles. The data is too large to attach to a jira.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
