[
https://issues.apache.org/jira/browse/DRILL-3555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731007#comment-14731007
]
Deneche A. Hakim commented on DRILL-3555:
-----------------------------------------
I couldn't find what's the limit doing, so I asked on the dev list and here is
Steven's reply:
{quote}
I think it probably isn't needed anymore. O believe it is a holdover from
before spilling was implemented. It doesn't seem to serve any purpose now.
{quote}
I looked at how the limit is being used, and after discussing this with other
developers I came to the conclusion it's not really doing anything except
putting a hard coded limit on some operators (external sort being one of them).
Assuming ExternalSort doesn't need this limit anymore, and looking at how
SortRecordBatchBuilder is being used in other operators
(OrderedPartitionRecordBatch, SortBatch and TopNBatch), this limit either has
no effect or we've been really lucky so far (see DRILL-3740).
I can revert my change to the original patch: basically adding another
constructor that uses Integer.MAX_VALUE for this limit and only use it for
ExternalSort. But someone should give some more information about what's that
limit for so I can add it as a comment to the code.
> Changing defaults for planner.memory.max_query_memory_per_node causes queries
> with window function to fail
> ----------------------------------------------------------------------------------------------------------
>
> Key: DRILL-3555
> URL: https://issues.apache.org/jira/browse/DRILL-3555
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
> Affects Versions: 1.1.0, 1.2.0
> Environment: 4 Nodes. Direct Memory= 48 GB each
> Reporter: Abhishek Girish
> Assignee: Deneche A. Hakim
> Priority: Critical
> Labels: window_function
> Fix For: 1.2.0
>
>
> Changing the default value for planner.memory.max_query_memory_per_node from
> 2 GB to anything higher causes queries with window functions to fail.
> Changed system options
> {code:sql}
> > select * from sys.options where status like '%CHANGE%';
> +-------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+
> | name | kind | type | status |
> num_val | string_val | bool_val | float_val |
> +-------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+
> | planner.enable_decimal_data_type | BOOLEAN | SYSTEM | CHANGED |
> null | null | true | null |
> | planner.memory.max_query_memory_per_node | LONG | SYSTEM | CHANGED |
> 8589934592 | null | null | null |
> +-------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+
> 2 rows selected (0.249 seconds)
> {code}
> Query
> {code:sql}
> > SELECT SUM(ss.ss_net_paid_inc_tax) OVER (PARTITION BY ss.ss_store_sk) FROM
> > store_sales ss LIMIT 20;
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> DrillRuntimeException: Adding this batch causes the total size to exceed max
> allowed size. Current runningBytes 1073638500, Incoming batchBytes 127875.
> maxBytes 1073741824
> Fragment 1:0
> [Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at
> sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:852)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.begin(SqlLine.java:612)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> {code}
> Log:
> {code}
> 2015-07-23 18:16:52,292 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:2:2] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:2:2: State change requested RUNNING -->
> FINISHED
> 2015-07-23 18:16:52,292 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:2:2] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:2:2: State to report: FINISHED
> 2015-07-23 18:17:05,485 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] ERROR
> o.a.d.e.p.i.s.SortRecordBatchBuilder - Adding this batch causes the total
> size to exceed max allowed size. Current runningBytes 1073638500, Incoming
> batchBytes 127875. maxBytes 1073741824
> 2015-07-23 18:17:05,486 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0: State change requested RUNNING -->
> FAILED
> ...
> 2015-07-23 18:17:05,990 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0: State change requested FAILED -->
> FINISHED
> 2015-07-23 18:17:05,999 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: DrillRuntimeException:
> Adding this batch causes the total size to exceed max allowed size. Current
> runningBytes 1073638500, Incoming batchBytes 127875. maxBytes 1073741824
> Fragment 1:0
> [Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> DrillRuntimeException: Adding this batch causes the total size to exceed max
> allowed size. Current runningBytes 1073638500, Incoming batchBytes 127875.
> maxBytes 1073741824
> Fragment 1:0
> [Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523)
> ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:178)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:292)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
> Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Adding
> this batch causes the total size to exceed max allowed size. Current
> runningBytes 1073638500, Incoming batchBytes 127875. maxBytes 1073741824
> at
> org.apache.drill.exec.physical.impl.sort.SortRecordBatchBuilder.add(SortRecordBatchBuilder.java:115)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:380)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:147)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:105)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> ...
> 2015-07-23 18:17:06,021 [BitServer-3] WARN o.a.d.e.w.b.ControlMessageHandler
> - Dropping request to cancel fragment.
> 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0 does not exist.
> 2015-07-23 18:17:06,022 [BitServer-3] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:4: State change requested RUNNING -->
> CANCELLATION_REQUESTED
> 2015-07-23 18:17:06,023 [BitServer-3] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:4: State to report:
> CANCELLATION_REQUESTED
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
