[jira] [Updated] (DRILL-6235) Flatten query leads to out of memory in RPC layer.

2018-04-26 Thread Kunal Khatua (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua updated DRILL-6235:

Affects Version/s: 1.13.0

> Flatten query leads to out of memory in RPC layer.
> --
>
> Key: DRILL-6235
> URL: https://issues.apache.org/jira/browse/DRILL-6235
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0, 1.13.0
>Reporter: Khurram Faraaz
>Assignee: Padma Penumarthy
>Priority: Critical
> Attachments: 25593391-512d-23ab-7c84-3651006931e2.sys.drill, 
> drillbit_snippet.log
>
>
> Flatten query leads to out of memory in RPC layer. Query profile is attached 
> here.
> Total number of JSON files = 4095
>  Each JSON file has nine rows
>  And each row in the JSON has an array with 1024 integer values, and there 
> are other string values outside of the array.
>  Two major fragments and eighty eight minor fragments were created
> On a 4 node CentOS cluster
>  number of CPU cores
>  [root@qa102-45 ~]# grep -c ^processor /proc/cpuinfo
>  32
> Details of memory
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select * from sys.memory;
> +--++---+-+-+-+-+
> | hostname | user_port | heap_current | heap_max | direct_current | 
> jvm_direct_current | direct_max |
> +--++---+-+-+-+-+
> | qa102-45.qa.lab | 31010 | 1130364912 | 4294967296 | 0 | 170528 | 8589934592 
> |
> | qa102-47.qa.lab | 31010 | 171823104 | 4294967296 | 0 | 21912 | 8589934592 |
> | qa102-48.qa.lab | 31010 | 201326576 | 4294967296 | 0 | 21912 | 8589934592 |
> | qa102-46.qa.lab | 31010 | 214780896 | 4294967296 | 0 | 21912 | 8589934592 |
> +--++---+-+-+-+-+
> 4 rows selected (0.166 seconds)
> {noformat}
> Steps to repro the failure.
> Reset all options and set slice_target=1 and run the below SQL query.
>  alter system reset all;
>  alter system set `planner.slice_target`=1;
> {noformat}
> SELECT * , FLATTEN(arr) FROM many_json_files
> ...
> Error: RESOURCE ERROR: One or more nodes ran out of memory while executing 
> the query.
> Failure allocating buffer.
> Fragment 1:38
> [Error Id: cf4fd273-d8a2-45e8-8d72-15c738e53b0f on qa102-45.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> Stack trace from drillbit.log fir above failing query.
> {noformat}
> 2018-03-12 11:52:33,849 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO 
> o.a.d.e.w.fragment.FragmentExecutor - 
> 25593391-512d-23ab-7c84-3651006931e2:0:0: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2018-03-12 11:52:33,849 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO 
> o.a.d.e.w.f.FragmentStatusReporter - 
> 25593391-512d-23ab-7c84-3651006931e2:0:0: State to report: RUNNING
> 2018-03-12 11:52:33,854 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO 
> o.a.d.e.c.ClassCompilerSelector - Java compiler policy: DEFAULT, Debug 
> option: true
> 2018-03-12 11:52:35,929 [BitServer-4] WARN 
> o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming 
> stream due to memory limits. Current Allocation: 92340224.
> 2018-03-12 11:52:35,929 [BitServer-3] WARN 
> o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming 
> stream due to memory limits. Current Allocation: 92340224.
> 2018-03-12 11:52:35,930 [BitServer-3] ERROR 
> o.a.drill.exec.rpc.data.DataServer - Out of memory in RPC layer.
> 2018-03-12 11:52:35,930 [BitServer-4] ERROR 
> o.a.drill.exec.rpc.data.DataServer - Out of memory in RPC layer.
> 2018-03-12 11:52:35,930 [BitServer-4] WARN 
> o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming 
> stream due to memory limits. Current Allocation: 83886080.
> 2018-03-12 11:52:35,930 [BitServer-3] WARN 
> o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming 
> stream due to memory limits. Current Allocation: 83886080.
> 2018-03-12 11:52:35,930 [BitServer-4] ERROR 
> o.a.drill.exec.rpc.data.DataServer - Out of memory in RPC layer.
> 2018-03-12 11:52:35,930 [BitServer-3] ERROR 
> o.a.drill.exec.rpc.data.DataServer - Out of memory in RPC layer.
> 2018-03-12 11:52:35,931 [BitServer-3] WARN 
> o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming 
> stream due to memory limits. Current Allocation: 83886080.
> 2018-03-12 11:52:35,931 [BitServer-4] WARN 
> o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming 
> stream due to memory limits. Current Allocation: 83886080.
> 2018-03-12 11:52:35,931 [BitServer-3] ERROR 
> o.a.drill.exec.rpc.data.DataServer - Out of memory

[jira] [Updated] (DRILL-6235) Flatten query leads to out of memory in RPC layer.

2018-03-21 Thread Khurram Faraaz (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Khurram Faraaz updated DRILL-6235:
--
Description: 
Flatten query leads to out of memory in RPC layer. Query profile is attached 
here.

Total number of JSON files = 4095
 Each JSON file has nine rows
 And each row in the JSON has an array with 1024 integer values, and there are 
other string values outside of the array.
 Two major fragments and eighty eight minor fragments were created

On a 4 node CentOS cluster
 number of CPU cores
 [root@qa102-45 ~]# grep -c ^processor /proc/cpuinfo
 32

Details of memory
{noformat}
0: jdbc:drill:schema=dfs.tmp> select * from sys.memory;
+--++---+-+-+-+-+
| hostname | user_port | heap_current | heap_max | direct_current | 
jvm_direct_current | direct_max |
+--++---+-+-+-+-+
| qa102-45.qa.lab | 31010 | 1130364912 | 4294967296 | 0 | 170528 | 8589934592 |
| qa102-47.qa.lab | 31010 | 171823104 | 4294967296 | 0 | 21912 | 8589934592 |
| qa102-48.qa.lab | 31010 | 201326576 | 4294967296 | 0 | 21912 | 8589934592 |
| qa102-46.qa.lab | 31010 | 214780896 | 4294967296 | 0 | 21912 | 8589934592 |
+--++---+-+-+-+-+
4 rows selected (0.166 seconds)
{noformat}
Steps to repro the failure.

Reset all options and set slice_target=1 and run the below SQL query.


 alter system reset all;
 alter system set `planner.slice_target`=1;
{noformat}
SELECT * , FLATTEN(arr) FROM many_json_files
...

Error: RESOURCE ERROR: One or more nodes ran out of memory while executing the 
query.

Failure allocating buffer.
Fragment 1:38

[Error Id: cf4fd273-d8a2-45e8-8d72-15c738e53b0f on qa102-45.qa.lab:31010] 
(state=,code=0)
{noformat}
Stack trace from drillbit.log fir above failing query.
{noformat}
2018-03-12 11:52:33,849 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO 
o.a.d.e.w.fragment.FragmentExecutor - 25593391-512d-23ab-7c84-3651006931e2:0:0: 
State change requested AWAITING_ALLOCATION --> RUNNING
2018-03-12 11:52:33,849 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO 
o.a.d.e.w.f.FragmentStatusReporter - 25593391-512d-23ab-7c84-3651006931e2:0:0: 
State to report: RUNNING
2018-03-12 11:52:33,854 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO 
o.a.d.e.c.ClassCompilerSelector - Java compiler policy: DEFAULT, Debug option: 
true
2018-03-12 11:52:35,929 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder 
- Failure allocating buffer on incoming stream due to memory limits. Current 
Allocation: 92340224.
2018-03-12 11:52:35,929 [BitServer-3] WARN o.a.d.exec.rpc.ProtobufLengthDecoder 
- Failure allocating buffer on incoming stream due to memory limits. Current 
Allocation: 92340224.
2018-03-12 11:52:35,930 [BitServer-3] ERROR o.a.drill.exec.rpc.data.DataServer 
- Out of memory in RPC layer.
2018-03-12 11:52:35,930 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer 
- Out of memory in RPC layer.
2018-03-12 11:52:35,930 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder 
- Failure allocating buffer on incoming stream due to memory limits. Current 
Allocation: 83886080.
2018-03-12 11:52:35,930 [BitServer-3] WARN o.a.d.exec.rpc.ProtobufLengthDecoder 
- Failure allocating buffer on incoming stream due to memory limits. Current 
Allocation: 83886080.
2018-03-12 11:52:35,930 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer 
- Out of memory in RPC layer.
2018-03-12 11:52:35,930 [BitServer-3] ERROR o.a.drill.exec.rpc.data.DataServer 
- Out of memory in RPC layer.
2018-03-12 11:52:35,931 [BitServer-3] WARN o.a.d.exec.rpc.ProtobufLengthDecoder 
- Failure allocating buffer on incoming stream due to memory limits. Current 
Allocation: 83886080.
2018-03-12 11:52:35,931 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder 
- Failure allocating buffer on incoming stream due to memory limits. Current 
Allocation: 83886080.
2018-03-12 11:52:35,931 [BitServer-3] ERROR o.a.drill.exec.rpc.data.DataServer 
- Out of memory in RPC layer.
2018-03-12 11:52:35,931 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer 
- Out of memory in RPC layer.
...
...
2018-03-12 11:52:35,939 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder 
- Failure allocating buffer on incoming stream due to memory limits. Current 
Allocation: 67174400.
2018-03-12 11:52:35,939 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer 
- Out of memory in RPC layer.
2018-03-12 11:52:35,939 [BitServer-2] WARN o.a.d.exec.rpc.ProtobufLengthDecoder 
- Failure allocating buffer on incoming stream due to memory limits. Current 
Allocation: 84017152.
2018-03-12 11:52:35,939 [BitServer-2] ERROR o.a.drill.exec.rpc.data.DataServer 
- Out of memory in RPC layer.
2018-03-12

[jira] [Updated] (DRILL-6235) Flatten query leads to out of memory in RPC layer.

2018-03-13 Thread Khurram Faraaz (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Khurram Faraaz updated DRILL-6235:
--
Attachment: drillbit_snippet.log

> Flatten query leads to out of memory in RPC layer.
> --
>
> Key: DRILL-6235
> URL: https://issues.apache.org/jira/browse/DRILL-6235
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
>Reporter: Khurram Faraaz
>Assignee: Padma Penumarthy
>Priority: Critical
> Attachments: 25593391-512d-23ab-7c84-3651006931e2.sys.drill, 
> drillbit_snippet.log
>
>
> Flatten query leads to out of memory in RPC layer. Query profile is attached 
> here.
> Total number of JSON files = 4095
> Each JSON file has nine rows
> And each row in the JSON has an array with 1024 integer values, and there are 
> other string values outside of the array.
> Two major fragments and eighty eight minor fragments were created
> On a 4 node CentOS cluster
> number of CPU cores
> [root@qa102-45 ~]# grep -c ^processor /proc/cpuinfo
> 32
> Details of memory
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select * from sys.memory;
> +--++---+-+-+-+-+
> | hostname | user_port | heap_current | heap_max | direct_current | 
> jvm_direct_current | direct_max |
> +--++---+-+-+-+-+
> | qa102-45.qa.lab | 31010 | 1130364912 | 4294967296 | 0 | 170528 | 8589934592 
> |
> | qa102-47.qa.lab | 31010 | 171823104 | 4294967296 | 0 | 21912 | 8589934592 |
> | qa102-48.qa.lab | 31010 | 201326576 | 4294967296 | 0 | 21912 | 8589934592 |
> | qa102-46.qa.lab | 31010 | 214780896 | 4294967296 | 0 | 21912 | 8589934592 |
> +--++---+-+-+-+-+
> 4 rows selected (0.166 seconds)
> {noformat}
> Reset all options and set slice_target=1
> alter system reset all;
> alter system set `planner.slice_target`=1;
> {noformat}
> SELECT * , FLATTEN(arr) FROM many_json_files
> ...
> Error: RESOURCE ERROR: One or more nodes ran out of memory while executing 
> the query.
> Failure allocating buffer.
> Fragment 1:38
> [Error Id: cf4fd273-d8a2-45e8-8d72-15c738e53b0f on qa102-45.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> Stack trace from drillbit.log fir above failing query.
> {noformat}
> 2018-03-12 11:52:33,849 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO 
> o.a.d.e.w.fragment.FragmentExecutor - 
> 25593391-512d-23ab-7c84-3651006931e2:0:0: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2018-03-12 11:52:33,849 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO 
> o.a.d.e.w.f.FragmentStatusReporter - 
> 25593391-512d-23ab-7c84-3651006931e2:0:0: State to report: RUNNING
> 2018-03-12 11:52:33,854 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO 
> o.a.d.e.c.ClassCompilerSelector - Java compiler policy: DEFAULT, Debug 
> option: true
> 2018-03-12 11:52:35,929 [BitServer-4] WARN 
> o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming 
> stream due to memory limits. Current Allocation: 92340224.
> 2018-03-12 11:52:35,929 [BitServer-3] WARN 
> o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming 
> stream due to memory limits. Current Allocation: 92340224.
> 2018-03-12 11:52:35,930 [BitServer-3] ERROR 
> o.a.drill.exec.rpc.data.DataServer - Out of memory in RPC layer.
> 2018-03-12 11:52:35,930 [BitServer-4] ERROR 
> o.a.drill.exec.rpc.data.DataServer - Out of memory in RPC layer.
> 2018-03-12 11:52:35,930 [BitServer-4] WARN 
> o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming 
> stream due to memory limits. Current Allocation: 83886080.
> 2018-03-12 11:52:35,930 [BitServer-3] WARN 
> o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming 
> stream due to memory limits. Current Allocation: 83886080.
> 2018-03-12 11:52:35,930 [BitServer-4] ERROR 
> o.a.drill.exec.rpc.data.DataServer - Out of memory in RPC layer.
> 2018-03-12 11:52:35,930 [BitServer-3] ERROR 
> o.a.drill.exec.rpc.data.DataServer - Out of memory in RPC layer.
> 2018-03-12 11:52:35,931 [BitServer-3] WARN 
> o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming 
> stream due to memory limits. Current Allocation: 83886080.
> 2018-03-12 11:52:35,931 [BitServer-4] WARN 
> o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming 
> stream due to memory limits. Current Allocation: 83886080.
> 2018-03-12 11:52:35,931 [BitServer-3] ERROR 
> o.a.drill.exec.rpc.data.DataServer - Out of memory in RPC layer.
> 2018-03-12 11:52:35,931 [BitServer-4] ERROR 
>