[jira] [Created] (DRILL-8455) mergejion memory leak when exception
shihuafeng created DRILL-8455:
-
Summary: mergejion memory leak when exception
Key: DRILL-8455
URL: https://issues.apache.org/jira/browse/DRILL-8455
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.20.0
Reporter: shihuafeng
when mergerjoin operator spilling data to disk ,it generate following
exception:
mergerjoin operator can not untimely invoke release() to free memory
+
{code:java}
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: External Sort
encountered an error while spilling to disk{code}
+
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (DRILL-8479) mergejion memory leak when exception
shihuafeng created DRILL-8479:
-
Summary: mergejion memory leak when exception
Key: DRILL-8479
URL: https://issues.apache.org/jira/browse/DRILL-8479
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Affects Versions: 1.21.1
Reporter: shihuafeng
Attachments: 0001-mergejoin-leak.patch
*Describe the bug*
megerjoin leak when RecordIterator allocate memory exception with
OutOfMemoryException{*}{*}
{*}Steps to reproduce the behavior{*}:
# prepare data for tpch 1s
# set direct memory 5g
# set planner.enable_hashjoin =false to ensure use mergejoin operator。
# set drill.memory.debug.allocator =true (Check for memory leaks )
# 20 concurrent for tpch sql8
# when it had OutOfMemoryException or null EXCEPTION , stopped all sql.
# finding memory leak
*Expected behavior*
when all sql sop , we should find direct memory is 0 AND could not find
leak log like following.
{code:java}
Allocator(op:2:0:11:MergeJoinPOP) 100/73728/4874240/100
(res/actual/peak/limit){code}
*Error detail, log output or screenshots*
{code:java}
Unable to allocate buffer of size XX (rounded from XX) due to memory limit ().
Current allocation: xx{code}
[^0001-mergejoin-leak.patch]
sql
{code:java}
// code placeholder
select o_year, sum(case when nation = 'CHINA' then volume else 0 end) /
sum(volume) as mkt_share from ( select extract(year from o_orderdate) as
o_year, l_extendedprice * 1.0 as volume, n2.n_name as nation from
hive.tpch1s.part, hive.tpch1s.supplier, hive.tpch1s.lineitem,
hive.tpch1s.orders, hive.tpch1s.customer, hive.tpch1s.nation n1,
hive.tpch1s.nation n2, hive.tpch1s.region where p_partkey = l_partkey and
s_suppkey = l_suppkey and l_orderkey = o_orderkey and o_custkey = c_custkey and
c_nationkey = n1.n_nationkey and n1.n_regionkey = r_regionkey and r_name =
'ASIA' and s_nationkey = n2.n_nationkey and o_orderdate between date
'1995-01-01' and date '1996-12-31' and p_type = 'LARGE BRUSHED BRASS') as
all_nations group by o_year order by o_year
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (DRILL-8478) mergejion memory leak when exception
shihuafeng created DRILL-8478:
-
Summary: mergejion memory leak when exception
Key: DRILL-8478
URL: https://issues.apache.org/jira/browse/DRILL-8478
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Affects Versions: 1.21.1
Reporter: shihuafeng
Fix For: 1.21.2
*Describe the bug*
hashpartition leak when allocate memory exception with OutOfMemoryException
*To Reproduce*
Steps to reproduce the behavior:
# prepare data for tpch 1s
# 20 concurrent for tpch sql8
# set direct memory 5g
# when it had OutOfMemoryException , stopped all sql.
# finding memory leak
*Expected behavior*
(1)i set \{DRILL_MAX_DIRECT_MEMORY:-"5G"}
(2) i run sql8 (sql detail as Additional context) with 20 concurrent
(3) it had OutOfMemoryException when create hashPartion
*Error detail, log output or screenshots*
Unable to allocate buffer of size 262144 (rounded from 262140) due to memory
limit (41943040). Current allocation: 20447232
sql
{code:java}
// code placeholder
{code}
select o_year, sum(case when nation = 'CHINA' then volume else 0 end) /
sum(volume) as mkt_share from ( select extract(year from o_orderdate) as
o_year, l_extendedprice * 1.0 as volume, n2.n_name as nation from
hive.tpch1s.part, hive.tpch1s.supplier, hive.tpch1s.lineitem,
hive.tpch1s.orders, hive.tpch1s.customer, hive.tpch1s.nation n1,
hive.tpch1s.nation n2, hive.tpch1s.region where p_partkey = l_partkey and
s_suppkey = l_suppkey and l_orderkey = o_orderkey and o_custkey = c_custkey and
c_nationkey = n1.n_nationkey and n1.n_regionkey = r_regionkey and r_name =
'ASIA' and s_nationkey = n2.n_nationkey and o_orderdate between date
'1995-01-01' and date '1996-12-31' and p_type = 'LARGE BRUSHED BRASS') as
all_nations group by o_year order by o_year
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (DRILL-8490) Sender operator fake memory leak result to sql failed or other exception
shihuafeng created DRILL-8490: - Summary: Sender operator fake memory leak result to sql failed or other exception Key: DRILL-8490 URL: https://issues.apache.org/jira/browse/DRILL-8490 Project: Apache Drill Issue Type: Bug Components: Server Affects Versions: 1.21.1 Reporter: shihuafeng Fix For: 1.22.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DRILL-8488) HashJoinPOP memory leak is caused by an oom exception
shihuafeng created DRILL-8488: - Summary: HashJoinPOP memory leak is caused by an oom exception Key: DRILL-8488 URL: https://issues.apache.org/jira/browse/DRILL-8488 Project: Apache Drill Issue Type: Bug Components: Server Affects Versions: 1.21.1 Reporter: shihuafeng Fix For: 1.22.0 [DRILL-8485|[DRILL-8485] HashJoinPOP memory leak is caused by an oom exception when read data from InputStream - ASF JIRA (apache.org)] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DRILL-8489) sender memory
shihuafeng created DRILL-8489: - Summary: sender memory Key: DRILL-8489 URL: https://issues.apache.org/jira/browse/DRILL-8489 Project: Apache Drill Issue Type: Bug Components: Server Affects Versions: 1.21.1 Reporter: shihuafeng Fix For: 1.22.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DRILL-8483) SpilledRecordBatch memory leak when the program threw an exception during the process of building a hash table
shihuafeng created DRILL-8483:
-
Summary: SpilledRecordBatch memory leak when the program threw an
exception during the process of building a hash table
Key: DRILL-8483
URL: https://issues.apache.org/jira/browse/DRILL-8483
Project: Apache Drill
Issue Type: Bug
Components: Server
Affects Versions: 1.21.1
Reporter: shihuafeng
Fix For: 1.21.2
During the process of reading data from disk to building hash tables in memory,
if an exception is thrown, it will result in a memory SpilledRecordBatch leak
exception log as following
{code:java}
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to
allocate buffer of size 8192 due to memory limit (41943040). Current
allocation: 3684352
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:241)
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216)
at
org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:411)
at
org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:270)
at
org.apache.drill.exec.physical.impl.common.HashPartition.allocateNewVectorContainer(HashPartition.java:215)
at
org.apache.drill.exec.physical.impl.common.HashPartition.allocateNewCurrentBatchAndHV(HashPartition.java:238)
at
org.apache.drill.exec.physical.impl.common.HashPartition.(HashPartition.java:165){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (DRILL-8484) An oom exception occurred When read data from Stream with container ,resulting in hashJoinPOP memory leak
shihuafeng created DRILL-8484:
-
Summary: An oom exception occurred When read data from Stream with
container ,resulting in hashJoinPOP memory leak
Key: DRILL-8484
URL: https://issues.apache.org/jira/browse/DRILL-8484
Project: Apache Drill
Issue Type: Bug
Components: Server
Affects Versions: 1.21.1
Reporter: shihuafeng
Fix For: 1.22.0
*Describe the bug*
An oom exception occurred When read data from Stream with container ,resulting
in hashJoinPOP memory leak
*To Reproduce*
prepare data for tpch 1s
# 30 concurrent for tpch sql8
# set direct memory 5g
# when it had OutOfMemoryException , stopped all sql.
# finding memory leak
*leak info*
{code:java}
Allocator(frag:5:0) 500/100/31067136/40041943040
(res/actual/peak/limit)
child allocators: 1
Allocator(op:5:0:1:HashJoinPOP) 100/16384/22822912/41943040
(res/actual/peak/limit)
child allocators: 0
ledgers: 2
ledger[1882757] allocator: op:5:0:1:HashJoinPOP), isOwning: true,
size: 8192, references: 2, life: 16936270178816167..0, allocatorManager:
[1703465, life: 16936270178813617..0] holds 4 buffers.
DrillBuf[2041995], udle: [1703441 0..957]{code}
execption info
{code:java}
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to
allocate buffer of size 16384 (rounded from 14359) due to memory limit
(41943040). Current allocation: 22583616
at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:241)
at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216)
at
org.apache.drill.exec.cache.VectorAccessibleSerializable.readFromStreamWithContainer(VectorAccessibleSerializable.java:172)
at
org.apache.drill.exec.physical.impl.aggregate.SpilledRecordBatch.next(SpilledRecordBatch.java:157)
... 18 common frames omitted{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (DRILL-8485) HashJoinPOP memory leak is caused by an oom exception when read data from InputStream
shihuafeng created DRILL-8485: - Summary: HashJoinPOP memory leak is caused by an oom exception when read data from InputStream Key: DRILL-8485 URL: https://issues.apache.org/jira/browse/DRILL-8485 Project: Apache Drill Issue Type: Bug Components: Server Affects Versions: 1.21.1 Reporter: shihuafeng Fix For: 1.21.1 when traversing fieldList druing read data from InputStream, if the intermediate process throw exception,we can not release previously constructed vectors. it result in memory leak。 it is similar to [DRILL-8484] HashJoinPOP memory leak is caused by an oom exception when read data from Stream with container - ASF JIRA (apache.org) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DRILL-8482) Assign region throw exception when some region is deployed on affinity node and some on non-affinity node
shihuafeng created DRILL-8482:
-
Summary: Assign region throw exception when some region is
deployed on affinity node and some on non-affinity node
Key: DRILL-8482
URL: https://issues.apache.org/jira/browse/DRILL-8482
Project: Apache Drill
Issue Type: Bug
Components: Storage - HBase
Affects Versions: 1.21.1
Reporter: shihuafeng
Fix For: 1.22.0
*Describe the bug*
Assign region throw exception when some region is deployed on affinity node
and some on non-affinity node。
*To Reproduce*
Steps to reproduce the behavior:
#
{code:java}
NavigableMap regionsToScan = Maps.newTreeMap();
regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[0], splits[1]), SERVER_A);
regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[1], splits[2]), SERVER_A);
regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[2], splits[3]), SERVER_B);
regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[3], splits[4]), SERVER_B);
regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[6], splits[7]), SERVER_D);
regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[7], splits[8]), SERVER_D);
regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[8], splits[9]), SERVER_D);
regionsToScan.put(new HRegionInfo(TABLE_NAME, splits[9], splits[10]), SERVER_D);
final List endpoints = Lists.newArrayList();
endpoints.add(DrillbitEndpoint.newBuilder().setAddress(HOST_A).setControlPort(1234).build());
endpoints.add(DrillbitEndpoint.newBuilder().setAddress(HOST_B).setControlPort(1234).build());
endpoints.add(DrillbitEndpoint.newBuilder().setAddress(HOST_C).setControlPort(1234).build());
HBaseGroupScan scan = new HBaseGroupScan();
scan.setRegionsToScan(regionsToScan);
scan.setHBaseScanSpec(new HBaseScanSpec(TABLE_NAME_STR, splits[0], splits[0],
null));
scan.applyAssignments(endpoints);{code}
*Expected behavior*
*Error detail, log output or screenshots*
{code:java}
Caused by: java.lang.NullPointerException: null
at
org.apache.drill.exec.store.hbase.HBaseGroupScan.applyAssignments(HBaseGroupScan.java:283){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
