date:20170413

[GitHub] drill pull request #783: DRILL-5324: Provide simplified column reader/writer...

2017-04-13 Thread bitblender

Github user bitblender commented on a diff in the pull request:

https://github.com/apache/drill/pull/783#discussion_r111289506
  
--- Diff: exec/vector/src/main/codegen/templates/ColumnAccessors.java ---
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+<@pp.dropOutputFile />
+<@pp.changeOutputFile 
name="/org/apache/drill/exec/vector/accessor/ColumnAccessors.java" />
+<#include "/@includes/license.ftl" />
+<#macro getType label>
+@Override
+public ValueType valueType() {
+  <#if label == "Int">
+  return ValueType.INTEGER;
+  <#else>
+  return ValueType.${label?upper_case};
+  
+}
+
+<#macro bindReader prefix drillType>
+  <#if drillType = "Decimal9" || drillType == "Decimal18">
+private MaterializedField field;
+  
+private ${prefix}${drillType}Vector.Accessor accessor;
+
+@Override
+public void bind(RowIndex vectorIndex, ValueVector vector) {
+  bind(vectorIndex);
+  <#if drillType = "Decimal9" || drillType == "Decimal18">
+  field = vector.getField();
+  
+  accessor = ((${prefix}${drillType}Vector) vector).getAccessor();
+}
+
+  <#if drillType = "Decimal9" || drillType == "Decimal18">
+@Override
+public void bind(RowIndex vectorIndex, MaterializedField field, 
VectorAccessor va) {
+  bind(vectorIndex, field, va);
+  this.field = field;
+}
+
+ 
+   private ${prefix}${drillType}Vector.Accessor accessor() {
+  if (vectorAccessor == null) {
+return accessor;
+  } else {
+return ((${prefix}${drillType}Vector) 
vectorAccessor.vector()).getAccessor();
+  }
+}
+
+<#macro get drillType accessorType label isArray>
+@Override
+public ${accessorType} get${label}(<#if isArray>int index) {
+  <#if isArray>
+<#assign index=", index"/>
+<#assign getObject="getSingleObject">
+  <#else>
+<#assign index=""/>
+<#assign getObject="getObject">
+  
+  <#if drillType == "VarChar">
+  return new String(accessor().get(vectorIndex.index()${index}), 
Charsets.UTF_8);
+  <#elseif drillType == "Var16Char">
+  return new String(accessor().get(vectorIndex.index()${index}), 
Charsets.UTF_16);
+  <#elseif drillType == "VarBinary">
+  return accessor().get(vectorIndex.index()${index});
+  <#elseif drillType == "Decimal9" || drillType == "Decimal18">
+  return DecimalUtility.getBigDecimalFromPrimitiveTypes(
+accessor().get(vectorIndex.index()${index}),
+field.getScale(),
+field.getPrecision());
+  <#elseif accessorType == "Decimal18">
+  return 
DecimalUtilities.getBigDecimalFromPrimitiveTypes(accessor().${getObject}(vectorIndex.index()${index});
--- End diff --

As discusses offline, this seems to be deadcode as there is no 
DecimalUtilities class in the Drill source base.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #805: Drill-4139: Exception while trying to prune partiti...

2017-04-13 Thread jinfengni

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/805#discussion_r111501238
  
--- Diff: 
exec/java-exec/src/main/codegen/templates/CastVarBinaryInterval.java ---
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+<@pp.dropOutputFile />
+
+<#list cast.types as type>
+<#if type.major == "VarBinaryInterval">  <#-- Template to convert from 
VarBinary to Interval, IntervalYear, IntervalDay -->
+
+<@pp.changeOutputFile 
name="/org/apache/drill/exec/expr/fn/impl/gcast/Cast${type.from}To${type.to}.java"
 />
+
+<#include "/@includes/license.ftl" />
+
+package org.apache.drill.exec.expr.fn.impl.gcast;
+
+import io.netty.buffer.ByteBuf;
+
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import 
org.apache.drill.exec.expr.annotations.FunctionTemplate.NullHandling;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.holders.IntervalDayHolder;
+import org.apache.drill.exec.expr.holders.IntervalHolder;
+import org.apache.drill.exec.expr.holders.IntervalYearHolder;
+import org.apache.drill.exec.expr.holders.VarBinaryHolder;
+
+/*
+ * This class is generated using freemarker and the ${.template_name} 
template.
+ */
+@SuppressWarnings("unused")
+@FunctionTemplate(name = "cast${type.to?upper_case}",
+  scope = FunctionTemplate.FunctionScope.SIMPLE,
+  nulls = NullHandling.NULL_IF_NULL)
+public class Cast${type.from}To${type.to} implements DrillSimpleFunc {
+
+  @Param ${type.from}Holder in;
+  @Output ${type.to}Holder out;
+
+  public void setup() {
+  }
+
+  public void eval() {
+final byte[] buf = new byte[in.end - in.start];
+in.buffer.getBytes(in.start, buf, 0, in.end - in.start);
+  <#if type.to == "Interval">
+out.months = 
org.apache.drill.exec.store.parquet.ParquetReaderUtility.getIntFromLEBytes(buf, 
0);
--- End diff --

I'm a bit confused.  Are you assuming VarBinary is encoded as 3 int values 
: # of months, # of days and # of millisecon?  If varbinary is encoded in such 
way, which is parquet used to encode logical type "interval", why do we have to 
cast ? 

My understanding is varbinary looks similar to varchar, except that the 
former contains byte while the latter contains character.   

 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #805: Drill-4139: Exception while trying to prune partiti...

2017-04-13 Thread jinfengni

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/805#discussion_r111505132
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java
 ---
@@ -508,7 +511,10 @@ public void populatePruningVector(ValueVector v, int 
index, SchemaPath column, S
 NullableVarBinaryVector varBinaryVector = 
(NullableVarBinaryVector) v;
 Object s = partitionValueMap.get(f).get(column);
 byte[] bytes;
-if (s instanceof Binary) {
+if (s == null) {
+  varBinaryVector.getMutator().setNull(index);
--- End diff --

Should we apply this setNull() to all other types? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #805: Drill-4139: Exception while trying to prune partiti...

2017-04-13 Thread jinfengni

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/805#discussion_r111504302
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/Metadata.java 
---
@@ -1668,9 +1666,25 @@ public ColumnMetadata_v3(String[] name, 
PrimitiveTypeName primitiveType, Object
   return nulls;
 }
 
+/**
+ * Checks that the column chunk has single value.
+ * Returns true if minValue and maxValue are the same, but not null,
+ * and all column chunk values are not null.
+ * Returns true if minValue and maxValue are null and null values 
count in
+ * the column chunk is greater than 0.
+ *
+ * @return true if column has single value
+ */
 @Override
 public boolean hasSingleValue() {
-  return (minValue !=null && maxValue != null && 
minValue.equals(maxValue));
+  if (nulls != null) {
--- End diff --

Should this applies to v2 and v1 of ColumnMetadata? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5436) Need a way to input password which contains space when calling sqlline

2017-04-13 Thread Hao Zhu (JIRA)

Hao Zhu created DRILL-5436:
--

 Summary: Need a way to input password which contains space when 
calling sqlline
 Key: DRILL-5436
 URL: https://issues.apache.org/jira/browse/DRILL-5436
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - CLI
Affects Versions: 1.10.0
Reporter: Hao Zhu


create a user named "spaceuser" with password "hello world".
All below failed:
{code}
sqlline -u jdbc:drill:zk=xxx -n spaceuser -p 'hello world'
sqlline -u jdbc:drill:zk=xxx -n spaceuser -p "hello world"
sqlline -u jdbc:drill:zk=xxx -n spaceuser -p 'hello\ world'
sqlline -u jdbc:drill:zk=xxx -n spaceuser -p "hello\ world"
{code}

Need a way to input password which contains space when calling sqlline



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] drill issue #815: DRILL-5424: Fix IOBE for reverse function

2017-04-13 Thread arina-ielchiieva

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/815
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5435) Using Limit causes Memory Leaked Error since 1.10

2017-04-13 Thread JIRA

F Méthot created DRILL-5435:
---

 Summary: Using Limit causes Memory Leaked Error since 1.10
 Key: DRILL-5435
 URL: https://issues.apache.org/jira/browse/DRILL-5435
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - Parquet
Affects Versions: 1.10.0
Reporter: F Méthot


Here is the details I can provide:

  We migrated our production system from Drill 1.9 to 1.10 just 5 days ago. 
(220 nodes cluster)

Our log show there was some 900+ queries ran without problem in first 4 days.  
(similar queries, that never use the `limit` clause)

Yesterday we started doing simple adhoc select * ... limit 10 queries (like we 
often do, that was our first use of limit with 1.10)
and we got a `Memory was leaked` exception below.

Also, once we get the error, Most of all subsequent user queries fails with 
Channel Close Exception. We need to restart Drill to bring it back to normal.

A day later, I used a similar select * limit 10 queries, and the same thing 
happen, had to restart Drill.

In the exception, it was refering to a file (1_0_0.parquet)
I moved that file to smaller test cluster (12 nodes) and got the error on the 
first attempt. but I am no longer able to reproduce the issue on that file. 
Between the 12 and 220 nodes cluster, a different Column name and Row Group 
Start was listed in the error.
The parquet file was generated by Drill 1.10.

I tried the same file with a local drill-embedded 1.9 and 1.10 and had no issue.


Here is the error (manually typed), if you think of anything obvious, let us 
know.


AsyncPageReader - User Error Occured: Exception Occurred while reading from 
disk (can not read class o.a.parquet.format.PageHeader: java.io.IOException: 
input stream is closed.)

File:/1_0_0.parquet
Column: StringColXYZ
Row Group Start: 115215476

[Error Id: ]
  at UserException.java:544)
  at 
o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.handleAndThrowException(AsynvPageReader.java:199)
  at 
o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.access(AsynvPageReader.java:81)
  at 
o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.AsyncPageReaderTask.call(AsyncPageReader.java:483)
  at 
o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.AsyncPageReaderTask.call(AsyncPageReader.java:392)
  at 
o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.AsyncPageReaderTask.call(AsyncPageReader.java:392)
...
Caused by: java.io.IOException: can not read class 
org.apache.parquet.format.PageHeader: java.io.IOException: Input Stream is 
closed.
   at o.a.parquet.format.Util.read(Util.java:216)
   at o.a.parquet.format.Util.readPageHeader(Util.java:65)
   at 
o.a.drill.exec.store.parquet.columnreaders.AsyncPageReader(AsyncPageReaderTask:430)
Caused by: parquet.org.apache.thrift.transport.TTransportException: Input 
stream is closed
   at ...read(TIOStreamTransport.java:129)
   at TTransport.readAll(TTransport.java:84)
   at TCompactProtocol.readByte(TCompactProtocol.java:474)
   at TCompactProtocol.readFieldBegin(TCompactProtocol.java:481)
   at InterningProtocol.readFieldBegin(InterningProtocol.java:158)
   at o.a.parquet.format.PageHeader.read(PageHeader.java:828)
   at o.a.parquet.format.Util.read(Util.java:213)


Fragment 0:0
[Error id: ...]
o.a.drill.common.exception.UserException: SYSTEM ERROR: IllegalStateException: 
Memory was leaked by query. Memory leaked: (524288)
Allocator(op:0:0:4:ParquetRowGroupScan) 100/524288/39919616/100
  at o.a.d.common.exceptions.UserException (UserException.java:544)
  at 
o.a.d.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
  at o.a.d.exec.work.fragment.FragmentExecutor.cleanup( 
FragmentExecutor.java:160)
  at o.a.d.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
...
Caused by: IllegalStateException: Memory was leaked by query. Memory leaked: 
(524288)
  at o.a.d.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
  at o.a.d.exec.ops.OperatorContextImpl(OperatorContextImpl.java:149)
  at o.a.d.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422)
  at o.a.d.exec.ops.FragmentContext.close(FragmentContext.java:411)
  at 
o.a.d.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318)
  at 
o.a.d.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)




This fixed the problem:
alter  set `store.parquet.reader.pagereader.async`=false;



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Memory was Leaked error when using "limit" in 1.10

2017-04-13 Thread François Méthot

Yes it did, the problem is gone. Thanks

I will share the details I have on a Jira ticket now.



On Tue, Apr 11, 2017 at 9:22 PM, Kunal Khatua  wrote:

> Did this help resolve the memory leak, Francois?
>
>
> Could you share the stack trace and other relevant logs on a JIRA?
>
>
> Thanks
>
> Kunal
>
>
>
>
> 
> From: Kunal Khatua 
> Sent: Wednesday, April 5, 2017 2:03:19 PM
> To: dev@drill.apache.org
> Subject: Re: Memory was Leaked error when using "limit" in 1.10
>
> Hi Francois
>
> Could you try those queries with the AsyncPageReader turned off?
>
> alter  set `store.parquet.reader.pagereader.async`=false;
>
> For Drill 1.9+ , this feature is enabled. However, there were some perf
> related improvements that Drill 1.10 carried out.
>
> If the problem goes away, could you file a JIRA and share the sample query
> and data to allow us a repro ?
>
> Thanks
>
> Kunal
>
> 
> From: François Méthot 
> Sent: Wednesday, April 5, 2017 1:39:38 PM
> To: dev@drill.apache.org
> Subject: Memory was Leaked error when using "limit" in 1.10
>
> Hi,
>
>   I am still investigating this problem, but I will describe the symptom to
> you in case there is known issue with drill 1.10.
>
>   We migrated our production system from Drill 1.9 to 1.10 just 5 days ago.
> (220 nodes cluster)
>
> Our log show there was some 900+ queries ran without problem in first 4
> days.  (similar queries, that never use the `limit` clause)
>
> Yesterday we started doing simple adhoc select * ... limit 10 queries (like
> we often do, that was our first use of limit with 1.10)
> and we got a `Memory was leaked` exception below.
>
> Also, once we get the error, Most of all subsequent user queries fails with
> Channel Close Exception. We need to restart Drill to bring it back to
> normal.
>
> A day later, I used a similar select * limit 10 queries, and the same thing
> happen, had to restart Drill.
>
> In the exception, it was refering to a file (1_0_0.parquet)
> I moved that file to smaller test cluster (12 nodes) and got the error on
> the first attempt. but I am no longer able to reproduce the issue on that
> file. Between the 12 and 220 nodes cluster, a different Column name and Row
> Group Start was listed in the error.
> The parquet file was generated by Drill 1.10.
>
> I tried the same file with a local drill-embedded 1.9 and 1.10 and had no
> issue.
>
>
> Here is the error (manually typed), if you think of anything obvious, let
> us know.
>
>
> AsyncPageReader - User Error Occured: Exception Occurred while reading from
> disk (can not read class o.a.parquet.format.PageHeader:
> java.io.IOException: input stream is closed.)
>
> File:/1_0_0.parquet
> Column: StringColXYZ
> Row Group Start: 115215476
>
> [Error Id: ]
>   at UserException.java:544)
>   at
> o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.
> handleAndThrowException(AsynvPageReader.java:199)
>   at
> o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.
> access(AsynvPageReader.java:81)
>   at
> o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.
> AsyncPageReaderTask.call(AsyncPageReader.java:483)
>   at
> o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.
> AsyncPageReaderTask.call(AsyncPageReader.java:392)
>   at
> o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.
> AsyncPageReaderTask.call(AsyncPageReader.java:392)
> ...
> Caused by: java.io.IOException: can not read class
> org.apache.parquet.format.PageHeader: java.io.IOException: Input Stream is
> closed.
>at o.a.parquet.format.Util.read(Util.java:216)
>at o.a.parquet.format.Util.readPageHeader(Util.java:65)
>at
> o.a.drill.exec.store.parquet.columnreaders.AsyncPageReader(
> AsyncPageReaderTask:430)
> Caused by: parquet.org.apache.thrift.transport.TTransportException: Input
> stream is closed
>at ...read(TIOStreamTransport.java:129)
>at TTransport.readAll(TTransport.java:84)
>at TCompactProtocol.readByte(TCompactProtocol.java:474)
>at TCompactProtocol.readFieldBegin(TCompactProtocol.java:481)
>at InterningProtocol.readFieldBegin(InterningProtocol.java:158)
>at o.a.parquet.format.PageHeader.read(PageHeader.java:828)
>at o.a.parquet.format.Util.read(Util.java:213)
>
>
> Fragment 0:0
> [Error id: ...]
> o.a.drill.common.exception.UserException: SYSTEM ERROR:
> IllegalStateException: Memory was leaked by query. Memory leaked: (524288)
> Allocator(op:0:0:4:ParquetRowGroupScan) 100/524288/39919616/
> 100
>   at o.a.d.common.exceptions.UserException (UserException.java:544)
>   at
> o.a.d.exec.work.fragment.FragmentExecutor.sendFinalState(
> FragmentExecutor.java:293)
>   at o.a.d.exec.work.fragment.FragmentExecutor.cleanup(
> FragmentExecutor.java:160)
>   at
> o.a.d.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
> ...
> Caused by: IllegalStateException: Memory was leaked

Re: [jira] [Created] (DRILL-5432) Want a memory format for PCAP files

2017-04-13 Thread François Méthot

Hi Ted,

  We did a proof of concept with reading pcap from drill. Our approach was
to avoid writing yet another pcap decoder so we tried to adapt Drill to use
an existing one. We took Tshark as an example. It already comes with 1000s
of dissectors.

We approached the problem from a different angle: How to drive and read the
output of an external application from a SQL query within Drill.

Our experiment started with the Text Input Storage plugin from Drill, we
modified slightly to be a .pcap plugin.

When a Drill query is run on a pcap file,  the plugin RecordReader setup
function launches the TShark external app for each file that drill needs to
scan.
The column specified in the select statement are passed as an input
parameter to the external application.

In RecordReader next() method, it reads each record streamed back by
TShark. The stream output of the process is parsed by a slightly modified
TextInput. Once the data is streamed in the drill space, user can leverage
on the SQL Language to do all kind of data aggregation.

For this technique to work, the external application needs to support
Streaming in and out data.

To run on HDFS with a native application that has not been build for HDFS,
the storage plugin launches: "hdfs cat test.pcap | tshark "

For this to work, TShark needs to be deployed everywhere a drill bit is
running.

I don't have any metrics on performance, this was a proof of concept, but
it works. It will probably not beat the performance of the solution you are
aiming, but it leverages on years of development of an existing tool.

Francois

On Wed, Apr 12, 2017 at 2:25 PM, Ted Dunning (JIRA)  wrote:

> Ted Dunning created DRILL-5432:
> --
>
>  Summary: Want a memory format for PCAP files
>  Key: DRILL-5432
>  URL: https://issues.apache.org/jira/browse/DRILL-5432
>  Project: Apache Drill
>   Issue Type: New Feature
> Reporter: Ted Dunning
>
>
> PCAP files [1] are the de facto standard for storing network capture data.
> In security and protocol applications, it is very common to want to extract
> particular packets from a capture for further analysis.
>
> At a first level, it is desirable to query and filter by source and
> destination IP and port or by protocol. Beyond that, however, it would be
> very useful to be able to group packets by TCP session and eventually to
> look at packet contents. For now, however, the most critical requirement is
> that we should be able to scan captures at very high speed.
>
> I previously wrote a (kind of working) proof of concept for a PCAP decoder
> that did lazy deserialization and could traverse hundreds of MB of PCAP
> data per second per core. This compares to roughly 2-3 MB/s for widely
> available Apache-compatible open source PCAP decoders.
>
> This JIRA covers the integration and extension of that proof of concept as
> a Drill file format.
>
> Initial work is available at https://github.com/mapr-demos/pcap-query
>
>
> [1] https://en.wikipedia.org/wiki/Pcap
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.15#6346)
>

[jira] [Created] (DRILL-5434) IllegalStateException: Memory was leaked by query.

2017-04-13 Thread Khurram Faraaz (JIRA)

Khurram Faraaz created DRILL-5434:
-

 Summary: IllegalStateException: Memory was leaked by query.
 Key: DRILL-5434
 URL: https://issues.apache.org/jira/browse/DRILL-5434
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Khurram Faraaz


Issue a long running COUNT query.
While the query is being executed, stop the foreman drillbit, ./drillbit.sh stop
A memory leak is reported in the drillbit.log

Apache Drill 1.11.0 
git.commit.id.abbrev=06e1522

Stack trace from drillbit.log
{noformat}
2017-04-13 06:14:36,828 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO  
o.a.drill.exec.work.foreman.Foreman - Query text for query id 
2710e8b2-d4dc-1bee-016e-b69fd4966916: SELECT COUNT(*) FROM `twoKeyJsn.json`
2017-04-13 06:14:36,929 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-04-13 06:14:36,929 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-04-13 06:14:36,929 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-04-13 06:14:36,929 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-04-13 06:14:36,930 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-04-13 06:14:36,930 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-04-13 06:14:36,930 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-04-13 06:14:36,932 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-04-13 06:14:36,934 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO  
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 
1 threads. Time: 2ms total, 2.102992ms avg, 2ms max.
2017-04-13 06:14:36,934 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO  
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 
1 threads. Earliest start: 0.555000 μs, Latest start: 0.555000 μs, Average 
start: 0.555000 μs .
2017-04-13 06:14:36,949 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 2710e8b2-d4dc-1bee-016e-b69fd4966916:0:0: 
State change requested AWAITING_ALLOCATION --> RUNNING
2017-04-13 06:14:36,949 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] INFO  
o.a.d.e.w.f.FragmentStatusReporter - 2710e8b2-d4dc-1bee-016e-b69fd4966916:0:0: 
State to report: RUNNING
Thu Apr 13 06:14:40 UTC 2017 Terminating drillbit pid 5107
2017-04-13 06:14:40,756 [Drillbit-ShutdownHook#0] INFO  
o.apache.drill.exec.server.Drillbit - Received shutdown request.
2017-04-13 06:14:47,819 [pool-169-thread-2] INFO  
o.a.drill.exec.rpc.data.DataServer - closed eventLoopGroup 
io.netty.channel.nio.NioEventLoopGroup@4f3ee67c in 1024 ms
2017-04-13 06:14:47,819 [pool-169-thread-2] INFO  
o.a.drill.exec.service.ServiceEngine - closed dataPool in 1024 ms
2017-04-13 06:14:49,806 [Drillbit-ShutdownHook#0] WARN  
o.apache.drill.exec.work.WorkManager - Closing WorkManager but there are 1 
running fragments.
2017-04-13 06:14:49,807 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 2710e8b2-d4dc-1bee-016e-b69fd4966916:0:0: 
State change requested RUNNING --> FAILED
2017-04-13 06:14:49,807 [Drillbit-ShutdownHook#0] INFO  
o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 6964, cache miss 
count: 335, hit rate: 95%
2017-04-13 06:14:49,807 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 2710e8b2-d4dc-1bee-016e-b69fd4966916:0:0: 
State change requested FAILED --> FINISHED
2017-04-13 06:14:49,809 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] ERROR 
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException

Fragment 0:0

[Error Id: 94817261-98a9-4153-8b3a-2d9c95d80cc1 on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
NullPointerException

Fragment 0:0

[Error Id: 94817261-98a9-4153-8b3a-2d9c95d80cc1 on centos-01.qa.lab:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
 ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)

[GitHub] drill pull request #783: DRILL-5324: Provide simplified column reader/writer...

[GitHub] drill pull request #805: Drill-4139: Exception while trying to prune partiti...

[GitHub] drill pull request #805: Drill-4139: Exception while trying to prune partiti...

[GitHub] drill pull request #805: Drill-4139: Exception while trying to prune partiti...

[jira] [Created] (DRILL-5436) Need a way to input password which contains space when calling sqlline

[GitHub] drill issue #815: DRILL-5424: Fix IOBE for reverse function

[jira] [Created] (DRILL-5435) Using Limit causes Memory Leaked Error since 1.10

Re: Memory was Leaked error when using "limit" in 1.10

Re: [jira] [Created] (DRILL-5432) Want a memory format for PCAP files

[jira] [Created] (DRILL-5434) IllegalStateException: Memory was leaked by query.

10 matches

Site Navigation

Mail list logo

Footer information