Re: [ANNOUNCE] New PMC Chair of Apache Drill

[Khurram Faraaz]

Congratulations Charles, and thank you Arina.

Regards,
Khurram

On Fri, Aug 23, 2019 at 2:54 PM Niels Basjes  wrote:

> Congratulations Charles.
>
> Niels Basjes
>
> On Thu, Aug 22, 2019, 09:28 Arina Ielchiieva  wrote:
>
> > Hi all,
> >
> > It has been a honor to serve as Drill Chair during the past year but it's
> > high time for the new one...
> >
> > I am very pleased to announce that the Drill PMC has voted to elect
> Charles
> > Givre as the new PMC chair of Apache Drill. He has also been approved
> > unanimously by the Apache Board in last board meeting.
> >
> > Congratulations, Charles!
> >
> > Kind regards,
> > Arina
> >
>

Re: Multi char csv delimiter

[Khurram Faraaz]

Hi Matthias,

Like Paul mentioned, that information with examples can be found here.
Our documentation mentions that, "As of Drill 1.8, Drill supports
multi-byte delimiters, such as \r\n. "
https://drill.apache.org/docs/plugin-configuration-basics/

Thanks,
Khurram

On Mon, Jun 24, 2019 at 2:31 PM Paul Rogers 
wrote:

> Hi Matthias,
>
> Field delimiters, quotes and quote escapes can be only one character. The
> line delimiter can be multi.
>
> Are you setting the line delimiter?
>
> - Paul
>
> Sent from my iPhone
>
> > On Jun 24, 2019, at 12:10 PM, Arina Yelchiyeva <
> arina.yelchiy...@gmail.com> wrote:
> >
> > Hi Matthias,
> >
> > Attachments are not supported on the mailing list, please include text
> describing your configuration.
> >
> > Kind regards,
> > Arina
> >
> >> On Jun 24, 2019, at 2:21 PM, Rosenthaler Matthias (PS-DI/ETF1.1)
>  wrote:
> >>
> >> Hi,
> >>
> >> It seems that multi char delimiter “\n\r” is not supported for csv
> format drill 1.16.
> >> The documentation mentions it should work, but it does not work for me.
> It always says “invalid JSON syntax” if I try to change the storage plugin
> configuration.
> >>
> >>
> >>
> >> Mit freundlichen Grüßen / Best regards
> >>
> >> Matthias Rosenthaler
> >>
> >> Powertrain Solutions, Engine Testing (PS-DI/ETF1.1)
> >> Robert Bosch AG | Robert-Bosch-Straße 1 | 4020 Linz | AUSTRIA |
> www.bosch.at 
> >> Tel. +43 732 7667-479 | matthias.rosentha...@at.bosch.com  matthias.rosentha...@at.bosch.com>
> >>
> >> Sitz: Robert Bosch Aktiengesellschaft, A-1030 Wien, Göllnergasse 15-17
> , Registergericht: FN 55722 w HG-Wien
> >> Aufsichtsratsvorsitzender: Dr. Uwe Thomas; Geschäftsführung: Dr. Klaus
> Peter Fouquet
> >> DVR-Nr.: 0418871- ARA-Lizenz-Nr.: 1831 - UID-Nr.: ATU14719303 -
> Steuernummer 140/4988
> >
>
>

[jira] [Created] (DRILL-7275) CTAS + CTE query fails with IllegalStateException: Read batch count [%d] should be greater than zero [0]

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-7275:
-

 Summary: CTAS + CTE query fails with IllegalStateException: Read 
batch count [%d] should be greater than zero [0]
 Key: DRILL-7275
 URL: https://issues.apache.org/jira/browse/DRILL-7275
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - Parquet
Affects Versions: 1.15.0
Reporter: Khurram Faraaz


CTAS + CTE query fails with IllegalStateException: Read batch count [%d] should 
be greater than zero [0]

Precondition check fails on line 47 in VarLenFixedEntryReader.java

{noformat}
44 final int expectedDataLen = columnPrecInfo.precision;
45 final int entrySz = 4 + columnPrecInfo.precision;
46 final int readBatch = getFixedLengthMaxRecordsToRead(valuesToRead, entrySz);
47 Preconditions.checkState(readBatch > 0, "Read batch count [%d] should be 
greater than zero", readBatch);
{noformat}


Stack trace from drillbit.log, also has the failing query.

{noformat}
2019-05-13 14:40:14,090 [23268c40-ef3a-6349-5901-5762f6888971:foreman] INFO 
o.a.drill.exec.work.foreman.Foreman - Query text for query with id 
23268c40-ef3a-6349-5901-5762f6888971 issued by scoop_stc: CREATE TABLE 
TEST_TEMPLATE_SCHEMA_creid.tbl_c_EquityProxyDailyReturn AS
WITH
che AS (
 SELECT * FROM 
TEST_TEMPLATE_SCHEMA_creid.tbl_c_CompositeHierarchyEntry_TimeVarying
 WHERE CompositeHierarchyName = 'AxiomaRegion/AxiomaSector/VectorUniverse'
 AND state = 'DupesRemoved'
 AND CompositeLevel = 'AxiomaRegion_1/AxiomaSector_1/VectorUniverse_0'
),
ef AS (SELECT * FROM 
TEST_TEMPLATE_SCHEMA_creid.tbl_c_EquityDailyReturn_FXAdjusted WHERE Status = 
'PresentInRawData'),
d AS (SELECT * FROM TEST_TEMPLATE_SCHEMA_creid.tbl_r_BusinessDate WHERE 
IsWeekday),
x AS
(
 SELECT
 che.CompositeHierarchyName,
 che.State,
 che.CompositeNodeName,
 d.`Date` AS RecordDate,
 COUNT(che.CompositeNodeName) AS countDistinctConstituents,
 COUNT(ef.VectorListingId) AS countDataPoints,
 AVG(ef.DailyReturn) AS AvgReturn, 
 AVG(ef.DailyReturnUSD) AS AvgReturnUSD,
 AVG(ef.NotionalReturnUSD) AS AvgNotionalReturnUSD
 FROM d
 INNER JOIN che ON d.`Date` BETWEEN che.CompositeUltimateChildStartDate AND 
che.CompositeUltimateChildEndDate
 LEFT OUTER JOIN ef ON d.`Date` = ef.RecordDate AND 'VectorListingId_' || 
CAST(ef.VectorListingId AS VARCHAR(100)) = che.UltimateChild
 GROUP BY che.CompositeHierarchyName, che.State, che.CompositeNodeName, 
d.`Date`, d.IsWeekday, d.IsHoliday
)
SELECT * FROM x
2019-05-13 14:40:16,971 [23268c40-ef3a-6349-5901-5762f6888971:foreman] INFO 
o.a.d.e.p.s.h.CreateTableHandler - Creating persistent table 
[tbl_c_EquityProxyDailyReturn].
...
...
2019-05-13 14:40:20,036 [23268c40-ef3a-6349-5901-5762f6888971:frag:6:10] INFO 
o.a.d.exec.physical.impl.ScanBatch - User Error Occurred: Error in parquet 
record reader.
Message:
Hadoop path: /DEV/tbl_c_EquityDailyReturn_FXAdjusted/1_32_32.parquet
Total records read: 0
Row group index: 0
Records in row group: 3243
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {
 optional int64 VectorListingId;
 optional int32 RecordDate (DATE);
 required binary Status (UTF8);
 required binary CurrencyISO (UTF8);
 optional double DailyReturn;
 optional double DailyReturnUSD;
 optional double NotionalReturnUSD;
}
, metadata: \{drill-writer.version=2, drill.version=1.15.0.0-mapr}}, blocks: 
[BlockMetaData\{3243, 204762 [ColumnMetaData{UNCOMPRESSED [VectorListingId] 
optional int64 VectorListingId [RLE, BIT_PACKED, PLAIN], 4}, 
ColumnMetaData\{UNCOMPRESSED [RecordDate] optional int32 RecordDate (DATE) 
[RLE, BIT_PACKED, PLAIN], 26021}, ColumnMetaData\{UNCOMPRESSED [Status] 
required binary Status (UTF8) [BIT_PACKED, PLAIN], 39050}, 
ColumnMetaData\{UNCOMPRESSED [CurrencyISO] required binary CurrencyISO (UTF8) 
[BIT_PACKED, PLAIN], 103968}, ColumnMetaData\{UNCOMPRESSED [DailyReturn] 
optional double DailyReturn [RLE, BIT_PACKED, PLAIN], 126715}, 
ColumnMetaData\{UNCOMPRESSED [DailyReturnUSD] optional double DailyReturnUSD 
[RLE, BIT_PACKED, PLAIN], 152732}, ColumnMetaData\{UNCOMPRESSED 
[NotionalReturnUSD] optional double NotionalReturnUSD [RLE, BIT_PACKED, PLAIN], 
178749}]}]} (Error in parquet record reader.
...
...
Hadoop path: /DEV/tbl_c_EquityDailyReturn_FXAdjusted/1_32_32.parquet
Total records read: 0
Row group index: 0
Records in row group: 3243
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {
 optional int64 VectorListingId;
 optional int32 RecordDate (DATE);
 required binary Status (UTF8);
 required binary CurrencyISO (UTF8);
 optional double DailyReturn;
 optional double DailyReturnUSD;
 optional double NotionalReturnUSD;
}
, metadata: \{drill-writer.version=2, drill.version=1.15.0.0-mapr}}, blocks: 
[BlockMetaData\{3243, 204762 [ColumnMetaData{UNCOMPRESSED [VectorListingId] 
optional int64 VectorListingId [RLE, BIT_PACKED, PLAIN], 4}, 
ColumnMetaData\{UNCOMPRESSED [RecordDate] optional int32 RecordDate

[jira] [Created] (DRILL-7256) Query over empty Hive tables fails, we will need to print heap usagePercent details in error message

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-7256:
-

 Summary: Query over empty Hive tables fails, we will need to print 
heap usagePercent details in error message
 Key: DRILL-7256
 URL: https://issues.apache.org/jira/browse/DRILL-7256
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.15.0
Reporter: Khurram Faraaz


The below query from Drill's web UI on Hive tables failed due to not enough 
heap memory to run this query. 
It fails intermittently from Drill web UI, and note that the two Hive tables 
used in the query are empty, meaning they have no data in them. The query does 
not fail when run from sqlline.

The error message does not provide information about the usagePercent of heap.
It will be useful to provide heap usagePercent information as part of the error 
message in QueryWrapper.java when usagePercent > HEAP_MEMORY_FAILURE_THRESHOLD

Drill 1.15.0

Failing query.
{noformat}
SELECT a.event_id
 FROM hive.cust_bhsf_ce_blob a, hive.t_fct_clinical_event b
 where 
 a.event_id=b.event_id
 and a.blob_contents not like '%dd:contenttype="TESTS"%'
 and b.EVENT_RELATIONSHIP_CD='B'
and b.EVENT_CLASS_CD in ('DOC')
and b.entry_mode_cd='Web'
and b.RECORD_STATUS_CD='Active'
and b.RESULT_STATUS_CD ='Auth (Verified)'
and substring(b.valid_until_dt_tm,1,10) >='2017-12-30'
and substring(b.event_end_date,1,10) >='2018-01-01'
{noformat}

Stack trace from drillbit.log 
{noformat}
2019-05-09 16:25:58,472 [qtp1934687-790] ERROR 
o.a.d.e.server.rest.QueryResources - Query from Web UI Failed
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: There is not 
enough heap memory to run this query using the web interface.

Please try a query with fewer columns or with a filter or limit condition to 
limit the data returned.
You can also try an ODBC/JDBC client.

[Error Id: 91668f42-d88e-426b-b1fe-c0d042700500 ]
 at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
 ~[drill-common-1.15.0.5-mapr.jar:1.15.0.5-mapr]
 at org.apache.drill.exec.server.rest.QueryWrapper.run(QueryWrapper.java:103) 
~[drill-java-exec-1.15.0.5-mapr.jar:1.15.0.5-mapr]
 at 
org.apache.drill.exec.server.rest.QueryResources.submitQueryJSON(QueryResources.java:72)
 ~[drill-java-exec-1.15.0.5-mapr.jar:1.15.0.5-mapr]
 at 
org.apache.drill.exec.server.rest.QueryResources.submitQuery(QueryResources.java:87)
 ~[drill-java-exec-1.15.0.5-mapr.jar:1.15.0.5-mapr]
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151]
 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[na:1.8.0_151]
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[na:1.8.0_151]
 at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
 at 
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
 [jersey-server-2.8.jar:na]
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
 [jersey-server-2.8.jar:na]
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
 [jersey-server-2.8.jar:na]
 at 
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
 [jersey-server-2.8.jar:na]
 at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
 [jersey-server-2.8.jar:na]
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387)
 [jersey-server-2.8.jar:na]
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331)
 [jersey-server-2.8.jar:na]
 at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103)
 [jersey-server-2.8.jar:na]
 at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) 
[jersey-server-2.8.jar:na]
 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) 
[jersey-common-2.8.jar:na]
 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) 
[jersey-common-2.8.jar:na]
 at org.glassfish.jersey.internal.Errors.process(Errors.java:315) 
[jersey-common-2.8.jar:na]
 at org.glassfish.jersey.internal.Errors.process(Errors.java:297) 
[jersey-common-2.8.jar:na]
 at org.glassfish.jersey.internal.Errors.process(Errors.java:267) 
[jersey-common-2.8.jar:na]
 at 
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
 [jersey-common-2.8.jar:na]
 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:252) 
[jerse

Re: [ANNOUNCE] New Committer: Jyothsna Donapati

[Khurram Faraaz]

Congratulations Jyothsna!

On Thu, May 9, 2019 at 2:38 PM salim achouche  wrote:

> Congratulations Jyothsna!
>
> On Thu, May 9, 2019 at 2:28 PM Aman Sinha  wrote:
>
> > The Project Management Committee (PMC) for Apache Drill has invited
> > Jyothsna
> > Donapati to become a committer, and we are pleased to announce that she
> has
> > accepted.
> >
> > Jyothsna has been contributing to Drill for about 1 1/2 years.  She
> > initially contributed the graceful shutdown capability and more recently
> > has made several crucial improvements in the parquet metadata caching
> which
> > have gone into the 1.16 release.  She also co-authored the design
> document
> > for this feature.
> >
> > Welcome Jyothsna, and thank you for your contributions.  Keep up the good
> > work
> > !
> >
> > -Aman
> > (on behalf of Drill PMC)
> >
>
>
> --
> Regards,
> Salim
>

Re: [VOTE] Apache Drill Release 1.16.0 - RC1

[Khurram Faraaz]

i see the correct version and commit, I deployed the binaries to test.

Apache Drill 1.16.0
"Start your SQL engine."
apache drill> select * from sys.version;
+-+--+-+---+---+---+
| version |commit_id |
 commit_message|commit_time|
build_email|build_time |
+-+--+-+---+---+---+
| 1.16.0  | cf5b758e0a4c22b75bfb02ac2653ff09415ddf53 |
[maven-release-plugin] prepare release drill-1.16.0 | 22.04.2019 @ 09:08:36
PDT | sor...@apache.org | 22.04.2019 @ 09:53:25 PDT |
+-+--+-+---+---+---+
1 row selected (0.274 seconds)
apache drill>

Thanks,
Khurram

On Wed, Apr 24, 2019 at 9:52 AM SorabhApache  wrote:

> Hi Volodymyr/Anton,
> I can verify that I am seeing both the below issues as reported by Anton
> and Volodymyr. I will investigate further why we are seeing these issues.
> Thanks for catching this. Can you please open JIRA's for these issues ?
>
> 1) Wrong result for sys.version query when built from source tarball.
> 2) git.properties file in drill-format-mapr-1.16.0.jar has wrong commit id
> but as Volodymyr mentioned because of order in which jars are picked up
> it's not showing the issue when tried from prebuilt tarball.
>
> @Volodymyr Vysotskyi 
> Regarding the GPG key I am not sure if we mandate it to use apache.org,
> there other keys in the file which are using gmail address as well. As far
> as the signing person is authenticating the key and details associated with
> it, I think it should be fine. But since it's recommended I will use
> apache.org email address instead.
>
> Thanks,
> Sorabh
>
> On Wed, Apr 24, 2019 at 8:53 AM Volodymyr Vysotskyi 
> wrote:
>
> > Hi Aman,
> >
> > There are two different issues connected with *git.properties* file.
> > Regarding the problem I have mentioned, prebuilt tar
> > (apache-drill-1.16.0.tar.gz) contains *drill-format-mapr-1.16.0.jar* jar
> > which contains a *git.properties* file with the incorrect version.
> > When *select * from sys.version* query is submitted, class loader finds
> the
> > first file named as *git.properties* from the classpath (each drill jar
> > contains its own *git.properties* file) and for my case file from
> > *drill-format-mapr-1.16.0.jar *is picked up, so the incorrect result is
> > returned. But it may not be reproducible for other machines since it
> > depends on the order of files for the class loader.
> >
> > Regarding the problem Anton has mentioned, Drill should be built from the
> > sources (apache-drill-1.16.0-src.tar.gz), and for that version, *select *
> > from sys.version* returns the result without information about commit.
> >
> > Kind regards,
> > Volodymyr Vysotskyi
> >
> >
> > On Wed, Apr 24, 2019 at 6:33 PM Aman Sinha  wrote:
> >
> > > This works fine for me with the binary tarball that I installed on my
> > Mac.
> > > ..it shows the correct commit message.
> > >
> > > Apache Drill 1.16.0
> > >
> > > "This isn't your grandfather's SQL."
> > >
> > > apache drill> *select* * *from* sys.version;
> > >
> > >
> >
> +-+--+-+---+---+---+
> > > | version |commit_id |
> > >  commit_message|commit_time|
> > > build_email|build_time |
> > >
> > >
> >
> +-+--+-+---+---+---+
> > > | 1.16.0  | cf5b758e0a4c22b75bfb02ac2653ff09415ddf53 |
> > > [maven-release-plugin] prepare release drill-1.16.0 | 22.04.2019 @
> > 09:08:36
> > > PDT | sor...@apache.org | 22.04.2019 @ 09:54:09 PDT |
> > >
> > >
> >
> +-+--+-+---+---+---+
> > >
> > > I don't see any extraneous git.properties anywhere in the source
> > > distribution that I downloaded:
> > >
> > > [root@aman1 apache-drill-1.16.0-src]# find . -name "git.properties"
> > >
> > >
> > >
> >
> ./distribution/target/apache-drill-1.16.0/apache-drill-1.16.0/git.properties
> > >
> > > ./git.properties
> > >
> > >
> > >
> > > On Wed, Apr 24, 2019 at 4:51 AM Arina Ielchiieva 
> > wrote:
> > >
> > > > Taking into account previous emails, looks like we'll need to have
> 

reg Apache Drill Client for NodeJS

[Khurram Faraaz]

Hi Charles,

Can you please update the usage documentation for Apache Drill Client for
NodeJS, do we need the Drill JDBC driver in there ?
I tried the following,

Node version is
[root@test-01 drill-1.15.0]# node --version
v6.16.0

[root@test-01 drill-1.15.0]# cat /etc/centos-release
CentOS Linux release 7.3.1611 (Core)

yum install npm
npm install drill-client

On the page available on below URL
Apache Drill Client for NodeJS
https://github.com/cgivre/drill-nodejs

Usage
To query Drill, the .execute() or .query() functions

var client = new Drill.Client({hostname:'localhost', port:8047});

client.query('SELECT * FROM cp.`employee.json` LIMIT 10', function(error,
data, columns){
  console.log({data: data['rows'], columns: data['columns'], error: error});
});

I hit this error when I tried the above example on my machine which is
running drill-1.15.0

[root@test-01 ~]# node test-node-drill.js
/root/test-node-drill.js:1
(function (exports, require, module, __filename, __dirname) { var client =
new Drill.Client({hostname:'localhost', port:8047});

   ^

ReferenceError: Drill is not defined
at Object. (/root/test-node-drill.js:1:80)
at Module._compile (module.js:577:32)
at Object.Module._extensions..js (module.js:586:10)
at Module.load (module.js:494:32)
at tryModuleLoad (module.js:453:12)
at Function.Module._load (module.js:445:3)
at Module.runMain (module.js:611:10)
at run (bootstrap_node.js:394:7)
at startup (bootstrap_node.js:160:9)
at bootstrap_node.js:507:3

Thanks,
Khurram

[jira] [Created] (DRILL-7163) Join query fails with java.lang.IllegalArgumentException: null

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-7163:
-

 Summary: Join query fails with java.lang.IllegalArgumentException: 
null
 Key: DRILL-7163
 URL: https://issues.apache.org/jira/browse/DRILL-7163
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.15.0
Reporter: Khurram Faraaz


Join query fails with java.lang.IllegalArgumentException: null

Drill : 1.15.0

Failing query is

{noformat}
Select * 
From 
( 
select 
convert_from(t.itm.iUUID, 'UTF8') iUUID, 
convert_from(t.UPC.UPC14, 'UTF8') UPC14, 
convert_from(t.itm.upcDesc, 'UTF8') upcDesc, 
convert_from(t.ris.mstBrdOid, 'UTF8') mstBrdOid, 
convert_from(t.ris.vrfLgyMtch, 'UTF8') vrfLgyMtch, 
convert_from(t.itm.mtch.cfdMtch, 'UTF8') cfdMtch, 
convert_from(t.itm.uoM, 'UTF8') uoM, 
convert_from(t.uomRec.valVal, 'UTF8') uomVal, 
case when a.iUUID is null then 0 else 1 end as keyind 
from hbase.`/mapr/tables/item-master` t 
left outer join 
( 
select distinct 
convert_from(t.m.iUUID, 'UTF8') iUUID 
from hbase.`/mapr/tables/items` t 
) a 
on t.itm.iUUID = a.iUUID 
) i 
where (i.mstBrdOid is null 
or i.vrfLgyMtch is null) 
and i.keyind=1 
{noformat}

Stack trace from drillbit.log
{noformat}
2019-03-27 11:45:44,563 [23646564-3d23-f32b-6f68-11d7c4dd7a19:frag:1:0] ERROR 
o.a.d.e.physical.impl.BaseRootExec - Batch dump started: dumping last 2 failed 
batches
2019-03-27 11:45:44,564 [23646564-3d23-f32b-6f68-11d7c4dd7a19:frag:1:0] ERROR 
o.a.d.e.p.i.p.ProjectRecordBatch - 
ProjectRecordBatch[projector=Projector[vector2=null, selectionVectorMode=NONE], 
hasRemainder=false, remainderIndex=0, recordCount=0, 
container=org.apache.drill.exec.record.VectorContainer@2133fd0e[recordCount = 
0, schemaChanged = false, schema = BatchSchema [fields=[[`row_key` 
(VARBINARY:REQUIRED)], [`clnDesc` (MAP:REQUIRED), children=([`bndlCnt` 
(VARBINARY:OPTIONAL)], [`by` (VARBINARY:OPTIONAL)], [`desc` 
(VARBINARY:OPTIONAL)], [`dt` (VARBINARY:OPTIONAL)], [`descExt` 
(VARBINARY:OPTIONAL)])], [`dup` (MAP:REQUIRED), children=([`dupBy` 
(VARBINARY:OPTIONAL)], [`dupDt` (VARBINARY:OPTIONAL)], [`duplicate` 
(VARBINARY:OPTIONAL)], [`preferred` (VARBINARY:OPTIONAL)])], [`itm` 
(MAP:REQUIRED), children=([`iUUID` (VARBINARY:OPTIONAL)], [`cfdLgyMtch` 
(VARBINARY:OPTIONAL)], [`uoM` (VARBINARY:OPTIONAL)], [`upcCd` 
(VARBINARY:OPTIONAL)], [`upcDesc` (VARBINARY:OPTIONAL)], [`promo` 
(VARBINARY:OPTIONAL)])], [`lckSts` (MAP:REQUIRED), children=([`lckBy` 
(VARBINARY:OPTIONAL)], [`lckDt` (VARBINARY:OPTIONAL)])], [`lgy` (MAP:REQUIRED), 
children=([`lgyBr` (VARBINARY:OPTIONAL)])], [`obs` (MAP:REQUIRED), 
children=([`POSFile` (VARBINARY:OPTIONAL)])], [`prmRec` (MAP:REQUIRED)], [`ris` 
(MAP:REQUIRED), children=([`UPC` (VARBINARY:OPTIONAL)], [`brdDesc` 
(VARBINARY:OPTIONAL)], [`brdExtDesc` (VARBINARY:OPTIONAL)], [`brdFamDesc` 
(VARBINARY:OPTIONAL)], [`brdTypeCd` (VARBINARY:OPTIONAL)], [`flvDesc` 
(VARBINARY:OPTIONAL)], [`mfgDesc` (VARBINARY:OPTIONAL)], [`modBy` 
(VARBINARY:OPTIONAL)], [`modDt` (VARBINARY:OPTIONAL)], [`msaCatCd` 
(VARBINARY:OPTIONAL)])], [`rjr` (MAP:REQUIRED)], [`uomRec` (MAP:REQUIRED), 
children=([`valBy` (VARBINARY:OPTIONAL)], [`valDt` (VARBINARY:OPTIONAL)], 
[`valVal` (VARBINARY:OPTIONAL)], [`recBy` (VARBINARY:OPTIONAL)], [`recDt` 
(VARBINARY:OPTIONAL)], [`recRat` (VARBINARY:OPTIONAL)], [`recVal` 
(VARBINARY:OPTIONAL)])], [`upc` (MAP:REQUIRED), children=([`UPC14` 
(VARBINARY:OPTIONAL)], [`allUPCVar` (VARBINARY:OPTIONAL)])], [`$f12` 
(VARBINARY:OPTIONAL)], [`iUUID` (VARCHAR:OPTIONAL)]], selectionVector=NONE], 
wrappers = [org.apache.drill.exec.vector.VarBinaryVector@b23a384[field = 
[`row_key` (VARBINARY:REQUIRED)], ...], 
org.apache.drill.exec.vector.complex.MapVector@61c779ff, 
org.apache.drill.exec.vector.complex.MapVector@575c0f96, 
org.apache.drill.exec.vector.complex.MapVector@69b943fe, 
org.apache.drill.exec.vector.complex.MapVector@7f90e2ce, 
org.apache.drill.exec.vector.complex.MapVector@25c27442, 
org.apache.drill.exec.vector.complex.MapVector@12d5ffd3, 
org.apache.drill.exec.vector.complex.MapVector@3150f8c4, 
org.apache.drill.exec.vector.complex.MapVector@49aefab2, 
org.apache.drill.exec.vector.complex.MapVector@7f78e7a1, 
org.apache.drill.exec.vector.complex.MapVector@426ea4fa, 
org.apache.drill.exec.vector.complex.MapVector@74cee2ab, 
org.apache.drill.exec.vector.NullableVarBinaryVector@4a0bfdea[field = [`$f12` 
(VARBINARY:OPTIONAL)], ...], 
org.apache.drill.exec.vector.NullableVarCharVector@72f64ee5[field = [`iUUID` 
(VARCHAR:OPTIONAL)], ...]], ...]]
2019-03-27 11:45:44,565 [23646564-3d23-f32b-6f68-11d7c4dd7a19:frag:1:0] ERROR 
o.a.d.e.p.impl.join.HashJoinBatch - 
HashJoinBatch[container=org.apache.drill.exec.record.VectorContainer@45887d35[recordCount
 = 0, schemaChanged = false, schema = BatchSchema [fields=[[`row_key` 
(VARBINARY:REQUIRED)], [`clnDesc` (MAP:REQUIRED), children=([`bndlCnt` 
(VARBINARY:OPTIONAL)], [`by` (VARBINARY:OPTIONAL)], [`desc` 
(VARBINARY:OPTIONAL

[jira] [Created] (DRILL-7144) sqlline option : !set useLineContinuation false, fails with ParseException

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-7144:
-

 Summary: sqlline option : !set useLineContinuation false, fails 
with ParseException
 Key: DRILL-7144
 URL: https://issues.apache.org/jira/browse/DRILL-7144
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.15.0, 1.13.0
Reporter: Khurram Faraaz
Assignee: Arina Ielchiieva


sqlline option does not work as intended. Returns ParseException instead.
!set useLineContinuation false

On mapr-drill-1.13.0 we hit the below Exception.

{noformat}
0: jdbc:drill:drillbit=drill-abcd-dev.dev.schw> !set useLineContinuation false
Error setting configuration: useLineContinuation: 
java.lang.IllegalArgumentException: No method matching "setuseLineContinuation" 
was found in sqlline.SqlLineOpts.
{noformat}

It does not work on drill-1.15.0-mapr-r1

git.branch=drill-1.15.0-mapr-r1
git.commit.id=ebc9fe49d4477b04701fdd81884d5a0b748a13ae

{noformat}
[test@test-ab bin]# ./sqlline -u 
"jdbc:drill:schema=dfs.tmp;auth=MAPRSASL;drillbit=test-ab.qa.lab" -n mapr -p 
mapr
Apache Drill 1.15.0.3-mapr
"Start your SQL engine."
0: jdbc:drill:schema=dfs.tmp> !set useLineContinuation false
0: jdbc:drill:schema=dfs.tmp> select * from sys.version
> select * from sys.memory
Error: PARSE ERROR: Encountered "select" at line 2, column 1.
Was expecting one of:
 
 "ORDER" ...
 "LIMIT" ...
 "OFFSET" ...
 "FETCH" ...
 "NATURAL" ...
 "JOIN" ...
 "INNER" ...
 "LEFT" ...
 "RIGHT" ...
 "FULL" ...
 "CROSS" ...
 "," ...
 "OUTER" ...
 "EXTEND" ...
 "(" ...
 "MATCH_RECOGNIZE" ...
 "AS" ...
  ...
  ...
  ...
  ...
  ...
 "TABLESAMPLE" ...
 "WHERE" ...
 "GROUP" ...
 "HAVING" ...
 "WINDOW" ...
 "UNION" ...
 "INTERSECT" ...
 "EXCEPT" ...
 "MINUS" ...
 "." ...
 "[" ...


SQL Query select * from sys.version
select * from sys.memory
^

[Error Id: 067d5402-b965-4660-8981-34491ab5a051 on test-ab.qa.lab:31010] 
(state=,code=0)
{noformat}


{noformat}
[Error Id: 067d5402-b965-4660-8981-34491ab5a051 ]
 at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
 ~[drill-common-1.15.0.3-mapr.jar:1.15.0.3-mapr]
 at org.apache.drill.exec.planner.sql.SqlConverter.parse(SqlConverter.java:185) 
[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
 at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:138)
 [drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
 at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:110)
 [drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
 at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:76)
 [drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
 at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:584) 
[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
 at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:272) 
[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[na:1.8.0_151]
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[na:1.8.0_151]
 at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
Caused by: org.apache.calcite.sql.parser.SqlParseException: Encountered 
"select" at line 2, column 1.
Was expecting one of:
 
 "ORDER" ...
 "LIMIT" ...
 "OFFSET" ...
 "FETCH" ...
 ...
 "[" ...

at 
org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.convertException(DrillParserImpl.java:350)
 ~[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
 at 
org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.normalizeException(DrillParserImpl.java:131)
 ~[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
 at org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:137) 
~[calcite-core-1.17.0-drill-r2.jar:1.17.0-drill-r2]
 at org.apache.calcite.sql.parser.SqlParser.parseStmt(SqlParser.java:162) 
~[calcite-core-1.17.0-drill-r2.jar:1.17.0-drill-r2]
 at org.apache.drill.exec.planner.sql.SqlConverter.parse(SqlConverter.java:177) 
[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
 ... 8 common frames omitted
Caused by: org.apache.drill.exec.planner.sql.parser.impl.ParseException: 
Encountered "select" at line 2, column 1.
Was expecting one of:
 
 "ORDER" ...
 "LIMIT" ...
 "OFFSET" ...
 "FETCH" ...
 "NATURAL" ...
 ...
 ...
 "[" ...

at 
org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.generateParseException(DrillParserImpl.java:24076)
 ~[drill-java

[jira] [Created] (DRILL-7128) IllegalStateException: Read batch count [0] should be greater than zero

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-7128:
-

 Summary: IllegalStateException: Read batch count [0] should be 
greater than zero
 Key: DRILL-7128
 URL: https://issues.apache.org/jira/browse/DRILL-7128
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - Parquet
Affects Versions: 1.15.0
Reporter: Khurram Faraaz


Source table is a Hive table stored as parquet.
Issue is seen only when querying datacapturekey column, which is of VARCHAR 
type.

Hive 2.3
MapR Drill : 1.15.0.0-mapr 
commit id : 951ef156fb1025677a2ca2dcf84e11002bf4b513

{noformat}
0: jdbc:drill:drillbit=test.a.node1> describe bt_br_cc_invalid_leads ;
+-++--+
| COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
+-++--+
| wrapup | CHARACTER VARYING | YES |
| datacapturekey | CHARACTER VARYING | YES |
| leadgendate | CHARACTER VARYING | YES |
| crla1 | CHARACTER VARYING | YES |
| crla2 | CHARACTER VARYING | YES |
| invalid_lead | INTEGER | YES |
| destination_advertiser_vendor_name | CHARACTER VARYING | YES |
| source_program_key | CHARACTER VARYING | YES |
| publisher_publisher | CHARACTER VARYING | YES |
| areaname | CHARACTER VARYING | YES |
| data_abertura_ficha | CHARACTER VARYING | YES |
+-++--+
11 rows selected (1.85 seconds)
0: jdbc:drill:drillbit=test.a.node1>

// from the view definition, note that column datacapturekey is of type 
VARVCHAR with precision 2000
{
"name" : "bt_br_cc_invalid_leads",
"sql" : "SELECT CAST(`wrapup` AS VARCHAR(2000)) AS `wrapup`, 
CAST(`datacapturekey` AS VARCHAR(2000)) AS `datacapturekey`, CAST(`leadgendate` 
AS VARCHAR(2000)) AS `leadgendate`, CAST(`crla1` AS VARCHAR(2000)) AS `crla1`, 
CAST(`crla2` AS VARCHAR(2000)) AS `crla2`, CAST(`invalid_lead` AS INTEGER) AS 
`invalid_lead`, CAST(`destination_advertiser_vendor_name` AS VARCHAR(2000)) AS 
`destination_advertiser_vendor_name`, CAST(`source_program_key` AS 
VARCHAR(2000)) AS `source_program_key`, CAST(`publisher_publisher` AS 
VARCHAR(2000)) AS `publisher_publisher`, CAST(`areaname` AS VARCHAR(2000)) AS 
`areaname`, CAST(`data_abertura_ficha` AS VARCHAR(2000)) AS 
`data_abertura_ficha`\nFROM 
`dfs`.`root`.`/user/bigtable/logs/hive/warehouse/bt_br_cc_invalid_leads`",
"fields" : [ {
"name" : "wrapup",
"type" : "VARCHAR",
"precision" : 2000,
"isNullable" : true
}, {
"name" : "datacapturekey",
"type" : "VARCHAR",
"precision" : 2000,
"isNullable" : true
...
...

// total number of rows in bt_br_cc_invalid_leads
0: jdbc:drill:drillbit=test.a.node1> select count(*) from 
bt_br_cc_invalid_leads ;
+-+
| EXPR$0 |
+-+
| 20599 |
+-+
1 row selected (0.173 seconds)
{noformat}

Stack trace from drillbit.log
{noformat}
2019-03-18 12:19:01,610 [237010da-6eda-a913-0424-32f63fbe01be:foreman] INFO 
o.a.drill.exec.work.foreman.Foreman - Query text for query with id 
237010da-6eda-a913-0424-32f63fbe01be issued by bigtable: SELECT 
`bt_br_cc_invalid_leads`.`datacapturekey` AS `datacapturekey`
FROM `dfs.drill_views`.`bt_br_cc_invalid_leads` `bt_br_cc_invalid_leads`
GROUP BY `bt_br_cc_invalid_leads`.`datacapturekey`

2019-03-18 12:19:02,495 [237010da-6eda-a913-0424-32f63fbe01be:frag:0:0] INFO 
o.a.d.e.w.fragment.FragmentExecutor - 237010da-6eda-a913-0424-32f63fbe01be:0:0: 
State change requested AWAITING_ALLOCATION --> RUNNING
2019-03-18 12:19:02,495 [237010da-6eda-a913-0424-32f63fbe01be:frag:0:0] INFO 
o.a.d.e.w.f.FragmentStatusReporter - 237010da-6eda-a913-0424-32f63fbe01be:0:0: 
State to report: RUNNING
2019-03-18 12:19:02,502 [237010da-6eda-a913-0424-32f63fbe01be:frag:0:0] INFO 
o.a.d.exec.physical.impl.ScanBatch - User Error Occurred: Error in parquet 
record reader.
Message:
Hadoop path: /user/bigtable/logs/hive/warehouse/bt_br_cc_invalid_leads/08_0
Total records read: 0
Row group index: 0
Records in row group: 1551
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message hive_schema {
 optional binary wrapup (UTF8);
 optional binary datacapturekey (UTF8);
 optional binary leadgendate (UTF8);
 optional binary crla1 (UTF8);
 optional binary crla2 (UTF8);
 optional binary invalid_lead (UTF8);
 optional binary destination_advertiser_vendor_name (UTF8);
 optional binary source_program_key (UTF8);
 optional binary publisher_publisher (UTF8);
 optional binary areaname (UTF8);
 optional binary data_abertura_ficha (UTF8);
}
, metadata: {}}, blocks: [BlockMetaData\{1551, 139906 
[ColumnMetaData{UNCOMPRESSED [wrapup] optional binary wrapup (UTF8) 
[PLAIN_DICTIONARY, RLE, BIT_PACKED], 4}, ColumnMetaData\{UNCOMPRESSED 
[datacapturekey] optional b

Re: [DISCUSS] 1.16.0 release

[Khurram Faraaz]

Can we also have the fix fox DRILL-7061
 in Drill 1.16 ?

Thanks,
Khurram

On Mon, Mar 18, 2019 at 11:40 AM Karthikeyan Manivannan <
kmanivan...@mapr.com> wrote:

> Please include DRILL-7107 <
> https://issues.apache.org/jira/browse/DRILL-7107>
> I will open the PR today.
> It is a small change and fixes a basic usability issue.
>
> Thanks.
>
> Karthik
>
> On Thu, Mar 14, 2019 at 4:50 PM Charles Givre  wrote:
>
> > One more… DRILL-7014 is basically done and I’d like to see that get into
> > Drill 1.16.
> >
> > > On Mar 14, 2019, at 19:44, Charles Givre  wrote:
> > >
> > > Who should I add as a reviewer for 7032?
> > >
> > >> On Mar 14, 2019, at 19:42, Sorabh Hamirwasia  >
> > wrote:
> > >>
> > >> Hi Charles,
> > >> Can you please add reviewer for DRILL-7032 ?
> > >> For DRILL-6970 the PR is closed by the author, I have pinged in JIRA
> > asking
> > >> to re-open so that it can be reviewed.
> > >>
> > >> Thanks,
> > >> Sorabh
> > >>
> > >> On Thu, Mar 14, 2019 at 4:29 PM Charles Givre 
> wrote:
> > >>
> > >>> Hi Sorabh,
> > >>> I have 3 PRs that are almost done, awaiting final review.
> Drill-7077,
> > >>> DRILL-7032, DRILL-7021.  I owe @ariina some fixes for 7077, but I’m
> > waiting
> > >>> for review of the others.  Also, there is that DRILL-6970 about the
> > buffer
> > >>> overflows in the logRegex reader that isn’t mine but I’d like to see
> > >>> included.
> > >>> Thanks,
> > >>> —C
> > >>>
> >  On Mar 14, 2019, at 13:13, Sorabh Hamirwasia <
> sohami.apa...@gmail.com
> > >
> > >>> wrote:
> > 
> >  Hi Arina,
> >  Thanks for your response. With ETA of two weeks we are looking at
> end
> > of
> >  the month or beginning next month. I will wait until Monday for
> > others to
> >  respond and then will finalize on a cut-off date.
> > 
> >  Thanks,
> >  Sorabh
> > 
> >  On Wed, Mar 13, 2019 at 4:28 AM Arina Ielchiieva <
> > >>> arina.yelchiy...@gmail.com>
> >  wrote:
> > 
> > > Hi Sorabh,
> > >
> > > thanks for volunteering to do the release.
> > >
> > > Paul and I are working on file schema provisioning for text file
> > storage
> > > which is aimed for 1.16. To wrap up the work we need to deliver two
> > >>> Jiras:
> > > DRILL-7095 and DRILL-7011. ETA: 2 weeks.
> > > Please plan the release date accordingly.
> > >
> > > Kind regards,
> > > Arina
> > >
> > > On Tue, Mar 12, 2019 at 9:16 PM Sorabh Hamirwasia <
> > >>> sohami.apa...@gmail.com
> > >>
> > > wrote:
> > >
> > >> Hi All,
> > >> It's around two and a half month since we did 1.15.0 release for
> > Apache
> > >> Drill. Based on our 3 months release cadence I think it's time to
> > >>> discuss
> > >> our next release. I will volunteer to manage the next release.
> > >>
> > >> **Below is the current JIRA stats:*
> > >> *[1] Open#: 15*
> > >>
> > >> - Would be great if everyone can look into their assigned tickets
> > and
> > >> update the fix version as needed. Please keep the ones which you
> > find
> > >> must
> > >> have and can be completed sooner.
> > >>
> > >> *[2] InProgress#: 11*
> > >>
> > >> - If you think we *must* include any issues from this list then
> > >>> please
> > >> reply on this thread. Also would be great to know how much time
> you
> > >> think
> > >> is needed for these issues. Based on that we can take a call which
> > >>> one
> > >> to
> > >> target for this release.
> > >>
> > >> *[3] Reviewable#: 14*
> > >>
> > >> - All the reviewers and authors should try to close these as soon
> as
> > >> possible. If you think that any of the PR needs rework or should
> be
> > >> postponed to next release then please update the status and fix
> > > version
> > >> for
> > >> those JIRA's as well.
> > >>
> > >> After above JIRA's are reviewed by everyone and based on their
> > inputs
> > >>> we
> > >> can define a cut off date.
> > >>
> > >> *Approximate numbers as it can change based on JIRA updates.
> > >>
> > >> Thanks,
> > >> Sorabh
> > >>
> > >> [1]:
> > >>
> > >>
> > >
> > >>>
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20DRILL%20AND%20issuetype%20in%20(standardIssueTypes()%2C%20subTaskIssueTypes())%20AND%20fixVersion%20%3D%201.16.0%20AND%20status%20%3D%20Open%20ORDER%20BY%20assignee%20ASC
> > >> [2]:
> > >>
> > >>
> > >
> > >>>
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20DRILL%20AND%20status%20in%20(%22In%20Progress%22)%20AND%20fixVersion%20%3D%201.16.0%20ORDER%20BY%20assignee%20ASC
> > >> [3]:
> > >>
> > >>
> > >
> > >>>
> >
> 

[jira] [Created] (DRILL-7100) parquet RecordBatchSizerManager : IllegalArgumentException: the requested size must be non-negative

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-7100:
-

 Summary: parquet RecordBatchSizerManager : 
IllegalArgumentException: the requested size must be non-negative
 Key: DRILL-7100
 URL: https://issues.apache.org/jira/browse/DRILL-7100
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - Parquet
Affects Versions: 1.15.0
Reporter: Khurram Faraaz


Table has string columns that can range from 1024 bytes to 32MB in length, we 
should be able to handle such wide string columns in parquet, when querying 
from Drill.

Hive Version 2.3.3
Drill Version 1.15

{noformat}
CREATE TABLE temp.cust_bhsf_ce_blob_parquet (
 event_id DECIMAL, 
 valid_until_dt_tm string, 
 blob_seq_num DECIMAL, 
 valid_from_dt_tm string, 
 blob_length DECIMAL, 
 compression_cd DECIMAL, 
 blob_contents string, 
 updt_dt_tm string, 
 updt_id DECIMAL, 
 updt_task DECIMAL, 
 updt_cnt DECIMAL, 
 updt_applctx DECIMAL, 
 last_utc_ts string, 
 ccl_load_dt_tm string, 
 ccl_updt_dt_tm string )
 STORED AS PARQUET;
{noformat}
 
The source table is stored as ORC format.

Failing query.
{noformat}
SELECT event_id, BLOB_CONTENTS FROM hive.temp.cust_bhsf_ce_blob_parquet WHERE 
event_id = 3443236037

2019-03-07 14:40:17,886 [237e8c79-0e9b-45d6-9134-0da95dba462f:frag:1:269] INFO 
o.a.d.exec.physical.impl.ScanBatch - User Error Occurred: the requested size 
must be non-negative (the requested size must be non-negative)
org.apache.drill.common.exceptions.UserException: INTERNAL_ERROR ERROR: the 
requested size must be non-negative
{noformat}

Snippet from drillbit.log file
{noformat}
[Error Id: 41a4d597-f54d-42a6-be6d-5dbeb7f642ba ]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
 ~[drill-common-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:293) 
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:126)
 [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:116)
 [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at 
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
 [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
 [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:126)
 [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:116)
 [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at 
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:69)
 [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
 [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) 
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at 
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93)
 [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) 
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:297)
 [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:284)
 [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_181]
at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_181]
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
 [hadoop-common-2.7.0-mapr-1808.jar:na]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:284)
 [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[na:1.8.0_181]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[na:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: java.lang.IllegalArgumentException: the requested size must be 
non-negative
at 
org.apache.drill.shaded.guava.com.google.common.base.Preconditions.checkArgument(Preconditions.java:135)
 ~[drill-shaded-guava-23.0.jar:23.0]
at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:224) 
~[drill-memory-base-1.15.0.0-mapr.jar

[jira] [Created] (DRILL-7061) Selecting option to limit results to 1000 on web UI causes parse error

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-7061:
-

 Summary: Selecting option to limit results to 1000 on web UI 
causes parse error
 Key: DRILL-7061
 URL: https://issues.apache.org/jira/browse/DRILL-7061
 Project: Apache Drill
  Issue Type: Bug
  Components: Web Server
Affects Versions: 1.16.0
Reporter: Khurram Faraaz
 Attachments: image-2019-02-27-14-17-24-348.png

Selecting option to Limit results to 1,000 causes a parse error on web UI, 
screen shot is attached. Browser used was Chrome.

Drill version => 1.16.0-SNAPSHOT

commit = e342ff5

Error reported on web UI when we press Submit button on web UI
{noformat}
Query Failed: An Error Occurred 
org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR: 'LIMIT 
start, count' is not allowed under the current SQL conformance level SQL Query 
-- [autoLimit: 1,000 rows] select * from ( select length(varStr) from 
dfs.`/root/many_json_files` ) limit 1,000 [Error Id: 
e252d1cc-54d4-4530-837c-a1726a5be89f on qa102-45.qa.lab:31010]{noformat}
 

!image-2019-02-27-14-17-24-348.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-7023) Query fails with IndexOutOfBoundsException after upgrade from drill 1.13.0 to drill 1.14.0

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-7023:
-

 Summary: Query fails with IndexOutOfBoundsException after upgrade 
from drill 1.13.0 to drill 1.14.0
 Key: DRILL-7023
 URL: https://issues.apache.org/jira/browse/DRILL-7023
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz


Query fails with IndexOutOfBoundsException after upgrade from drill 1.13.0 to 
drill 1.14.0

{noformat}
2018-12-06 21:43:00,538 [23f5f79c-3777-eb37-ee46-f73be74381ef:frag:2:1] ERROR 
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IndexOutOfBoundsException

Fragment 2:1

[Error Id: 3b653503-b6da-4853-a395-317a169468ce on am1397.test.net:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
IndexOutOfBoundsException

Fragment 2:1

[Error Id: 3b653503-b6da-4853-a395-317a169468ce on am1397.test.net:31010]
 at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
 ~[drill-common-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361)
 [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216)
 [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327)
 [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.14.0-mapr.jar:1.14.0-mapr]
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[na:1.8.0_152]
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[na:1.8.0_152]
 at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
Caused by: java.lang.IndexOutOfBoundsException: null
 at io.netty.buffer.DrillBuf.getBytes(DrillBuf.java:677) 
~[drill-memory-base-1.14.0-mapr.jar:4.0.48.Final]
 at org.apache.drill.exec.vector.BigIntVector.copyEntry(BigIntVector.java:389) 
~[vector-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.test.generated.HashJoinProbeGen480.appendProbe(HashJoinProbeTemplate.java:190)
 ~[na:na]
 at 
org.apache.drill.exec.test.generated.HashJoinProbeGen480.outputOuterRow(HashJoinProbeTemplate.java:223)
 ~[na:na]
 at 
org.apache.drill.exec.test.generated.HashJoinProbeGen480.executeProbePhase(HashJoinProbeTemplate.java:357)
 ~[na:na]
 at 
org.apache.drill.exec.test.generated.HashJoinProbeGen480.probeAndProject(HashJoinProbeTemplate.java:400)
 ~[na:na]
 at 
org.apache.drill.exec.physical.impl.join.HashJoinBatch.innerNext(HashJoinBatch.java:465)
 ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
 ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
 ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
 ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:69)
 ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
 ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
 ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:142)
 ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
 ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr

[jira] [Created] (DRILL-6994) TIMESTAMP type DOB column in Spark parquet is treated as VARBINARY in Drill

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6994:
-

 Summary: TIMESTAMP type DOB column in Spark parquet is treated as 
VARBINARY in Drill
 Key: DRILL-6994
 URL: https://issues.apache.org/jira/browse/DRILL-6994
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.14.0
Reporter: Khurram Faraaz


A timestamp type column in a parquet file created from Spark is treated as 
VARBINARY by Drill 1.14.0., Trying to cast DOB column to DATE results in an 
Exception, although the monthOfYear field is in the allowed range.

Data used in the test
{noformat}
[test@md123 spark_data]# cat inferSchema_example.csv
Name,Department,years_of_experience,DOB
Sam,Software,5,1990-10-10
Alex,Data Analytics,3,1992-10-10
{noformat}

Create the parquet file using the above CSV file
{noformat}
[test@md123 bin]# ./spark-shell
19/01/22 21:21:34 WARN NativeCodeLoader: Unable to load native-hadoop library 
for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://md123.qa.lab:4040
Spark context available as 'sc' (master = local[*], app id = 
local-1548192099796).
Spark session available as 'spark'.
Welcome to
  __
 / __/__ ___ _/ /__
 _\ \/ _ \/ _ `/ __/ '_/
 /___/ .__/\_,_/_/ /_/\_\ version 2.3.1-mapr-SNAPSHOT
 /_/

Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_191)
Type in expressions to have them evaluated.
Type :help for more information.

scala> import org.apache.spark.sql.\{DataFrame, SQLContext}
import org.apache.spark.sql.\{DataFrame, SQLContext}

scala> import org.apache.spark.\{SparkConf, SparkContext}
import org.apache.spark.\{SparkConf, SparkContext}

scala> val sqlContext: SQLContext = new SQLContext(sc)
warning: there was one deprecation warning; re-run with -deprecation for details
sqlContext: org.apache.spark.sql.SQLContext = 
org.apache.spark.sql.SQLContext@2e0163cb

scala> val df = 
sqlContext.read.format("com.databricks.spark.csv").option("header", 
"true").option("inferSchema", "true").load("/apps/inferSchema_example.csv")
df: org.apache.spark.sql.DataFrame = [Name: string, Department: string ... 2 
more fields]

scala> df.printSchema
test
 |-- Name: string (nullable = true)
 |-- Department: string (nullable = true)
 |-- years_of_experience: integer (nullable = true)
 |-- DOB: timestamp (nullable = true)

scala> df.write.parquet("/apps/infer_schema_example.parquet")

// Read the parquet file
scala> val data = sqlContext.read.parquet("/apps/infer_schema_example.parquet")
data: org.apache.spark.sql.DataFrame = [Name: string, Department: string ... 2 
more fields]

// Print the schema of the parquet file from Spark
scala> data.printSchema
test
 |-- Name: string (nullable = true)
 |-- Department: string (nullable = true)
 |-- years_of_experience: integer (nullable = true)
 |-- DOB: timestamp (nullable = true)

// Display the contents of parquet file on spark-shell
// register temp table and do a show on all records,to display.

scala> data.registerTempTable("employee")
warning: there was one deprecation warning; re-run with -deprecation for details

scala> val allrecords = sqlContext.sql("SELeCT * FROM employee")
allrecords: org.apache.spark.sql.DataFrame = [Name: string, Department: string 
... 2 more fields]

scala> allrecords.show()
++--+---+---+
|Name| Department|years_of_experience| DOB|
++--+---+---+
| Sam| Software| 5|1990-10-10 00:00:00|
|Alex|Data Analytics| 3|1992-10-10 00:00:00|
++--+---+---+
{noformat}

Querying the parquet file from Drill 1.14.0-mapr, results in the DOB column 
(timestamp type in Spark) being treated as VARBINARY.

{noformat}
apache drill 1.14.0-mapr
"a little sql for your nosql"
0: jdbc:drill:schema=dfs.tmp> select * from 
dfs.`/apps/infer_schema_example.parquet`;
+---+-+--+--+
| Name | Department | years_of_experience | DOB |
+---+-+--+--+
| Sam | Software | 5 | [B@2bef51f2 |
| Alex | Data Analytics | 3 | [B@650eab8 |
+---+-+--+--+
2 rows selected (0.229 seconds)

// typeof(DOB) column returns a VARBINARY type, whereas the parquet schema in 
Spark for DOB: timestamp (nullable = true)

0: jdbc:drill:schema=dfs.tmp> select typeof(DOB) from 
dfs.`/apps/infer_schema_example.parquet`;
++
| EXPR$0 |
++
| VARBINARY |
| VARBINARY |
++
2 rows selected (0.199 seconds)
{noformat}

// CAST to DATE type results in Exception, though the monthOfYear is in the 
range [1,12]

{noformat}
0: jdbc:drill:schema=dfs.tmp> select cast(DOB as DATE) from 
d

[jira] [Created] (DRILL-6990) IllegalStateException: The current reader doesn't support getting next information

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6990:
-

 Summary: IllegalStateException: The current reader doesn't support 
getting next information
 Key: DRILL-6990
 URL: https://issues.apache.org/jira/browse/DRILL-6990
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
 Attachments: parqt_nestedArray.parquet.tar

Reading a parquet file created from Spark, returns IllegalStateException: The 
current reader doesn't support getting next information

Drill 1.14.0, parquet file created from Spark is attached here.

//Steps to create parquet file from Spark 2.3.1

[root@ba102-495 ~]# cd /opt/mapr/spark/spark-2.3.1
[root@ba102-495 spark-2.3.1]# cd bin
[root@ba102-495 bin]# ./spark-shell
19/01/21 22:57:05 WARN NativeCodeLoader: Unable to load native-hadoop library 
for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://qa102-45.qa.lab:4040
Spark context available as 'sc' (master = local[*], app id = 
local-1548111430809).
Spark session available as 'spark'.
Welcome to
  __
 / __/__ ___ _/ /__
 _\ \/ _ \/ _ `/ __/ '_/
 /___/ .__/\_,_/_/ /_/\_\ version 2.3.1-mapr-SNAPSHOT
 /_/

Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_191)
Type in expressions to have them evaluated.
Type :help for more information.

scala> import spark.implicits._
import spark.implicits._

scala> val df = spark.read.json("/apps/nestedDataJson.json")
df: org.apache.spark.sql.DataFrame = [id: bigint, nested_array: 
array>]

scala> df.write.parquet("/apps/parqt_nestedArray.parquet")

Data used in test

{noformat}
[root@ba102-495 ~]# cat nestedDataJson.json
{"id":19,"nested_array":[[1,2,3,4],[5,6,7,8],[9,10,12]]}
{"id":14121,"nested_array":[[1,3,4],[5,6,8],[9,11,12]]}
{"id":18894,"nested_array":[[1,3,4],[5,6,7,8],[9,10,11,12]]}
{"id":12499,"nested_array":[[1,4],[5,7,8],[9,11,12]]}
{"id":120,"nested_array":[[1,4],[5,7,8],[9,10,11,12]]}
{"id":12,"nested_array":[[1,2,3,4],[5,6,7,8],[11,12]]}
{"id":13,"nested_array":[[1,2,3,4],[5,8],[9,10,11,12]]}
{"id":14,"nested_array":[[1,2,3,4],[5,68],[9,10,11,12]]}
{"id":123,"nested_array":[[1,2,3,4],[5,8],[9,10,11,12]]}
{"id":124,"nested_array":[[1,2,4],[5,6,7,8],[9,10,11,12]]}
{"id":134,"nested_array":[[1,4],[5,8],[9,12]]}
{noformat}

>From drillbit.log

{noformat}
Query Failed: An Error Occurred
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: 
IllegalStateException: The current reader doesn't support getting next 
information. Fragment 0:0 [Error Id: c16c70dd-6565-463f-83a7-118ccd8442e2 on 
ba102-495.qa.lab:31010]
...
...
2019-01-21 23:08:11,268 [23b9af24-10b9-ad11-5583-ecc3e0c562e6:frag:0:0] ERROR 
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: The 
current reader doesn't support getting next information.

Fragment 0:0

[Error Id: c16c70dd-6565-463f-83a7-118ccd8442e2 on ba102-495.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
IllegalStateException: The current reader doesn't support getting next 
information.

Fragment 0:0

[Error Id: c16c70dd-6565-463f-83a7-118ccd8442e2 on ba102-495.qa.lab:31010]
 at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
 ~[drill-common-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361)
 [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216)
 [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327)
 [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.14.0-mapr.jar:1.14.0-mapr]
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[na:1.8.0_181]
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[na:1.8.0_181]
 at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: java.lang.IllegalStateException: The current reader doesn't support 
getting next information.
 at 
org.apache.drill.exec.vector.complex.impl.AbstractBaseReader.next(AbstractBaseReader.java:64)
 ~[vector-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.vector.complex.impl.SingleMapReaderImpl.next(SingleMapReaderImpl.java:31)
 ~[vector-1.14.0-mapr.jar:1.14.0-mapr]
 at 
org.apache.drill.exec.test.generated.ProjectorGen971.doEval(ProjectorTemplate.java:35)
 ~[na:na]
 at 

[jira] [Created] (DRILL-6979) Add autofocus attribute to username on login page, and to query textbox on Query tab.

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6979:
-

 Summary: Add autofocus attribute to username on login page, and to 
query textbox on Query tab.
 Key: DRILL-6979
 URL: https://issues.apache.org/jira/browse/DRILL-6979
 Project: Apache Drill
  Issue Type: Improvement
  Components: Web Server
Affects Versions: 1.16.0
Reporter: Khurram Faraaz
Assignee: Khurram Faraaz


Add autofocus attribute to username on login page, and to query textbox on 
Query tab.
The two text boxes that need the change are in these files

./exec/java-exec/src/main/resources/rest/query/query.ftl
./exec/java-exec/src/main/resources/rest/login.ftl



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6961) Error Occurred: Cannot connect to the db. query INFORMATION_SCHEMA.VIEWS : Maybe you have incorrect connection params or db unavailable now (timeout)

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6961:
-

 Summary: Error Occurred: Cannot connect to the db. query 
INFORMATION_SCHEMA.VIEWS : Maybe you have incorrect connection params or db 
unavailable now (timeout)
 Key: DRILL-6961
 URL: https://issues.apache.org/jira/browse/DRILL-6961
 Project: Apache Drill
  Issue Type: Improvement
  Components: Storage - Information Schema
Affects Versions: 1.13.0
Reporter: Khurram Faraaz


Trying to query drill information_schema.views table returns error. Disabling 
openTSDB plugin resolves the problem.

Drill 1.13.0

Failing query :
{noformat}
SELECT TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, VIEW_DEFINITION FROM 
INFORMATION_SCHEMA.`VIEWS` where VIEW_DEFINITION not like 'kraken';
{noformat}

Stack Trace from drillbit.log

{noformat}
2019-01-07 15:36:21,975 [23cc39aa-2618-e9f0-e77e-4fafa6edc314:foreman] INFO 
o.a.drill.exec.work.foreman.Foreman - Query text for query id 
23cc39aa-2618-e9f0-e77e-4fafa6edc314: SELECT TABLE_CATALOG, TABLE_SCHEMA, 
TABLE_NAME, VIEW_DEFINITION FROM INFORMATION_SCHEMA.`VIEWS` where 
VIEW_DEFINITION not like 'kraken'
2019-01-07 15:36:35,221 [23cc39aa-2618-e9f0-e77e-4fafa6edc314:frag:0:0] INFO 
o.a.d.e.s.o.c.services.ServiceImpl - User Error Occurred: Cannot connect to the 
db. Maybe you have incorrect connection params or db unavailable now (timeout)
org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Cannot 
connect to the db. Maybe you have incorrect connection params or db unavailable 
now


[Error Id: f8b4c074-ba62-4691-b142-a8ea6e4f6b2a ]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
 ~[drill-common-1.13.0-mapr.jar:1.13.0-mapr]
at 
org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl.getTableNames(ServiceImpl.java:107)
 [drill-opentsdb-storage-1.13.0-mapr.jar:1.13.0-mapr]
at 
org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl.getAllMetricNames(ServiceImpl.java:70)
 [drill-opentsdb-storage-1.13.0-mapr.jar:1.13.0-mapr]
at 
org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory$OpenTSDBSchema.getTableNames(OpenTSDBSchemaFactory.java:78)
 [drill-opentsdb-storage-1.13.0-mapr.jar:1.13.0-mapr]
at 
org.apache.calcite.jdbc.SimpleCalciteSchema.addImplicitTableToBuilder(SimpleCalciteSchema.java:106)
 [calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
at org.apache.calcite.jdbc.CalciteSchema.getTableNames(CalciteSchema.java:318) 
[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
at 
org.apache.calcite.jdbc.CalciteSchema$SchemaPlusImpl.getTableNames(CalciteSchema.java:587)
 [calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
at 
org.apache.calcite.jdbc.CalciteSchema$SchemaPlusImpl.getTableNames(CalciteSchema.java:548)
 [calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
at 
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.visitTables(InfoSchemaRecordGenerator.java:227)
 [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at 
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:216)
 [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at 
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:209)
 [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at 
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:196)
 [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at 
org.apache.drill.exec.store.ischema.InfoSchemaTableType.getRecordReader(InfoSchemaTableType.java:58)
 [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at 
org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:34)
 [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at 
org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:30)
 [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at org.apache.drill.exec.physical.impl.ImplCreator$2.run(ImplCreator.java:146) 
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at org.apache.drill.exec.physical.impl.ImplCreator$2.run(ImplCreator.java:142) 
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_144]
at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_144]
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1633)
 [hadoop-common-2.7.0-mapr-1710.jar:na]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:142)
 [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:182)
 [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:137)
 [drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:182

[jira] [Created] (DRILL-6942) Provide ability to sort list of profiles on Drill web UI

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6942:
-

 Summary: Provide ability to sort list of profiles on Drill web UI
 Key: DRILL-6942
 URL: https://issues.apache.org/jira/browse/DRILL-6942
 Project: Apache Drill
  Issue Type: Improvement
  Components: Web Server
Affects Versions: 1.15.0
Reporter: Khurram Faraaz


We need to provide a way to sort the query profiles, on the profiles tab on 
Drill web UI.

The use case is when users run many queries (several hundred queries), they 
want a way to list the queries that have taken the longest time (i.e. duration) 
to complete. All queries that completed, failed or were canceled and took very 
long, all such queries should be listed on the top in the profiles tab page on 
Drill web ui.

An option to list the query profiles based on total duration should be provided 
to user. That way user can easily identify such long running queries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6937) sys.functions table needs a fix in the names column

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6937:
-

 Summary: sys.functions table needs a fix in the names column
 Key: DRILL-6937
 URL: https://issues.apache.org/jira/browse/DRILL-6937
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.15.0
Reporter: Khurram Faraaz


The function names in the name column of sys.functions in some cases, are the 
operators, this is not the expected behavior, the name column should have 
actual names and not the operators.

I am on Drill 1.15.0 commit : 8743e8f1e8d5bca4d67c94d07a8560ad356ff2b6

{noformat}
Apache Drill 1.15.0
"Data is the new oil. Ready to Drill some?"
0: jdbc:drill:schema=dfs.tmp> select count(*) from sys.functions;
+-+
| EXPR$0 |
+-+
| 2846 |
+-+
1 row selected (0.327 seconds)
0: jdbc:drill:schema=dfs.tmp>
{noformat}

{noformat}
0: jdbc:drill:schema=dfs.tmp> select distinct name from sys.functions limit 12;
++
| name |
++
| != |
| $sum0 |
| && |
| - |
| /int |
| < |
| <= |
| <> |
| = |
| == |
| > |
| >= |
++
12 rows selected (0.175 seconds)
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Apache Drill release 1.15.0 - RC2

[Khurram Faraaz]

Thanks Kunal, I will report a JIRA for that.

Regards,
Khurram

On Fri, Dec 28, 2018 at 9:59 AM Kunal Khatua  wrote:

> The functions issue that Khurram reported is probably showing up due to
> some new functions that might have been added. These might be internal
> functions that dont seem to have been registered as such.
>
> Khurram, are the number of functions (select count * ...)  different from
> 1.14?
> Please file a JIRA so that we can investigate this.
> On 12/28/2018 6:05:38 AM, Vitalii Diravka  wrote:
> Karthikeyan,
> It looks like an env issue. I have sent you possible solution for that
> failure.
>
> Khurram,
> Not sure about your question. It is a new feature.
> But you can send the separate mail to discuss whether it should be
> improved or not in Drill 1.16.0.
>
> Kind regards
> Vitalii
>
>
> On Fri, Dec 28, 2018 at 6:34 AM Hanumath Rao Maduri
> wrote:
>
> > - Downloaded tarball and also built from source from [3]
> > - Tried on my Mac
> > - Ran unit tests.
> >
> > LGTM (+1)
> >
> >
> > On Thu, Dec 27, 2018 at 4:45 PM Khurram Faraaz wrote:
> >
> > > Downloaded binaries and deployed on a 4 node CentOS 7.5 cluster.
> > > Executed basic SQL queries
> > > - from sqlline
> > > - from web UI
> > > - and from POSTMAN
> > >
> > > Verified Web UI, performed sanity tests.
> > >
> > > Looks good.
> > > Here is one question related to querying the new sys.functions system
> > > table.
> > > The function names in the name column of sys.functions table in some
> > cases,
> > > are the operators, is this expected behavior, or should that column
> have
> > > actual names and not the operators.
> > >
> > > 0: jdbc:drill:schema=dfs.tmp> select distinct name from sys.functions
> > limit
> > > 12;
> > > ++
> > > | name |
> > > ++
> > > | != |
> > > | $sum0 |
> > > | && |
> > > | - |
> > > | /int |
> > > | < |="">
> > > | <= |="">
> > > | <> |
> > > | = |
> > > | == |
> > > | > |
> > > | >= |
> > > ++
> > > 12 rows selected (0.175 seconds)
> > >
> > > On Thu, Dec 27, 2018 at 3:02 PM Kunal Khatua wrote:
> > >
> > > > - Downloaded tarball and also built from source
> > > > - Tried on CentOS 7.5 against MapR profile
> > > > - Ran a couple of queries consisting of TPCH dataset in Parquet
> format
> > > > - WebUX interactions seem clean and without any apparent issue.
> > > >
> > > > +1 (binding)
> > > >
> > > > Thanks
> > > > Kunal
> > > > On 12/27/2018 2:37:05 PM, Boaz Ben-Zvi wrote:
> > > > -- Verified gpg signature on source and binaries.
> > > >
> > > > -- Checked the checksum sha512 - matched.
> > > >
> > > > -- Downloaded source to Linux VM - full build and unit tests passed.
> > > >
> > > > -- On the Mac - Build and unit tests passed, except the
> > > > `drill_derby_test` in the `contrib/storage-jdbc` which also fails for
> > > > 1.14.0 on my Mac (so it is a local environment issue).
> > > >
> > > > -- Manually ran on both Mac and Linux, and checked the Web-UI: All my
> > > > `semijoin` tests, and memory spilling tests for hash-join and
> > hash-aggr.
> > > > And a select number of large queries. All passed OK.
> > > >
> > > > ==> +1 (binding)
> > > >
> > > > Thanks,
> > > >
> > > > Boaz
> > > >
> > > > On 12/27/18 12:54 PM, Abhishek Girish wrote:
> > > > > +1
> > > > >
> > > > > - Brought up Drill in distributed mode on a 4 node cluster with
> MapR
> > > > > platform - looks good!
> > > > > - Ran regression tests from [6] - looks good!
> > > > > - Ran unit tests with default & mapr profile - looks good!
> > > > > - Basic sanity tests on Sqlline, Web UI - looks good!
> > > > >
> > > > > [6]
> > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mapr_drill-2Dtest-2Dframework=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=PqKay2uOMZUqopDRKNfBtZSlsp2meGOxWNAVHxHnXCk=7tE7GD3UydzyDZaH_H0xw7V_m-XWe0tj8frqvjH2h7w=Q8PqbATc4VPUWvGcy_V_7iSQu9uyi1iCqLV5v1Mg31k=
> > > > >
> > > > > On Thu

Re: [VOTE] Apache Drill release 1.15.0 - RC2

[Khurram Faraaz]

Downloaded binaries and deployed on a 4 node CentOS 7.5 cluster.
Executed basic SQL queries
- from sqlline
- from web UI
- and from POSTMAN

Verified Web UI, performed sanity tests.

Looks good.
Here is one question related to querying the new sys.functions system table.
The function names in the name column of sys.functions table in some cases,
are the operators, is this expected behavior, or should that column have
actual names and not the operators.

0: jdbc:drill:schema=dfs.tmp> select distinct name from sys.functions limit
12;
++
|  name  |
++
| != |
| $sum0  |
| && |
| -  |
| /int   |
| <  |
| <= |
| <> |
| =  |
| == |
| >  |
| >= |
++
12 rows selected (0.175 seconds)

On Thu, Dec 27, 2018 at 3:02 PM Kunal Khatua  wrote:

> - Downloaded tarball and also built from source
> - Tried on CentOS 7.5 against MapR profile
> - Ran a couple of queries consisting of TPCH dataset in Parquet format
> - WebUX interactions seem clean and without any apparent issue.
>
> +1 (binding)
>
> Thanks
> Kunal
> On 12/27/2018 2:37:05 PM, Boaz Ben-Zvi  wrote:
> -- Verified gpg signature on source and binaries.
>
> -- Checked the checksum sha512 - matched.
>
> -- Downloaded source to Linux VM - full build and unit tests passed.
>
> -- On the Mac - Build and unit tests passed, except the
> `drill_derby_test` in the `contrib/storage-jdbc` which also fails for
> 1.14.0 on my Mac (so it is a local environment issue).
>
> -- Manually ran on both Mac and Linux, and checked the Web-UI: All my
> `semijoin` tests, and memory spilling tests for hash-join and hash-aggr.
> And a select number of large queries. All passed OK.
>
> ==> +1 (binding)
>
> Thanks,
>
> Boaz
>
> On 12/27/18 12:54 PM, Abhishek Girish wrote:
> > +1
> >
> > - Brought up Drill in distributed mode on a 4 node cluster with MapR
> > platform - looks good!
> > - Ran regression tests from [6] - looks good!
> > - Ran unit tests with default & mapr profile - looks good!
> > - Basic sanity tests on Sqlline, Web UI - looks good!
> >
> > [6]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mapr_drill-2Dtest-2Dframework=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=PqKay2uOMZUqopDRKNfBtZSlsp2meGOxWNAVHxHnXCk=7tE7GD3UydzyDZaH_H0xw7V_m-XWe0tj8frqvjH2h7w=Q8PqbATc4VPUWvGcy_V_7iSQu9uyi1iCqLV5v1Mg31k=
> >
> > On Thu, Dec 27, 2018 at 11:12 AM Aman Sinha wrote:
> >
> >> - Downloaded source from [3] onto my Linux VM, built and ran unit
> tests. I
> >> had to run some test suites individually but got a clean run.
> >> - Verified extraneous directory issue (DRILL-6916) is resolved
> >> - Built the source using MapR profile and ran the secondary indexing
> tests
> >> within mapr format plugin
> >> - Downloaded binary tar ball from [3] on my Mac. Verified checksum of
> the
> >> file using shasum -a 512 *file *and comparing with the one on [3]
> >> - Verified Vitalii's signature through the following command: gpg
> --verify
> >> Downloads/apache-drill-1.15.0.tar.gz.asc apache-drill-1.15.0.tar.gz
> >> - Ran Drill in embedded mode and ran a few TPC-H queries. Checked query
> >> profiles through Web UI
> >>
> >> LGTM. +1
> >>
> >> Aman
> >>
> >> On Thu, Dec 27, 2018 at 6:17 AM Denys Ordynskiy
> >> wrote:
> >>
> >>> - downloaded source code, successfully built Drill with mapr profile;
> >>> - run Drill in distributed mode on Ubuntu on JDK8;
> >>> - connected from Drill Explorer, explored data on S3 and MapRFS
> storage;
> >>> - submitted some tests for Drill Web UI and Drill Rest API.
> >>>
> >>> +1
> >>>
> >>> On Wed, Dec 26, 2018 at 8:40 PM Arina Ielchiieva
> >> wrote:
>  Build from source on Linux, started in embedded mode, ran random
> >>> queries.
>  Downloaded tarball on Windows, started Drill in embedded mode, run
> >> random
>  queries.
>  Check Web UI: Profiles, Options, Plugins sections.
> 
>  Additionally checked:
>  - information_schema files table;
>  - new SqlLine version;
>  - JDBC using Squirrel;
>  - ODBC using Drill Explorer;
>  - return result set option.
> 
>  +1 (binding)
> 
>  Kind regards,
>  Arina
> 
>  On Wed, Dec 26, 2018 at 8:32 PM Volodymyr Vysotskyi
> >>> volody...@apache.org>
>  wrote:
> 
> > - Downloaded built tar, checked signatures and hashes for built and
>  source
> > tars
> > and for jars;
> > - run Drill in embedded mode on both Ubuntu and Windows on JDK8 and
>  JDK11;
> > - created views, submitted random TPCH queries from UI and SqlLine,
>  checked
> > that profiles are displayed correctly;
> > - downloaded source tar, ran unit tests and all tests are passed;
> > - built with mapr profile, started in distributed mode, submitted
> >>> several
> > tests for hive tables, checked logs, no errors are found;
> > - connected from SQuirrel, ran several queries, tested
> > exec.query.return_result_set_for_ddl
> > option;
> > - checked metadata 

Re: [ANNOUNCE] New Committer: Salim Achouche

[Khurram Faraaz]

Congratulations Salim!!

On Mon, Dec 17, 2018 at 9:44 AM Hanumanth Maduri  wrote:

> Congratulations Salim !
>
> > On Dec 17, 2018, at 9:38 AM, Timothy Farkas  wrote:
> >
> > Congrats!
> >
> >> On Mon, Dec 17, 2018 at 9:37 AM Aman Sinha  wrote:
> >>
> >> Congratulations Salim !  Thanks for your contributions !
> >>
> >> Aman
> >>
> >> On Mon, Dec 17, 2018 at 3:20 AM Vitalii Diravka 
> >> wrote:
> >>
> >>> Congratulations Salim!
> >>> Well deserved!
> >>>
> >>> Kind regards
> >>> Vitalii
> >>>
> >>>
> >>> On Mon, Dec 17, 2018 at 12:40 PM Arina Ielchiieva 
> >>> wrote:
> >>>
>  The Project Management Committee (PMC) for Apache Drill has invited
> >> Salim
>  Achouche to become a committer, and we are pleased to announce that he
> >>> has
>  accepted.
> 
>  Salim Achouche [1] started contributing to the Drill project in 2017.
> >> He
>  has made many improvements for the parquet reader, including
> >> performance
>  for flat data types, columnar parquet batch sizing functionality,
> fixed
>  various bugs and memory leaks. He also optimized implicit columns
> >>> handling
>  with scanner and improved sql pattern contains performance.
> 
>  Welcome Salim, and thank you for your contributions!
> 
>  - Arina
>  (on behalf of Drill PMC)
> 
> >>>
> >>
>

Re: [ANNOUNCE] New Committer: Karthikeyan Manivannan

[Khurram Faraaz]

Congratulations Karthik!!

On Fri, Dec 7, 2018 at 11:12 AM Abhishek Girish  wrote:

> Congratulations Karthik!
>
> On Fri, Dec 7, 2018 at 11:11 AM Arina Ielchiieva  wrote:
>
> > The Project Management Committee (PMC) for Apache Drill has invited
> > Karthikeyan
> > Manivannan to become a committer, and we are pleased to announce that he
> > has accepted.
> >
> > Karthik started contributing to the Drill project in 2016. He has
> > implemented changes in various Drill areas, including batch sizing,
> > security, code-gen, C++ part. One of his latest improvements is  ACL
> > support for Drill ZK nodes.
> >
> > Welcome Karthik, and thank you for your contributions!
> >
> > - Arina
> > (on behalf of Drill PMC)
> >
>

Re: [DISCUSS] 1.15.0 release

[Khurram Faraaz]

Hi Vitalii

We should investigate and fix this issue.
https://issues.apache.org/jira/browse/DRILL-6816

Thanks,
Khurram

On Tue, Nov 6, 2018 at 9:46 AM Vitalii Diravka  wrote:

> Hi Drillers,
>
> It's been 3 months since the last release and it is time to do the next
> one.
>
> I'll volunteer to manage the release :)
>
> There are 32 open tickets that are still intended to be included in 1.15.0
> release [1].
> What do you guys think which tickets do we want to include and what time
> will it take?
> If there are any other issues on which work is in progress, that you feel
> we *must* include in the release, please post in reply to this thread.
>
> Based on your input we'll define release cut off date.
>
> [1]
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_issues_-3Fjql-3Dproject-2520-253D-2520DRILL-2520AND-2520status-2520in-2520-28Open-252C-2520-2522In-2520Progress-2522-252C-2520Reopened-252C-2520Reviewable-252C-2520Accepted-29-2520AND-2520fixVersion-2520-253D-25201.15.0-2520AND-2520-28component-2520-21-253D-2520Documentation-2520OR-2520component-2520is-2520null-29-2520-2520AND-2520-28labels-2520-21-253D-2520ready-2Dto-2Dcommit-2520OR-2520labels-2520is-2520null-29-2520ORDER-2520BY-2520status-2520DESC-252C-2520updated-2520DESC=DwIFaQ=cskdkSMqhcnjZxdQVpwTXg=H5JEl9vb-mBIjic10QAbDD2vkUUKAxjO6wZO322RtdI=Y3YHmNkTAlAyrEa41_zIzzO0Zar0B7i9XwXs2aBEIKc=nZrfAS48g1mSos0XSURcgZ2Btz1TfV0GQLj8Wpob-Do=
>
> Kind regards
> Vitalii
>

[jira] [Created] (DRILL-6816) NPE - Concurrent query execution using PreparedStatement

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6816:
-

 Summary: NPE - Concurrent query execution using PreparedStatement 
 Key: DRILL-6816
 URL: https://issues.apache.org/jira/browse/DRILL-6816
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.14.0
Reporter: Khurram Faraaz


Concurrent query execution from JDBC program using PreparedStatement results in 
NPE.

Queries that were executed concurrently are (part of a query file),
{noformat}
select id from `test_tbl.json`
select count(id) from `test_tbl.json`
select count(*) from `test_tbl.json`
select * from `test_tbl.json`
{noformat}

Drill 1.14.0
git.commit.id=35a1ae23c9b280b9e73cb0f6f01808c996515454
MapR version => 6.1.0.20180911143226.GA (secure cluster)

Executing the above queries concurrently using a Statement object results in 
successful query execution.

{noformat}
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(query);
{noformat}

However, when the same queries listed above are executed using a 
PreparedStatement object we see an NPE 
{noformat}
PreparedStatement prdstmnt = conn.prepareStatement(query);
prdstmnt.executeUpdate();
{noformat}


Stack trace from drillbit.log
{noformat}
java.lang.NullPointerException
 at java.util.Objects.requireNonNull(Objects.java:203)
 at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
 at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1143)
 at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1150)
 at 
org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:511)
 at 
org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:146)
 at 
org.apache.drill.jdbc.impl.DrillPreparedStatementImpl.executeLargeUpdate(DrillPreparedStatementImpl.java:512)
 at 
org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:142)
 at RunQuery.executeQuery(RunQuery.java:61)
 at RunQuery.run(RunQuery.java:30)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] New Committer: Gautam Parai

[Khurram Faraaz]

Congrats Gautam!

On Mon, Oct 22, 2018 at 10:29 AM Abhishek Girish  wrote:

> Congrats Gautam!
>
> On Mon, Oct 22, 2018 at 10:19 AM Karthikeyan Manivannan <
> kmanivan...@mapr.com> wrote:
>
> > Congrats !
> >
> > On Mon, Oct 22, 2018 at 10:07 AM Kunal Khatua  wrote:
> >
> > > Congratulations, Gautam!
> > > On 10/22/2018 10:02:46 AM, Paul Rogers 
> > wrote:
> > > Congrats Guatam!
> > >
> > > - Paul
> > >
> > > Sent from my iPhone
> > >
> > > > On Oct 22, 2018, at 8:46 AM, salim achouche wrote:
> > > >
> > > > Congrats Gautam!
> > > >
> > > >> On Mon, Oct 22, 2018 at 7:25 AM Arina Ielchiieva wrote:
> > > >>
> > > >> The Project Management Committee (PMC) for Apache Drill has invited
> > > Gautam
> > > >> Parai to become a committer, and we are pleased to announce that he
> > has
> > > >> accepted.
> > > >>
> > > >> Gautam has become a contributor since 2016, making changes in
> various
> > > Drill
> > > >> areas including planning side. He is also one of the contributors of
> > the
> > > >> upcoming feature to support index based planning and execution.
> > > >>
> > > >> Welcome Gautam, and thank you for your contributions!
> > > >>
> > > >> - Arina
> > > >> (on behalf of Drill PMC)
> > > >>
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Salim
> > >
> > >
> >
>

Re: [ANNOUNCE] New Committer: Chunhui Shi

[Khurram Faraaz]

Congratulations Chunhui !

On Fri, Sep 28, 2018 at 11:12 AM Aman Sinha  wrote:

> Congratulations Chunhui !
>
> On Fri, Sep 28, 2018 at 10:46 AM Karthikeyan Manivannan <
> kmanivan...@mapr.com> wrote:
>
> > Congrats Chunhui!
> >
> > On Fri, Sep 28, 2018 at 10:04 AM Hanumath Rao Maduri  >
> > wrote:
> >
> > > Congratulations Chunhui.
> > >
> > > On Fri, Sep 28, 2018 at 9:26 AM Padma Penumarthy <
> > > penumarthy.pa...@gmail.com>
> > > wrote:
> > >
> > > > Congratulations Chunhui.
> > > >
> > > > Thanks
> > > > Padma
> > > >
> > > >
> > > > On Fri, Sep 28, 2018 at 2:17 AM Arina Ielchiieva 
> > > wrote:
> > > >
> > > > > The Project Management Committee (PMC) for Apache Drill has invited
> > > > Chunhui
> > > > > Shi to become a committer, and we are pleased to announce that he
> has
> > > > > accepted.
> > > > >
> > > > > Chunhui Shi has become a contributor since 2016, making changes in
> > > > various
> > > > > Drill areas. He has shown profound knowledge in Drill planning side
> > > > during
> > > > > his work to support lateral join. He is also one of the
> contributors
> > of
> > > > the
> > > > > upcoming feature to support index based planning and execution.
> > > > >
> > > > > Welcome Chunhui, and thank you for your contributions!
> > > > >
> > > > > - Arina
> > > > > (on behalf of Drill PMC)
> > > > >
> > > >
> > >
> >
>

Re: Drill in the distributed compute jungle

[Khurram Faraaz]

Paul,

I see that you mention about building a SQL++ engine in your email.
There has been some work done in that direction from folks from UC San
Diego, here is the link to the paper.

https://arxiv.org/pdf/1405.3631.pdf

The SQL++ Query Language:
Configurable, Unifying and Semi-structured

Regards,
Khurram

On Mon, Sep 10, 2018 at 12:37 PM, Timothy Farkas  wrote:

> It's an interesting idea, and I think the main inhibitor that prevents this
> from happening is that the popular big data projects are stuck on services.
> Specifically if you need distributed coordination you run a separate
> zookeeper cluster. If you need a batch compute engine you run a separate
> spark cluster. If you need a streaming engine you deploy a separate Flink
> or Apex pipeline. If you want to reuse and combine all these services to
> make a new engine, you find yourself maintaining several different clusters
> of machines, which just isn't practical.
>
> IMO the paradigm needs to shift from services to libraries. If you need
> distributed coordination import the zookeeper library and start the
> zookeeper client, which will run zookeeper threads and turn your
> application process into a member of the zookeeper quorum. If you need
> compute import the compute engine library and start the compute engine
> client and your application node will also turn into a worker node. When
> you start a library it will discover the other nodes in your application to
> form a cohesive cluster. I think this shift has already begun. Calcite is a
> library, not a query planning service. Also etcd allows you to run an etcd
> instance in your application's process using a simple function call. Arrow
> is also a library, not a service. And Apache Ignite is a compute engine
> that allows you to run the Ignite compute engine in your application's
> process https://urldefense.proofpoint.com/v2/url?u=https-3A__ignite.
> apache.org_=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=H5JEl9vb-
> mBIjic10QAbDD2vkUUKAxjO6wZO322RtdI=dNT0JmXhntaDqeyAD_
> Wm7M29lWrrWoNNAX49x3XiooI=QomY-hvPT5_IB8bHvlmYtV9z0yNUD_STHXAaWDDoEDM=
> .
>
> If we shift to thinking of libraries instead of services, then it becomes
> trivial to build new engines, since new engines would just be a library
> that depends on other libraries. Also you no longer manage several
> services, you only manage the service that you built.
>
> From the little I read about ray, it seems like ray is also moving in the
> library direction.
>
> Tim
>
>
>
> On Sun, Sep 9, 2018 at 10:21 PM Paul Rogers 
> wrote:
>
> > Hi All,
> >
> > Been reading up on distributed DB papers of late, including those passed
> > along by this group. Got me thinking about Arina's question about where
> > Drill might go in the long term.
> >
> > One thing I've noticed is that there are now quite a few distributed
> > compute frameworks, many of which support SQL in some form. A partial
> list
> > would include Drill, Presto, Impala, Hive LLAP, Spark SQL (sort of),
> > Dremio, Alibaba MaxCompute, Microsoft's Dryad, Scope and StreamS,
> Google's
> > Dremel and BigQuery and F1, the batch version of Flink -- and those are
> > just the ones off the top of my head. Seems every big Internet shop has
> > created one (Google, Facebook, Alibaba, Microsoft, etc.)
> >
> > There is probably some lesson in here for Drill. Being a distributed
> > compute engine seems to have become a commodity at this late stage of Big
> > Data. But, it is still extremely hard to build a distributed compute
> engine
> > that scales, especially for a small project like Drill.
> >
> > What unique value does Drill bring compared to the others? Certainly
> being
> > open source. Being in Java helps. Supporting xDBC is handy. Being able to
> > scan any type of data is great (but we tell people that when they get
> > serious, they should use only Parquet).
> >
> > As the team thinks about Arina's question about where Drill goes next,
> one
> > wonders if there is some way to share the load?  Rather than every
> project
> > building its own DAG optimizer and execution engine, its own distribution
> > framework, its own scanners, its own implementation of data types, and of
> > SQL functions, etc., is there a way to combine efforts?
> >
> > Ray [1] out of UC Berkeley is early days, but it promises to be exactly
> > the highly scalable, low-latency engine that Drill tries to be. Calcite
> is
> > the universal SQL parser and optimizer. Arrow wants to be the database
> > toolkit, including data format, network protocol, etc. YARN, Mesos,
> > Kubernetes and others want to manage the cluster load. Ranger and Sentry
> > want to do data security. There are now countless storage formats (HDFS
> > (classic, erasure coding, Ozone), S3, ADLS, Ceph, MapR, Aluxio, Druid,
> Kudu
> > and countless key-value stores. HMS is the metastore we all love to hate
> > and cries out for a newer, more scalable design -- but one shared by all
> > engines.
> >
> > Then, on the compute side, SQL is just one 

Re: How Does one set the S3 Signature Version in Drill?

[Khurram Faraaz]

Hi Rosenal,

You can find the details here -
https://drill.apache.org/docs/s3-storage-plugin/#configuring-the-s3-storage-plugin

Thanks,
Khurram

On Fri, Aug 31, 2018 at 10:54 AM, Rosenal, Diana J. (ELS-DAY) <
diana.j.rose...@elsevier.com> wrote:

> Hi,
>
> I am getting this error when I run a query on a json file in S3 and
> wondered how one specifies the S3 signature version in drill:
>
> org.apache.drill.common.exceptions.UserRemoteException: DATA_READ ERROR:
> Failure reading JSON file - org.jets3t.service.S3ServiceException:
> Service Error Message. -- ResponseCode: 400, ResponseStatus: Bad Request,
> XML Error Message: <
> Code>InvalidArgumentRequests specifying Server Side
> Encryption with AWS KMS managed keys require AWS Signature Version
>
> Thank you,
> Diana
>

Re: [ANNOUNCE] New Committer: Weijie Tong

[Khurram Faraaz]

Congratulations Weijie.

Thanks,
Khurram

On Fri, Aug 31, 2018 at 11:10 AM, Timothy Farkas  wrote:

> Congrats Weijie! You've done awesome work!
>
> Tim
>
> On Fri, Aug 31, 2018 at 11:02 AM Sorabh Hamirwasia 
> wrote:
>
> > Congratulations Weijie!
> >
> > Thanks,
> > Sorabh
> >
> > On Fri, Aug 31, 2018 at 10:28 AM, Paul Rogers  >
> > wrote:
> >
> > > Congratulations Weijie, thanks for your contributions to Drill.
> > > Thanks,
> > > - Paul
> > >
> > >
> > >
> > > On Friday, August 31, 2018, 8:51:30 AM PDT, Arina Ielchiieva <
> > > ar...@apache.org> wrote:
> > >
> > >  The Project Management Committee (PMC) for Apache Drill has invited
> > Weijie
> > > Tong to become a committer, and we are pleased to announce that he has
> > > accepted.
> > >
> > > Weijie Tong has become a very active contributor to Drill in recent
> > months.
> > > He contributed the Join predicate push down feature which will be
> > available
> > > in Apache Drill 1.15. The feature is non trivial and has covered
> changes
> > > to all aspects of Drill: RPC layer, Planning, and Execution.
> > >
> > > Welcome Weijie, and thank you for your contributions!
> > >
> > > - Arina
> > > (on behalf of Drill PMC)
> > >
> > >
> >
>

Re: Issue reading JSON file prohibiting from creating a Parquet file from it.

[Khurram Faraaz]

Hi,

Can you query the JSON, does the below query return results ?

select * from `test.json`;



Can you please paste the content of your JSON file here, there is no
attachment, I believe attachments will not go through. Please share the
contents of your JSON file, and the version of Drill you are on.


Thanks,

Khurram

On Thu, Aug 30, 2018 at 7:54 PM, Sri Krishna 
wrote:

> Hi,
>
>
>
> I am trying to convert a JSON file in Parquet files using Drill. The query
> is:
>
> ALTER SESSION SET `store.json.all_text_mode` = true;
>
> Use dfs.tmpp;
>
> ALTER SESSION SET `store.format` = 'parquet';
>
> CREATE TABLE `testParquet` as select * from `test.json`;
>
>
>
> The first line is done so that we don’t have to worry about numbers,
> integers etc. For now reading them as strings works. When I run this query
> I get this error message (not clear):
>
> Error: INTERNAL_ERROR ERROR: You tried to start when you are using a
> ValueWriter  of type SingleListWriter.
>
>
>
> Attached is the JSON file and the trouble is with the first line. The line
> by itself can be folded into Parquet file(above CTAS works) and so are the
> rest of them by themselves. Both together gives this error. I ran a query
> to just read the file and I get the same error with this line and others
> but not alone (just like CTAS). I can get around reading the file by
> setting mixed mode as:
>
> ALTER SESSION SET  `exec.enable_union_type` = true;
>
> But then I get an error that List type isn’t supported (I assume they are
> talking about mixed types in an array).
>
> Here is the stack trace (from enabling verbose) in case of write failure:
>
> Error: INTERNAL_ERROR ERROR: You tried to start when you are using a
> ValueWriter
>
> of type SingleListWriter.
>
>
>
> Fragment 0:0
>
>
>
> [Error Id: 1ae5c2ce-e1ef-40f9-afce-d1e00ac9fa15 on IMC28859.imc2.com:31010
> ]
>
>
>
>   (java.lang.IllegalStateException) You tried to start when you are using
> a Valu
>
> eWriter of type SingleListWriter.
>
> org.apache.drill.exec.vector.complex.impl.
> AbstractFieldWriter.start():78
>
> org.apache.drill.exec.vector.complex.impl.SingleListWriter.start():71
>
> org.apache.drill.exec.vector.complex.fn.JsonReader.
> writeDataAllText():430
>
> org.apache.drill.exec.vector.complex.fn.JsonReader.
> writeDataAllText():462
>
> org.apache.drill.exec.vector.complex.fn.JsonReader.
> writeDataAllText():462
>
> org.apache.drill.exec.vector.complex.fn.JsonReader.
> writeDataAllText():462
>
> org.apache.drill.exec.vector.complex.fn.JsonReader.
> writeDataAllText():462
>
> org.apache.drill.exec.vector.complex.fn.JsonReader.
> writeDataAllText():462
>
> org.apache.drill.exec.vector.complex.fn.JsonReader.
> writeDataSwitch():311
>
> org.apache.drill.exec.vector.complex.fn.JsonReader.writeToVector():254
>
> org.apache.drill.exec.vector.complex.fn.JsonReader.write():209
>
> org.apache.drill.exec.store.easy.json.JSONRecordReader.next():214
>
> org.apache.drill.exec.physical.impl.ScanBatch.next():177
>
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
>
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
>
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63
>
> org.apache.drill.exec.physical.impl.project.
> ProjectRecordBatch.innerNext():1
>
> 42
>
> org.apache.drill.exec.record.AbstractRecordBatch.next():172
>
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
>
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
>
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63
>
> org.apache.drill.exec.physical.impl.project.
> ProjectRecordBatch.innerNext():1
>
> 42
>
> org.apache.drill.exec.record.AbstractRecordBatch.next():172
>
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
>
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
>
> org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext():90
>
> org.apache.drill.exec.record.AbstractRecordBatch.next():172
>
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
>
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
>
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63
>
> org.apache.drill.exec.physical.impl.project.
> ProjectRecordBatch.innerNext():1
>
> 42
>
> org.apache.drill.exec.record.AbstractRecordBatch.next():172
>
> org.apache.drill.exec.physical.impl.BaseRootExec.next():103
>
> org.apache.drill.exec.physical.impl.ScreenCreator$
> ScreenRoot.innerNext():83
>
> org.apache.drill.exec.physical.impl.BaseRootExec.next():93
>
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():294
>
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():281
>
> java.security.AccessController.doPrivileged():-2
>
> javax.security.auth.Subject.doAs():422
>
> org.apache.hadoop.security.UserGroupInformation.doAs():1657
>
> 

[jira] [Resolved] (DRILL-6563) TPCDS query 10 has regressed

[Khurram Faraaz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Khurram Faraaz resolved DRILL-6563.
---
Resolution: Fixed

> TPCDS query 10 has regressed 
> -
>
> Key: DRILL-6563
> URL: https://issues.apache.org/jira/browse/DRILL-6563
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning  Optimization
>Affects Versions: 1.14.0
>    Reporter: Khurram Faraaz
>Assignee: Pritesh Maker
>Priority: Major
> Fix For: 1.15.0
>
> Attachments: 24ca3c6c-90e1-a4bf-6c6f-3f981fa2d043.sys.drill, 
> query10.fast_plan_old_commit, tpcds_query_10_plan_slow_140d09e.pdf, 
> tpcds_query_plan_10_140d09e.txt
>
>
> TPC-DS query 10 has regressed in performance from taking 3.5 seconds to 
> execute on Apache Drill 1.14.0 commit  b92f599 , to 07 min 51.851 sec to 
> complete execution on Apache Drill 1.14.0 commit 140d09e. Query was executed 
> over SF1 parquet views on a 4 node cluster.
> Query plan from old and newer commit is attached here, with the query profile 
> from newer commit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] New PMC member: Volodymyr Vysotskyi

[Khurram Faraaz]

Congratulations Volodymyr!

Regards,
Khurram

On Fri, Aug 24, 2018 at 10:25 AM, Hanumath Rao Maduri 
wrote:

> Congratulations Volodymyr!
>
> Thanks,
> -Hanu
>
> On Fri, Aug 24, 2018 at 10:22 AM Paul Rogers 
> wrote:
>
> > Congratulations Volodymyr!
> > Thanks,
> > - Paul
> >
> >
> >
> > On Friday, August 24, 2018, 5:53:25 AM PDT, Arina Ielchiieva <
> > ar...@apache.org> wrote:
> >
> >  I am pleased to announce that Drill PMC invited Volodymyr Vysotskyi to
> the
> > PMC and he has accepted the invitation.
> >
> > Congratulations Vova and thanks for your contributions!
> >
> > - Arina
> > (on behalf of Drill PMC)
> >
>

Re: [ANNOUNCE] New PMC member: Boaz Ben-Zvi

[Khurram Faraaz]

Congratulations Boaz.

On Fri, Aug 17, 2018 at 10:47 AM, shi.chunhui <
shi.chun...@aliyun.com.invalid> wrote:

> Congrats Boaz!
> --
> Sender:Arina Ielchiieva 
> Sent at:2018 Aug 17 (Fri) 17:51
> To:dev ; user 
> Subject:[ANNOUNCE] New PMC member: Boaz Ben-Zvi
>
> I am pleased to announce that Drill PMC invited Boaz Ben-Zvi to the PMC and
> he has accepted the invitation.
>
> Congratulations Boaz and thanks for your contributions!
>
> - Arina
> (on behalf of Drill PMC)
>

Re: [VOTE] Apache Drill release 1.14.0 - RC3

[Khurram Faraaz]

- Downloaded tarballs and deployed binaries on 4 node cluster.
- Executed basic SQL queries from sqlline and from Web UI.
- Verified features on the Web UI.

Looks good.
+1 (non-binding)

On Wed, Aug 1, 2018 at 9:55 PM, Kunal Khatua  wrote:

> Built from source and tried the binaries.
>
> Tested spill-to-disk behavior, a couple of concurrent queries and general
> UX checks. LGTM.
>
> +1 (non-binding)
> On 8/1/2018 4:58:08 PM, Boaz Ben-Zvi  wrote:
> Thanks Vlad for bringing these two points to our attention.
>
> Therefore the vote on RC3 should be open till Friday, August 3rd, at 6
> PM PDT.
>
> And we should (sans any new issue) get enough PMC +1 votes on RC3 by
> Friday.
>
> Thanks,
>
> Boaz
>
>
> On 8/1/18 8:40 AM, Vlad Rozov wrote:
> > Apache release votes should be open for at least 72 hours [1] and every
> new RC requires that PMC "Before voting +1 PMC members are required to
> download the signed source code package, compile it as provided, and test
> the resulting executable on their own platform, along with also verifying
> that the package meets the requirements of the ASF policy on releases".
> >
> > Thank you,
> >
> > Vlad
> >
> > [1]https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__www.apache.org_legal_release-2Dpolicy.html=DwIBaQ=
> cskdkSMqhcnjZxdQVpwTXg=PqKay2uOMZUqopDRKNfBtZSlsp2meG
> OxWNAVHxHnXCk=7cmJ-j2LSiMs7xMoaBk4MmgV24HxaR5LzmDYE6dlKvw=
> GuxbeqAnbOHCDsEpaMRlPd4ursbQfyIQl5wPywp0V2g=
> >
> > On 2018/07/31 22:09:04, Boaz Ben-Zvi wrote:
> >> Hi RC reviewers and testers,
> >>
> >> There are a couple of RC2 minor issues that are flagged as errors
> >> in the IDE ( in Eclipse, and may be in IntelliJ ). See DRILL-6650 and
> >> DRILL-6651 for detail.
> >>
> >> These two do *not* matter for the Maven build, or for testing the
> >> tarballs, etc. So if you started a long testing cycle with RC2, you may
> >> continue.
> >>
> >> I will produce a new RC3 soon to include the fixes for the above. (Note
> >> that this RC3 would be *force-pushed* into branch 1.14.0 , thus erasing
> >> the RC2 commit ID)
> >> And if no one objects, the voting deadline would remain as is (Aug 2nd)
> >> as the differences have a very minor impact.
> >>
> >> Thanks,
> >>
> >> Boaz
> >>
> >>
> >> On 7/30/18 3:57 PM, Boaz Ben-Zvi wrote:
> >>> Hi Drillers,
> >>>
> >>> Proposing the *third* Release Candidate (RC2) for the Apache Drill,
> >>> version 1.14.0 .
> >>>
> >>> This RC2 includes 235 committed Jiras [1]. Thanks to all the Drill
> >>> developers who works hard and contributed to this release.
> >>>
> >>> The RC2 tarballs are hosted at [2] , and the Maven artifacts are at
> [3].
> >>>
> >>> This Release Candidate is based on (Apache Drill branch named
> >>> "1.14.0") commit ID: 4da8aff88966adee5d7438024a826bb599450a6f ,
> >>> available at [4].
> >>>
> >>> Please download and try/test this Release Candidate.
> >>>
> >>> Given that our bylaws require 3 business days, the vote would end on
> >>> Thursday, August 2nd, 2018 at 5:00 PM PDT .
> >>>
> >>> [ ] +1
> >>> [ ] +0
> >>> [ ] -1
> >>>
> >>> My vote is +1 !!
> >>>
> >>> Thank you,
> >>>
> >>> Boaz
> >>>
> >>>
> >>> [1]
> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_secure_ReleaseNote.jspa-3FprojectId-3D12313820-26version-
> 3D12342097=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=
> PqKay2uOMZUqopDRKNfBtZSlsp2meGOxWNAVHxHnXCk=7cmJ-
> j2LSiMs7xMoaBk4MmgV24HxaR5LzmDYE6dlKvw=wPpmkjpk5reuOsI2Zhwg0-9vvYAQb_
> bWrhoJmkw4Bbc=
> >>>
> >>>
> >>> [2]https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__home.apache.org_-7Eboaz_drill_releases_1.14.0_rc2_=DwIBaQ=
> cskdkSMqhcnjZxdQVpwTXg=PqKay2uOMZUqopDRKNfBtZSlsp2meG
> OxWNAVHxHnXCk=7cmJ-j2LSiMs7xMoaBk4MmgV24HxaR5LzmD
> YE6dlKvw=dINsXomnONCGf0t39_J5VB1_T16dqj7yq34sNyU72_M=
> >>>
> >>> [3]
> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__
> repository.apache.org_content_repositories_orgapachedrill-
> 2D1050=DwIBaQ=cskdkSMqhcnjZxdQVpwTXg=PqKay2uOMZUqopDRKNfBtZSlsp2meG
> OxWNAVHxHnXCk=7cmJ-j2LSiMs7xMoaBk4MmgV24HxaR5LzmD
> YE6dlKvw=K2ffEjpPckQ9-7YybbeHm_ClMS-F7K9pv_N4xbAhs4E=
> >>>
> >>> [4]https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__github.com_apache_drill_tree_1.14.0=DwIBaQ=
> cskdkSMqhcnjZxdQVpwTXg=PqKay2uOMZUqopDRKNfBtZSlsp2meG
> OxWNAVHxHnXCk=7cmJ-j2LSiMs7xMoaBk4MmgV24HxaR5LzmDYE6dlKvw=
> x4FT03yAXZOcnqY0kI1vmRJmuGa25ZGVhKVKjXH2ML4=
> >>>
> >>> OR
> >>>
> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> com_Ben-2DZvi_drill_tree_drill-2D1.14.0=DwIBaQ=
> cskdkSMqhcnjZxdQVpwTXg=PqKay2uOMZUqopDRKNfBtZSlsp2meG
> OxWNAVHxHnXCk=7cmJ-j2LSiMs7xMoaBk4MmgV24HxaR5LzmDYE6dlKvw=
> aGU2JYBo3RzIdTX6yNrVzyoqz9evUi9smRYMs2YfG2A=
> >>>
> >>>
>
>

[jira] [Created] (DRILL-6602) Drill query cancellation should log the originator

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6602:
-

 Summary: Drill query cancellation should log the originator
 Key: DRILL-6602
 URL: https://issues.apache.org/jira/browse/DRILL-6602
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Reporter: Khurram Faraaz


The Drill query Cancel functionality does not log the origin of the Cancel 
request.

(1). security issue, we don't know who (which user) canceled the query.

(2). debugging such a canceled query can be a pain to find the origin and the 
root cause of why a query was canceled.

On Apache Drill 1.14.0 git.commit.id.abbrev=b0314a3, we have observed this 
problem, and it can be consistently reproduced. There is no information at all 
about the origin of why the query was canceled in the drillbit.log

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6595) IllegalAccessError: tried to access field org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files from class org.apache.drill.exec.store.pa

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6595:
-

 Summary: IllegalAccessError: tried to access field 
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files
 from class org.apache.drill.exec.store.parquet.metadata.Metadata_V3
 Key: DRILL-6595
 URL: https://issues.apache.org/jira/browse/DRILL-6595
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.14.0
Reporter: Khurram Faraaz


java.lang.IllegalAccessError reported in drillbit.out file 
Apache Drill 1.14.0 git.commit.id.abbrev=b0314a3

{noformat}
Jul 11, 2018 1:33:39 PM 
com.fasterxml.jackson.module.afterburner.deser.OptimizedSettableBeanProperty 
_reportProblem
WARNING: Disabling Afterburner deserialization for class 
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3
 (field #1; mutator 
com.fasterxml.jackson.module.afterburner.deser.SettableObjectFieldProperty), 
due to access error (type java.lang.IllegalAccessError, message=tried to access 
field 
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files
 from class 
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer48257508)

java.lang.IllegalAccessError: tried to access field 
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files
 from class 
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer48257508
 at 
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer48257508.objectField(org/apache/drill/exec/store/parquet/metadata/Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer.java)
 at 
com.fasterxml.jackson.module.afterburner.deser.SettableObjectFieldProperty.deserializeAndSet(SettableObjectFieldProperty.java:50)
 at 
com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288)
 at 
com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:189)
 at 
com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:120)
 at 
com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:130)
 at 
com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:97)
 at 
com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:254)
 at 
com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:68)
 at 
com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4001)
 at 
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3058)
 at 
org.apache.drill.exec.store.parquet.metadata.Metadata.readBlockMeta(Metadata.java:617)
 at 
org.apache.drill.exec.store.parquet.metadata.Metadata.readBlockMeta(Metadata.java:156)
 at 
org.apache.drill.exec.store.parquet.ParquetGroupScan.expandSelectionFromMetadataCache(ParquetGroupScan.java:374)
 at 
org.apache.drill.exec.store.parquet.ParquetGroupScan.expandIfNecessary(ParquetGroupScan.java:337)
 at 
org.apache.drill.exec.store.parquet.ParquetGroupScan.(ParquetGroupScan.java:121)
 at 
org.apache.drill.exec.store.parquet.ParquetGroupScan.(ParquetGroupScan.java:102)
 at 
org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(ParquetFormatPlugin.java:180)
 at 
org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(ParquetFormatPlugin.java:70)
 at 
org.apache.drill.exec.store.dfs.FileSystemPlugin.getPhysicalScan(FileSystemPlugin.java:136)
 at 
org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:114)
 at 
org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:109)
 at 
org.apache.drill.exec.planner.logical.DrillTable.getGroupScan(DrillTable.java:99)
 at 
org.apache.drill.exec.planner.logical.DrillPushProjectIntoScanRule.canPushProjectIntoScan(DrillPushProjectIntoScanRule.java:125)
 at 
org.apache.drill.exec.planner.logical.DrillPushProjectIntoScanRule.onMatch(DrillPushProjectIntoScanRule.java:81)
 at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
 at 
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:652)
 at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:368)
 at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:426)
 at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:366)
 at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRawDrel(DefaultSqlHandler.java:255

[jira] [Created] (DRILL-6590) DATA_WRITE ERROR: Hash Join failed to write to output file: /tmp/drill/spill/24bac407

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6590:
-

 Summary: DATA_WRITE ERROR: Hash Join failed to write to output 
file: /tmp/drill/spill/24bac407
 Key: DRILL-6590
 URL: https://issues.apache.org/jira/browse/DRILL-6590
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz


Apache Drill 1.14.0 git.commit.id.abbrev=eb946b0

There was enough space on /tmp, however Hash Join failed to write to spill file
[test@qa102-45 drill-1.14.0]# clush -a df -h /tmp
: Filesystem Size Used Avail Use% Mounted on
: /dev/mapper/vg_root-lv_root 500G 150G 351G 30% /
: Filesystem Size Used Avail Use% Mounted on
: /dev/mapper/vg_root-lv_root 500G 17G 484G 4% /
: Filesystem Size Used Avail Use% Mounted on
: /dev/mapper/vg_root-lv_root 500G 14G 487G 3% /
: Filesystem Size Used Avail Use% Mounted on
: /dev/mapper/vg_root-lv_root 500G 13G 488G 3% /

Stack trace from drillbit.log
{noformat}
2018-07-10 18:17:51,953 [BitServer-10] WARN o.a.d.exec.rpc.control.WorkEventBus 
- A fragment message arrived but there was no registered listener for that 
message: profile {
 state: FAILED
 error {
 error_id: "6e258de2-2d4f-4b48-967d-df1b329955cd"
 endpoint {
 address: "qa102-48.qa.lab"
 user_port: 31010
 control_port: 31011
 data_port: 31012
 version: "1.14.0-SNAPSHOT"
 state: STARTUP
 }
 error_type: DATA_WRITE
 message: "DATA_WRITE ERROR: Hash Join failed to write to output file: 
/tmp/drill/spill/24bac407-2adb-5763-ed08-cb5714dca2c0_HashJoin_4-22-53/spill15_outer\n\nFragment
 4:53\n\n[Error Id: 6e258de2-2d4f-4b48-967d-df1b329955cd on 
qa102-48.qa.lab:31010]"
 exception {
 exception_class: "java.nio.channels.ClosedByInterruptException"
 stack_trace {
 class_name: "..."
 line_number: 0
 method_name: "..."
 is_native_method: false
 }
 stack_trace {
 class_name: "com.google.protobuf.CodedOutputStream"
 file_name: "CodedOutputStream.java"
 line_number: 833
 method_name: "refreshBuffer"
 is_native_method: false
 }
 stack_trace {
 class_name: "com.google.protobuf.CodedOutputStream"
 file_name: "CodedOutputStream.java"
 line_number: 843
 method_name: "flush"
 is_native_method: false
 }
 stack_trace {
 class_name: "com.google.protobuf.AbstractMessageLite"
 file_name: "AbstractMessageLite.java"
 line_number: 91
 method_name: "writeDelimitedTo"
 is_native_method: false
 }
 stack_trace {
 class_name: "org.apache.drill.exec.cache.VectorSerializer$Writer"
 file_name: "VectorSerializer.java"
 line_number: 97
 method_name: "write"
 is_native_method: false
 }
 stack_trace {
 class_name: "org.apache.drill.exec.physical.impl.common.HashPartition"
 file_name: "HashPartition.java"
 line_number: 346
 method_name: "spillThisPartition"
 is_native_method: false
 }
 stack_trace {
 class_name: "org.apache.drill.exec.physical.impl.common.HashPartition"
 file_name: "HashPartition.java"
 line_number: 263
 method_name: "completeABatch"
 is_native_method: false
 }
 stack_trace {
 class_name: "org.apache.drill.exec.physical.impl.common.HashPartition"
 file_name: "HashPartition.java"
 line_number: 237
 method_name: "completeAnOuterBatch"
 is_native_method: false
 }
 stack_trace {
 class_name: "org.apache.drill.exec.physical.impl.common.HashPartition"
 file_name: "HashPartition.java"
 line_number: 232
 method_name: "appendOuterRow"
 is_native_method: false
 }
 stack_trace {
 class_name: "org.apache.drill.exec.test.generated.HashJoinProbeGen49"
 file_name: "HashJoinProbeTemplate.java"
 line_number: 306
 method_name: "executeProbePhase"
 is_native_method: false
 }
 stack_trace {
 class_name: "org.apache.drill.exec.test.generated.HashJoinProbeGen49"
 file_name: "HashJoinProbeTemplate.java"
 line_number: 393
 method_name: "probeAndProject"
 is_native_method: false
 }
 stack_trace {
 class_name: "org.apache.drill.exec.physical.impl.join.HashJoinBatch"
 file_name: "HashJoinBatch.java"
 line_number: 357
 method_name: "innerNext"
 is_native_method: false
 }
 stack_trace {
 class_name: "org.apache.drill.exec.record.AbstractRecordBatch"
 file_name: "AbstractRecordBatch.java"
 line_number: 172
 method_name: "next"
 is_native_method: false
 }
 stack_trace {
 class_name: "org.apache.drill.exec.record.AbstractRecordBatch"
 file_name: "AbstractRecordBatch.java"
 line_number: 119
 method_name: "next"
 is_native_method: false
 }
 stack_trace {
 class_name: "org.apache.drill.exec.physical.impl.join.HashJoinBatch"
 file_name: "HashJoinBatch.java"
 

[jira] [Created] (DRILL-6583) UI usability issue

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6583:
-

 Summary: UI usability issue
 Key: DRILL-6583
 URL: https://issues.apache.org/jira/browse/DRILL-6583
 Project: Apache Drill
  Issue Type: Bug
  Components: Web Server
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
 Attachments: UI_usability_issue_AD_1_14_0.png

When a query is under execution, on the web UI we see this text which is 
actually a set of different links that help navigate to different pages on the 
UI, below that query's profile.

Apache Drill 1.14.0

git.commit.id.abbrev=f481a7c

They all appear on a single line with no spacing and a typo, the formatting of 
the text for those links needs to be changed / improved.

Attached is a screenshot for the issue, look for "FirstPrevious1NextLast" on 
the bottom left of the screenshot.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] 1.14.0 release

[Khurram Faraaz]

Do we plan to fix this one too, because this is a regression from Apache
Drill 1.13.0.
https://issues.apache.org/jira/browse/DRILL-6453

On Mon, Jul 2, 2018 at 9:33 PM, Kunal Khatua  wrote:

> DRILL-6570 seems like a must-have (release blocker, IMHO).
> On 7/2/2018 8:02:00 PM, Boaz Ben-Zvi  wrote:
> Let's try to make progress on the 1.14 release, aiming for a Release
> Candidate towards the end of this week (a little ambitious, with the
> July 4th and people on vacations).
>
> Current Status of the previously requested Jiras:
>
> ==
>
> In Progress - DRILL-6104: Generic Logfile Format Plugin
>
> PR - DRILL-6422: Update Guava to 23.0 and shade it
>
> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg
>
> Ready2Commit: DRILL-5977: predicate pushdown support kafkaMsgOffset
>
> Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions
>
> Ready2Commit: DRILL-6577: Change Hash-Join default to not fallback (into
> pre-1.14 unlimited memory)
>
> Committed: DRILL-6353: Upgrade Parquet MR dependencies
>
> Committed: DRILL-6310: limit batch size for hash aggregate
>
> ===
>
> And there are few more open or in a PR state.
>
> Lets try and most of these ready by the end of the week.
>
> Boaz
>
>
>

[jira] [Created] (DRILL-6563) TPCDS query 10 has regressed

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6563:
-

 Summary: TPCDS query 10 has regressed 
 Key: DRILL-6563
 URL: https://issues.apache.org/jira/browse/DRILL-6563
 Project: Apache Drill
  Issue Type: Bug
  Components: Query Planning  Optimization
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
 Attachments: 24ca3c6c-90e1-a4bf-6c6f-3f981fa2d043.sys.drill, 
query10.fast_plan_old_commit, tpcds_query_10_plan_slow_140d09e.pdf, 
tpcds_query_plan_10_140d09e.txt

TPC-DS query 10 has regressed in performance from taking 3.5 seconds to execute 
on Apache Drill 1.14.0 commit  b92f599 , to 07 min 51.851 sec to complete 
execution on Apache Drill 1.14.0 commit 140d09e. Query was executed over SF1 
parquet views on a 4 node cluster.

Query plan from old and newer commit is attached here, with the query profile 
from newer commit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6548) IllegalStateException: Unexpected EMIT outcome received in buildSchema phase

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6548:
-

 Summary: IllegalStateException: Unexpected EMIT outcome received 
in buildSchema phase
 Key: DRILL-6548
 URL: https://issues.apache.org/jira/browse/DRILL-6548
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Assignee: Sorabh Hamirwasia


On a four node Apache Drill 1.14.0 master branch against TPC-DS SF1 parquet 
data (parquet views)
git.commit.id.abbrev=b92f599

TPC-DS query 69 fails with IllegalStateException: Unexpected EMIT outcome 
received in buildSchema phase

Failing query is,

{noformat}
2018-06-27 15:24:39,493 [24cbf157-e95c-42ab-7307-f75f5943a277:foreman] INFO 
o.a.drill.exec.work.foreman.Foreman - Query text for query id 
24cbf157-e95c-42ab-7307-f75f5943a277: SELECT cd_gender,
cd_marital_status,
cd_education_status,
Count(*) cnt1,
cd_purchase_estimate,
Count(*) cnt2,
cd_credit_rating,
FROM customer c,
customer_address ca,
customer_demographics
WHERE c.c_current_addr_sk = ca.ca_address_sk
AND ca_state IN ( 'KS', 'AZ', 'NE' )
AND cd_demo_sk = c.c_current_cdemo_sk
AND EXISTS (SELECT *
FROM store_sales,
date_dim
WHERE c.c_customer_sk = ss_customer_sk
AND ss_sold_date_sk = d_date_sk
AND d_year = 2004
AND d_moy BETWEEN 3 AND 3 + 2)
AND ( NOT EXISTS (SELECT *
FROM web_sales,
date_dim
WHERE c.c_customer_sk = ws_bill_customer_sk
AND ws_sold_date_sk = d_date_sk
AND d_year = 2004
AND d_moy BETWEEN 3 AND 3 + 2)
AND NOT EXISTS (SELECT *
FROM catalog_sales,
date_dim
WHERE c.c_customer_sk = cs_ship_customer_sk
AND cs_sold_date_sk = d_date_sk
AND d_year = 2004
AND d_moy BETWEEN 3 AND 3 + 2) )
GROUP BY cd_gender,
cd_marital_status,
cd_education_status,
cd_purchase_estimate,
cd_credit_rating
ORDER BY cd_gender,
cd_marital_status,
cd_education_status,
cd_purchase_estimate,
cd_credit_rating
cd_credit_rating
LIMIT 100
{noformat}

Stack trace from drillbit.log

{noformat}
2018-06-27 15:24:42,130 [24cbf157-e95c-42ab-7307-f75f5943a277:frag:0:0] ERROR 
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: 
Unexpected EMIT outcome received in buildSchema phase

Fragment 0:0

[Error Id: ba1a35e0-807e-4bab-b820-8aa6aad80e87 on qa102-45.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
IllegalStateException: Unexpected EMIT outcome received in buildSchema phase

Fragment 0:0

[Error Id: ba1a35e0-807e-4bab-b820-8aa6aad80e87 on qa102-45.qa.lab:31010]
 at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
 ~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361)
 [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216)
 [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327)
 [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[na:1.8.0_161]
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[na:1.8.0_161]
 at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.lang.IllegalStateException: Unexpected EMIT outcome received in 
buildSchema phase
 at 
org.apache.drill.exec.physical.impl.TopN.TopNBatch.buildSchema(TopNBatch.java:178)
 ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:152)
 ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
 ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
 ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
 ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT

[jira] [Created] (DRILL-6518) DESCRIBE command on Drill created parquet table does not return results

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6518:
-

 Summary: DESCRIBE command on Drill created parquet table does not 
return results
 Key: DRILL-6518
 URL: https://issues.apache.org/jira/browse/DRILL-6518
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - Parquet
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
 Attachments: 0_0_0.parquet, item.drill.parquet_metadata

Describe command on a Drill (1.14.0) created parquet table, does not return the 
table description.
Parquet file and parquet metadata cache file for item table is attached here, 
it has the details of column types in the metadata cache file.

{noformat}
Apache Drill 1.14.0-SNAPSHOT 
commit : b447260e49dc4a8c906f5b310c037fe6dd77166f
{noformat}

{noformat}

// DESCRIBE commands returns no information about the table.


0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> describe 
dfs.`/drill/testdata/tpcds_sf1/parquet/item`;
+--++--+
| COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
+--++--+
+--++--+
No rows selected (0.221 seconds)
0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> refresh table metadata 
dfs.`/drill/testdata/tpcds_sf1/parquet/item`;
+---+--+
| ok | summary |
+---+--+
| true | Successfully updated metadata for table 
/drill/testdata/tpcds_sf1/parquet/item. |
+---+--+
1 row selected (0.173 seconds)
0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> describe 
dfs.`/drill/testdata/tpcds_sf1/parquet/item`;
+--++--+
| COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
+--++--+
+--++--+
No rows selected (0.229 seconds)
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6517) IllegalStateException: Record count not set for this vector container

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6517:
-

 Summary: IllegalStateException: Record count not set for this 
vector container
 Key: DRILL-6517
 URL: https://issues.apache.org/jira/browse/DRILL-6517
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Assignee: Padma Penumarthy
 Attachments: 24d7b377-7589-7928-f34f-57d02061acef.sys.drill

TPC-DS query is Canceled after 2 hrs and 47 mins and we see an 
IllegalStateException: Record count not set for this vector container, in 
drillbit.log

Steps to reproduce the problem, query profile 
(24d7b377-7589-7928-f34f-57d02061acef) is attached here.

{noformat}
In drill-env.sh set max direct memory to 12G on all 4 nodes in cluster
export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-"12G"}

and set these options from sqlline,
alter system set `planner.memory.max_query_memory_per_node` = 10737418240;
alter system set `drill.exec.hashagg.fallback.enabled` = true;

To run the query (replace IP-ADDRESS with your foreman node's IP address)
cd /opt/mapr/drill/drill-1.14.0/bin
./sqlline -u 
"jdbc:drill:schema=dfs.tpcds_sf1_parquet_views;drillbit=" -f 
/root/query72.sql

{noformat}

Stack trace from drillbit.log

{noformat}
2018-06-18 20:08:51,912 [24d7b377-7589-7928-f34f-57d02061acef:frag:4:49] ERROR 
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: 
Record count not set for this vector container

Fragment 4:49

[Error Id: 73177a1c-f7aa-4c9e-99e1-d6e1280e3f27 on qa102-45.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
IllegalStateException: Record count not set for this vector container

Fragment 4:49

[Error Id: 73177a1c-f7aa-4c9e-99e1-d6e1280e3f27 on qa102-45.qa.lab:31010]
 at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
 ~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361)
 [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216)
 [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327)
 [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[na:1.8.0_161]
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[na:1.8.0_161]
 at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.lang.IllegalStateException: Record count not set for this 
vector container
 at com.google.common.base.Preconditions.checkState(Preconditions.java:173) 
~[guava-18.0.jar:na]
 at 
org.apache.drill.exec.record.VectorContainer.getRecordCount(VectorContainer.java:394)
 ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.getRecordCount(RemovingRecordBatch.java:49)
 ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.record.RecordBatchSizer.(RecordBatchSizer.java:690) 
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.record.RecordBatchSizer.(RecordBatchSizer.java:662) 
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.record.JoinBatchMemoryManager.update(JoinBatchMemoryManager.java:73)
 ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.record.JoinBatchMemoryManager.update(JoinBatchMemoryManager.java:79)
 ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.physical.impl.join.HashJoinBatch.prefetchFirstBatchFromBothSides(HashJoinBatch.java:242)
 ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.physical.impl.join.HashJoinBatch.buildSchema(HashJoinBatch.java:218)
 ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:152)
 ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.physical.impl.join.HashJoinBatch.sniffNonEmptyBatch(HashJoinBatch.java:276)
 ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.physical.impl.join.HashJoinBatch.prefetchFirstBatchFromBothSides(HashJoinBatch.java:238)
 ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.physical.impl.join.HashJ

Re: [ANNOUNCE] New Committer: Padma Penumarthy

[Khurram Faraaz]

Congratulations Padma! Well deserved.


Thanks,

Khurram


From: Paul Rogers 
Sent: Friday, June 15, 2018 7:50:05 PM
To: dev@drill.apache.org
Subject: Re: [ANNOUNCE] New Committer: Padma Penumarthy

Congratulations! Well deserved, if just from the number of times you've 
reviewed my code.

Thanks,
- Paul



On Friday, June 15, 2018, 9:36:44 AM PDT, Aman Sinha  
wrote:

 The Project Management Committee (PMC) for Apache Drill has invited Padma
Penumarthy to become a committer, and we are pleased to announce that she has
accepted.

Padma has been contributing to Drill for about 1 1/2 years.  She has made
improvements for work-unit assignment in the parallelizer, performance of
filter operator for pattern matching and (more recently) on the batch
sizing for several operators: Flatten, MergeJoin, HashJoin, UnionAll.

Welcome Padma, and thank you for your contributions.  Keep up the good work
!

-Aman
(on behalf of Drill PMC)

[jira] [Created] (DRILL-6453) TPC-DS query 72 has regressed

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6453:
-

 Summary: TPC-DS query 72 has regressed
 Key: DRILL-6453
 URL: https://issues.apache.org/jira/browse/DRILL-6453
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
 Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill

TPC-DS query 72 seems to have regressed, query profile for the case where it 
Canceled after 2 hours on Drill 1.14.0 is attached here.

{noformat}
On, Drill 1.14.0-SNAPSHOT 
commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took 
around 55 seconds to execute)
SF1 parquet data on 4 nodes; 
planner.memory.max_query_memory_per_node = 10737418240. 
drill.exec.hashagg.fallback.enabled = true

TPC-DS query 72 executed successfully & took 47 seconds to complete execution.
{noformat}


{noformat}
TPC-DS data in the below run has date values stored as DATE datatype and not 
VARCHAR type

On, Drill 1.14.0-SNAPSHOT
commit : 82e1a12
SF1 parquet data on 4 nodes; 
planner.memory.max_query_memory_per_node = 10737418240. 
drill.exec.hashagg.fallback.enabled = true
and
alter system set `exec.hashjoin.num_partitions` = 1;

TPC-DS query 72 executed for 2 hrs and 11 mins and did not complete, I had to 
Cancel it by stopping the Foreman drillbit.
As a result several minor fragments are reported to be in 
CANCELLATION_REQUESTED state on UI.
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6452) document steps to execute SQL queries from Postman (chrome extension) on Drill

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6452:
-

 Summary: document steps to execute SQL queries from Postman 
(chrome extension) on Drill
 Key: DRILL-6452
 URL: https://issues.apache.org/jira/browse/DRILL-6452
 Project: Apache Drill
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.14.0
Reporter: Khurram Faraaz


We need documentation to list the steps with screen shots about executing SQL 
queries from Postman (chrome extension) on Drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] New Committer: Timothy Farkas

[Khurram Faraaz]

Congrats, Tim!


Regards,

Khurram


From: Arina Yelchiyeva 
Sent: Friday, May 25, 2018 12:04:16 PM
To: dev@drill.apache.org
Subject: Re: [ANNOUNCE] New Committer: Timothy Farkas

Congrats, Tim!

Kind regards,
Arina

> On May 25, 2018, at 9:59 PM, Kunal Khatua  wrote:
>
> Congratulations, Timothy !
>
> On 5/25/2018 11:58:31 AM, Aman Sinha  wrote:
> The Project Management Committee (PMC) for Apache Drill has invited Timothy
> Farkas to become a committer, and we are pleased to announce that he
> has accepted.
>
> Tim has become an active contributor to Drill in less than a year. During
> this time he has contributed to addressing flaky unit tests, fixing memory
> leaks in certain operators, enhancing the system options framework to be
> more extensible and setting up the Travis CI tests. More recently, he
> worked on the memory sizing calculations for hash join.
>
> Welcome Tim, and thank you for your contributions. Keep up the good work !
>
> -Aman
> (on behalf of Drill PMC)

[jira] [Created] (DRILL-6441) IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6441:
-

 Summary: IllegalStateException: Allocator[ROOT] closed with 
outstanding child allocators.
 Key: DRILL-6441
 URL: https://issues.apache.org/jira/browse/DRILL-6441
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz


This is seen when the drillbit is shutdown using $DRILL_HOME/bin/drillbit.sh 
stop, it happens when BaseAllocator tries to close.
Apache Drill 1.14.0 commit c6c5d27d91468a29656bee2acba55d3321978aab

{noformat}
 
2018-05-11 15:19:44,510 [2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:frag:0:0] INFO 
o.a.d.e.w.f.FragmentStatusReporter - 2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:0:0: 
State to report: FINISHED
Wed May 16 10:17:13 PDT 2018 Terminating drillbit pid 32076
2018-05-16 10:17:13,793 [Drillbit-ShutdownHook#0] INFO 
o.apache.drill.exec.server.Drillbit - Received shutdown request.
2018-05-16 10:17:14,876 [Drillbit-ShutdownHook#0] INFO 
o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 20, cache miss 
count: 6, hit rate: 70%
2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] ERROR 
o.a.d.exec.server.BootStrapContext - Error while closing
java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding child 
allocators.
Allocator(ROOT) 0/0/9577600/34359738368 (res/actual/peak/limit)
 child allocators: 8
 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 
(res/actual/peak/limit)
 child allocators: 0
 ledgers: 0
 reservations: 0
 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 
(res/actual/peak/limit)
 child allocators: 0
 ledgers: 0
 reservations: 0
 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 
(res/actual/peak/limit)
 child allocators: 0
 ledgers: 0
 reservations: 0
 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 
(res/actual/peak/limit)
 child allocators: 0
 ledgers: 0
 reservations: 0
 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 
(res/actual/peak/limit)
 child allocators: 0
 ledgers: 0
 reservations: 0
 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 
(res/actual/peak/limit)
 child allocators: 0
 ledgers: 0
 reservations: 0
 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 
(res/actual/peak/limit)
 child allocators: 0
 ledgers: 0
 reservations: 0
 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 
(res/actual/peak/limit)
 child allocators: 0
 ledgers: 0
 reservations: 0
 ledgers: 0
 reservations: 0

at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:496) 
~[drill-memory-base-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81) 
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69) 
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at 
org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:259) 
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81) 
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69) 
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:263) 
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:363) 
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] INFO 
o.apache.drill.exec.server.Drillbit - Shutdown completed (1095 ms).


{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6427) outputBatchSize is missing from the DEBUG output for HashJoinBatch operator

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6427:
-

 Summary: outputBatchSize is missing from the DEBUG output for 
HashJoinBatch operator 
 Key: DRILL-6427
 URL: https://issues.apache.org/jira/browse/DRILL-6427
 Project: Apache Drill
  Issue Type: Bug
Reporter: Khurram Faraaz
Assignee: Padma Penumarthy


Drill 1.14.0-SNAPSHOT commit : f99d1f1323c0a5bac99842d6283d3025f3cb527f 
outputBatchSize is missing from the DEBUG output for HashJoinBatch operator 

Query used in test,
{noformat}
select count(*) from `twovarchar_asc_128MB.parquet` t1, 
`twovarchar_asc_16MB.parquet` t2 WHERE t1.Varbinaryvalue = t2.Varbinaryvalue
{noformat} 
 
Snippet from drillbit.log 
{noformat}
2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG 
o.a.d.e.p.impl.join.HashJoinBatch - left input: batch count : 1, avg batch 
bytes : 90145920, avg row bytes : 8166, record count : 11040
2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG 
o.a.d.e.p.impl.join.HashJoinBatch - right input: batch count : 1, avg batch 
bytes : 10567808, avg row bytes : 7506, record count : 1408
2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG 
o.a.d.e.p.impl.join.HashJoinBatch - output: batch count : 166, avg batch bytes 
: 15951453, avg row bytes : 15672, record count : 168960
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] New Committer: Sorabh Hamirwasia

[Khurram Faraaz]

Congratulations Sorabh!


From: Andries Engelbrecht 
Sent: Monday, April 30, 2018 11:04:11 AM
To: dev@drill.apache.org
Subject: Re: [ANNOUNCE] New Committer: Sorabh Hamirwasia

Congrats Sorabh!!!

--Andries

On 4/30/18, 8:35 AM, "Aman Sinha"  wrote:

The Project Management Committee (PMC) for Apache Drill has invited Sorabh
Hamirwasia  to become a committer, and we are pleased to announce that he
has accepted.

Over the last 1 1/2 years Sorabh's contributions have been in a few
different areas. He took
the lead in designing and implementing network encryption support for
Drill. He has contributed
to the web server and UI side.  More recently, he is involved in design and
implementation of the lateral join operator.

Welcome Sorabh, and thank you for your contributions.  Keep up the good
work !

-Aman
(on behalf of Drill PMC)

[jira] [Created] (DRILL-6330) TPC-DS query 95 failed to plan

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6330:
-

 Summary: TPC-DS query 95 failed to plan
 Key: DRILL-6330
 URL: https://issues.apache.org/jira/browse/DRILL-6330
 Project: Apache Drill
  Issue Type: Bug
  Components: Query Planning  Optimization
Affects Versions: 1.14.0
Reporter: Khurram Faraaz


TPC-DS query 95 planned and executed successfully on Drill 1.10.0 (git.commit 
id : bbcf4b76), it fails to plan on 1.14.0 master on 4 node cluster, SF1 
parquet data.

TPC-DS query 95 fails to plan on Drill 1.14.0

git.commit.id.abbrev=da24113
git.branch=master

{noformat}
[root@qa102-45 bin]# ./sqlline -u 
"jdbc:drill:schema=dfs.tpcds_sf1_parquet_views;drillbit=" -f 
~/tpcds_drill_1_14_0/query95.sql
1/2 WITH ws_wh AS
(
SELECT ws1.ws_order_number,
ws1.ws_warehouse_sk wh1,
ws2.ws_warehouse_sk wh2
FROM web_sales ws1,
web_sales ws2
WHERE ws1.ws_order_number = ws2.ws_order_number
AND ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk)
SELECT
Count(DISTINCT ws_order_number) AS `order count` ,
Sum(ws_ext_ship_cost) AS `total shipping cost` ,
Sum(ws_net_profit) AS `total net profit`
FROM web_sales ws1 ,
date_dim ,
customer_address ,
web_site
WHERE d_date BETWEEN '2000-4-01' AND (
Cast('2000-4-01' AS DATE) + INTERVAL '60' day)
AND ws1.ws_ship_date_sk = d_date_sk
AND ws1.ws_ship_addr_sk = ca_address_sk
AND ca_state = 'IN'
AND ws1.ws_web_site_sk = web_site_sk
AND web_company_name = 'pri'
AND ws1.ws_order_number IN
(
SELECT ws_order_number
FROM ws_wh)
AND ws1.ws_order_number IN
(
SELECT wr_order_number
FROM web_returns,
ws_wh
WHERE wr_order_number = ws_wh.ws_order_number)
ORDER BY count(DISTINCT ws_order_number)
LIMIT 100;

Error: UNSUPPORTED_OPERATION ERROR: This query cannot be planned possibly due 
to either a cartesian join or an inequality join


[Error Id: d2ec3faf-1253-492e-bc56-80689e62335c on qa102-45.qa.lab:31010]

(org.apache.drill.exec.work.foreman.UnsupportedRelOperatorException) This query 
cannot be planned possibly due to either a cartesian join or an inequality join
 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToPrel():450
 org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():179
 org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():146
 org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():84
 org.apache.drill.exec.work.foreman.Foreman.runSQL():567
 org.apache.drill.exec.work.foreman.Foreman.run():264
 java.util.concurrent.ThreadPoolExecutor.runWorker():1149
 java.util.concurrent.ThreadPoolExecutor$Worker.run():624
 java.lang.Thread.run():748 (state=,code=0)
{noformat}


>From the profile file under log/profiles directory, for TPC-DS query 95.

{noformat}

{"id":\{"part1":2679297541583898541,"part2":5048058602276109542},"type":1,"start":1523661076935,"end":1523661077389,"query":"WITH
 ws_wh AS \n( \nSELECT ws1.ws_order_number, \nws1.ws_warehouse_sk wh1, 
\nws2.ws_warehouse_sk wh2 \nFROM web_sales ws1, \nweb_sales ws2 \nWHERE 
ws1.ws_order_number = ws2.ws_order_number \nAND ws1.ws_warehouse_sk <> 
ws2.ws_warehouse_sk) \nSELECT \nCount(DISTINCT ws_order_number) AS `order 
count` , \nSum(ws_ext_ship_cost) AS `total shipping cost` , 
\nSum(ws_net_profit) AS `total net profit` \nFROM web_sales ws1 , \ndate_dim , 
\ncustomer_address , \nweb_site \nWHERE d_date BETWEEN '2000-4-01' AND ( 
\nCast('2000-4-01' AS DATE) + INTERVAL '60' day) \nAND ws1.ws_ship_date_sk = 
d_date_sk \nAND ws1.ws_ship_addr_sk = ca_address_sk \nAND ca_state = 'IN' \nAND 
ws1.ws_web_site_sk = web_site_sk \nAND web_company_name = 'pri' \nAND 
ws1.ws_order_number IN \n( \nSELECT ws_order_number \nFROM ws_wh) \nAND 
ws1.ws_order_number IN \n( \nSELECT wr_order_number \nFROM web_returns, \nws_wh 
\nWHERE wr_order_number = ws_wh.ws_order_number) \nORDER BY count(DISTINCT 
ws_order_number) \nLIMIT 
100","foreman":\{"address":"qa102-45.qa.lab","userPort":31010,"controlPort":31011,"dataPort":31012,"version":"1.14.0-SNAPSHOT","state":0},"state":4,"totalFragments":0,"finishedFragments":0,"user":"anonymous","error":"UNSUPPORTED_OPERATION
 ERROR: This query cannot be planned possibly due to either a cartesian join or 
an inequality join\n\n","verboseError":"UNSUPPORTED_OPERATION ERROR: This query 
cannot be planned possibly due to either a cartesian join or an inequality 
join\n\n\n\n 
(org.apache.drill.exec.work.foreman.UnsupportedRelOperatorException) This query 
cannot be planned possibly due to either a cartesian join or an inequality 
join\n 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToPrel():450\n
 org.apache.drill.exec.planner.sql.handlers.Defa

[jira] [Created] (DRILL-6329) TPC-DS Query 66 failed due to OOM

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6329:
-

 Summary: TPC-DS Query 66 failed due to OOM
 Key: DRILL-6329
 URL: https://issues.apache.org/jira/browse/DRILL-6329
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
 Attachments: 252f0f20-2774-43d7-ec31-911ee0f5f330.sys.drill, 
TPCDS_Query_66.sql, TPCDS_Query_66_PLAN.txt

TPC-DS Query 66 failed after 27 minutes on Drill 1.14.0 on a 4 node cluster 
against SF1 parquet data (dfs.tpcds_sf1_parquet_views). Query 66 and the query 
profile and the query plan are attached here.

This seems to be a regression, the same query worked fine on 1.10.0

On Drill 1.10.0 ( git.commit id : bbcf4b76) => 9.026 seconds (completed 
successfully).
On Drill 1.14.0 ( git.commit.id.abbrev=da24113 ) query 66 failed after running 
for 27 minutes, due to OutOfMemoryException

Stack trace from sqlline console, no stack trace was written to drillbit.log
{noformat}
Error: RESOURCE ERROR: One or more nodes ran out of memory while executing the 
query.

Too little memory available
Fragment 2:0

[Error Id: 5636a939-a318-4b59-b3e8-9eb93f6b82f3 on qa102-45.qa.lab:31010]

(org.apache.drill.exec.exception.OutOfMemoryException) Too little memory 
available
 org.apache.drill.exec.test.generated.HashAggregatorGen7120.delayedSetup():409
 org.apache.drill.exec.test.generated.HashAggregatorGen7120.doWork():579
 org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():176
 org.apache.drill.exec.record.AbstractRecordBatch.next():164
 org.apache.drill.exec.record.AbstractRecordBatch.next():119
 org.apache.drill.exec.record.AbstractRecordBatch.next():109
 org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
 org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():134
 org.apache.drill.exec.record.AbstractRecordBatch.next():164
 org.apache.drill.exec.physical.impl.BaseRootExec.next():105
 
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():93
 org.apache.drill.exec.physical.impl.BaseRootExec.next():95
 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():292
 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():279
 java.security.AccessController.doPrivileged():-2
 javax.security.auth.Subject.doAs():422
 org.apache.hadoop.security.UserGroupInformation.doAs():1595
 org.apache.drill.exec.work.fragment.FragmentExecutor.run():279
 org.apache.drill.common.SelfCleaningRunnable.run():38
 java.util.concurrent.ThreadPoolExecutor.runWorker():1149
 java.util.concurrent.ThreadPoolExecutor$Worker.run():624
 java.lang.Thread.run():748 (state=,code=0)
java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory 
while executing the query.

Too little memory available
Fragment 2:0

[Error Id: 5636a939-a318-4b59-b3e8-9eb93f6b82f3 on qa102-45.qa.lab:31010]

(org.apache.drill.exec.exception.OutOfMemoryException) Too little memory 
available
 org.apache.drill.exec.test.generated.HashAggregatorGen7120.delayedSetup():409
 org.apache.drill.exec.test.generated.HashAggregatorGen7120.doWork():579
 org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():176
 org.apache.drill.exec.record.AbstractRecordBatch.next():164
 org.apache.drill.exec.record.AbstractRecordBatch.next():119
 org.apache.drill.exec.record.AbstractRecordBatch.next():109
 org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
 org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():134
 org.apache.drill.exec.record.AbstractRecordBatch.next():164
 org.apache.drill.exec.physical.impl.BaseRootExec.next():105
 
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():93
 org.apache.drill.exec.physical.impl.BaseRootExec.next():95
 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():292
 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():279
 java.security.AccessController.doPrivileged():-2
 javax.security.auth.Subject.doAs():422
 org.apache.hadoop.security.UserGroupInformation.doAs():1595
 org.apache.drill.exec.work.fragment.FragmentExecutor.run():279
 org.apache.drill.common.SelfCleaningRunnable.run():38
 java.util.concurrent.ThreadPoolExecutor.runWorker():1149
 java.util.concurrent.ThreadPoolExecutor$Worker.run():624
 java.lang.Thread.run():748
 
...
Caused by: org.apache.drill.common.exceptions.UserRemoteException: RESOURCE 
ERROR: One or more nodes ran out of memory while executing the query.

Too little memory available
Fragment 2:0

[Error Id: 5636a939-a318-4b59-b3e8-9eb93f6b82f3 on qa102-45.qa.lab:31010]

(org.apache.drill.exec.exception.OutOfMemoryException) Too little memory 
available
 org.apache.drill.exec.test.generated.HashAggregatorGen7120.delayedSetup():409
 org.apache.drill.exec.test.generated.HashAggregatorGen7120.doWork():

[jira] [Created] (DRILL-6308) Exception handling - OOM

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6308:
-

 Summary: Exception handling - OOM
 Key: DRILL-6308
 URL: https://issues.apache.org/jira/browse/DRILL-6308
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Assignee: Sorabh Hamirwasia


For RpcHandlers in Drill, when there is an OutOfMemory condition ,then all the 
handlers call OutOfMemoryHandler with the caught exception. Every channel, 
except for DataServer the default instance of MemoryHandler is used which 
throws UnsupportedOperationException. This is misleading since actual exception 
is for OutOfMemory not UnsupportedOperationException.

 

Here is an example where it is not handled the right way.
{noformat}
Caused by: java.lang.UnsupportedOperationException: null
at 
org.apache.drill.exec.rpc.OutOfMemoryHandler$1.handle(OutOfMemoryHandler.java:25)
 ~[drill-rpc-1.13.0-mapr.jar:1.13.0-mapr]
at 
org.apache.drill.exec.rpc.SaslEncryptionHandler.encode(SaslEncryptionHandler.java:170)
 ~[drill-rpc-1.13.0-mapr.jar:1.13.0-mapr]
at 
org.apache.drill.exec.rpc.SaslEncryptionHandler.encode(SaslEncryptionHandler.java:44)
 ~[drill-rpc-1.13.0-mapr.jar:1.13.0-mapr]
at 
io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88)
 [netty-codec-4.0.48.Final.jar:4.0.48.Final]
... 25 common frames omitted{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6302) NPE in Drillbit.java in close method

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6302:
-

 Summary: NPE in Drillbit.java in close method
 Key: DRILL-6302
 URL: https://issues.apache.org/jira/browse/DRILL-6302
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.13.0
 Environment: git.commit.id=bb07ebbb9ba8742f44689f8bd8efb5853c5edea0
Reporter: Khurram Faraaz
Assignee: Sorabh Hamirwasia


registrationHandle in the close method of Drillbit.java (line 228) is null, 
which causes an NPE and as a result the server does not close.

[https://github.com/mapr/private-drill/blob/drill-1.12.0-mapr/exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java#L228]

registrationHandle = coord.update(registrationHandle, State.QUIESCENT);

Stack trace from drillbit.log
{noformat}
/opt/mapr/drill/drill-1.12.0/logs/drillbit.out
{noformat}
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was 
removed in 8.0
Exception in thread "main" java.lang.NullPointerException
at 
org.apache.drill.exec.coord.zk.ZKClusterCoordinator.update(ZKClusterCoordinator.java:223)
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:228)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:401)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:372)
at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:368)
{noformat}{noformat}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6285) Failure while attempting to fail rpc response.

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6285:
-

 Summary: Failure while attempting to fail rpc response.
 Key: DRILL-6285
 URL: https://issues.apache.org/jira/browse/DRILL-6285
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - RPC
Affects Versions: 1.14.0
Reporter: Khurram Faraaz


When a query is marked as Canceled on Web UI, there is this Exception in 
drillbit.log

Drill 1.13.0 commit cac2882d5a9e22fbc251e4caf622fe30242ad557
{noformat}
2018-03-21 15:35:43,198 [UserServer-1] INFO o.a.d.e.w.fragment.FragmentExecutor 
- 254d2796-7744-cc2a-d77e-5ec030b04211:0:0: State change requested RUNNING --> 
FAILED
2018-03-21 15:35:43,198 [254d2796-7744-cc2a-d77e-5ec030b04211:frag:0:0] INFO 
o.a.d.e.w.fragment.FragmentExecutor - 254d2796-7744-cc2a-d77e-5ec030b04211:0:0: 
State change requested FAILED --> FAILED
2018-03-21 15:35:43,199 [254d2796-7744-cc2a-d77e-5ec030b04211:frag:0:0] INFO 
o.a.d.e.w.fragment.FragmentExecutor - 254d2796-7744-cc2a-d77e-5ec030b04211:0:0: 
State change requested FAILED --> FAILED
2018-03-21 15:35:43,202 [254d2796-7744-cc2a-d77e-5ec030b04211:frag:0:0] INFO 
o.a.d.e.w.fragment.FragmentExecutor - 254d2796-7744-cc2a-d77e-5ec030b04211:0:0: 
State change requested FAILED --> FAILED
2018-03-21 15:35:43,202 [UserServer-1] WARN 
o.apache.drill.exec.rpc.RequestIdMap - Failure while attempting to fail rpc 
response.
java.lang.IllegalArgumentException: Self-suppression not permitted
at java.lang.Throwable.addSuppressed(Throwable.java:1043) ~[na:1.8.0_161]
at 
org.apache.drill.common.DeferredException.addException(DeferredException.java:88)
 ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at 
org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:97)
 ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:412)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.access$700(FragmentExecutor.java:55)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$ExecutorStateImpl.fail(FragmentExecutor.java:426)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at 
org.apache.drill.exec.ops.FragmentContextImpl.fail(FragmentContextImpl.java:233)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at 
org.apache.drill.exec.ops.FragmentContextImpl$1.accept(FragmentContextImpl.java:100)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at 
org.apache.drill.exec.ops.FragmentContextImpl$1.accept(FragmentContextImpl.java:97)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at org.apache.drill.exec.ops.StatusHandler.failed(StatusHandler.java:42) 
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at 
org.apache.drill.exec.rpc.RequestIdMap$RpcListener.setException(RequestIdMap.java:139)
 ~[drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at 
org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:76)
 [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at 
org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:66)
 [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at com.carrotsearch.hppc.IntObjectHashMap.forEach(IntObjectHashMap.java:692) 
[hppc-0.7.1.jar:na]
at org.apache.drill.exec.rpc.RequestIdMap.channelClosed(RequestIdMap.java:62) 
[drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at 
org.apache.drill.exec.rpc.AbstractRemoteConnection.channelClosed(AbstractRemoteConnection.java:192)
 [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at 
org.apache.drill.exec.rpc.AbstractServerConnection.channelClosed(AbstractServerConnection.java:165)
 [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at 
org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:167)
 [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at 
org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:144)
 [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at 
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
 [netty-common-4.0.48.Final.jar:4.0.48.Final]
at 
io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
 [netty-common-4.0.48.Final.jar:4.0.48.Final]
at 
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
 [netty-common-4.0.48.Final.jar:4.0.48.Final]
at 
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
 [netty-common-4.0.48.Final.jar:4.0.48.Final]
at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) 
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at 
io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
 [netty-transport-4.0.48.Final.jar:4.0.48.Final]
at 
io.netty.channel.Abs

Re: [VOTE] Apache Drill release 1.13.0 - RC0

[Khurram Faraaz]

Built from source and deployed binaries on a 4 node cluster.

Ran queries from sqlline and Web UI.

Performed basic sanity tests on the Web UI.


Looks good. (non-binding).


Thanks,

Khurram



From: Sorabh Hamirwasia 
Sent: Thursday, March 15, 2018 3:31:53 PM
To: dev@drill.apache.org
Subject: Re: [VOTE] Apache Drill release 1.13.0 - RC0

  *   Downloaded binary tarball from [2] and deployed 2 node cluster
  *   Ran some basic queries using sqlline and Web UI with and without security 
enabled
  *   Verified user to bit secure connections using Plain/Kerberos
  *   Verified bit to bit secure connections using Kerberos.
 *   During testing I found an issue that local control message is still 
creating a connection, but is not related to this release. The issue is a 
regression because of state introduced in Drillbit endpoint as part of shutdown 
feature in 1.12. I have opened 
DRILL-6255
 for this issue with details in it.
  *   Verified SPNEGO and FORM authentication for Web UI
  *   Ran and verify queries against sys.connections table
  *   Built C++ client on linux box using the source tarball from [2] and ran 
queries to secure and unsecure Drillbit


LGTM +1 (non-binding)


Thanks,
Sorabh


From: Vitalii Diravka 
Sent: Thursday, March 15, 2018 11:12 AM
To: dev@drill.apache.org
Subject: Re: [VOTE] Apache Drill release 1.13.0 - RC0

* Downloaded sources tarball from [2].
Ran drill in embedded mode on local debian machine. Ran tpch queries with
joins, group by, order by, limit, order by with limit statements.
Looked through logs - looks good.
* Build drill for [4] with MapR profile. Ran driillbit in distributed mode
on centos VM with MapR core. Ran queries for Hive 1.2, ran queries for Hive
2.1 (transactional and non-transactional tables). Connected to this
drillbit from remote machine via JDBC with java programm and with SQuirrel
using different drivers (prebuild "drill-jdbc-all-1.12.0.jar" from [2]
tarball, the driver after build [4] sources with default and MapR profiles
too) and ran a simple query. Ran the same with enabled custom
authentication (jdbc driver which is build with MapR profile works good
too).
In the process of testing jdbc connection I found the issue - DRILL-6251.
It is a regression. I have described the case in Jira. But I suppose isn't
critical for current Drill release.
* All unit test were passed for [4]. Total time on my machine was: 42:10 min

+1 (non-binding)

Kind regards
Vitalii

On Thu, Mar 15, 2018 at 4:49 PM, Vova Vysotskyi  wrote:

> - Downloaded source tar at [2], ran unit tests and all tests are passed.
> - Downloaded built tar at [2], submitted several TPCH queries from UI,
> checked that profiles are displayed correctly.
> - Connected from SQuirrel, ran several queries; ran queries from a java
> application, no issues were found.
>
> +1 (non-binding)
>
>
> 2018-03-15 13:19 GMT+02:00 Arina Yelchiyeva :
>
> >  - Built from the source [4] on Linux, run unit test.
> > - Downloaded the binary tarball [2], untarred and ran Drill in embedded
> > mode on Windows.
> > - Ran sample queries, checked system tables, profiles on Web UI, also
> logs
> > and index page.
> > - Created persistent and temporary tables, loaded custom UDFs.
> >
> > +1 (binding)
> >
> > Kind regards
> > Arina
> >
> > On Thu, Mar 15, 2018 at 1:39 AM, Aman Sinha 
> wrote:
> >
> > > - Downloaded the source tarball from [2] on my Linux VM, built and ran
> > the
> > > unit tests successfully
> > > - Downloaded the binary tarball onto my Macbook, untarred and ran Drill
> > in
> > > embedded mode
> > > - Ran several queries  against a TPC-DS SF1 data set, including CTAS
> > > statements with PARTITION BY and ran a few partition pruning queries
> > > - Tested query cancellation by cancelling a query that was taking long
> > time
> > > due to expanding join
> > > - Examined the run-time query profiles of these queries with and
> without
> > > parallelism.
> > > - Checked the maven artifacts on [3].
> > >
> > >  - Found one reference to JDK 7 : README.md says 'JDK 7' in the
> > > Prerequisites.  Ideally, this should be changed to JDK 8
> > >
> > > Overall, LGTM  +1 (binding)
> > >
> > >
> > > On Tue, Mar 13, 2018 at 3:58 AM, Parth Chandra 
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to propose the first release candidate (RC0) of Apache
> Drill,
> > > > version 1.13.0.
> > > >
> > > > The release candidate covers a total of 113 resolved JIRAs [1].
> Thanks
> > > > to everyone
> > > > who contributed to this release.
> > > >
> > > > The tarball artifacts are hosted at 

[jira] [Created] (DRILL-6254) IllegalArgumentException: the requested size must be non-negative

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6254:
-

 Summary: IllegalArgumentException: the requested size must be 
non-negative
 Key: DRILL-6254
 URL: https://issues.apache.org/jira/browse/DRILL-6254
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.13.0
Reporter: Khurram Faraaz
Assignee: Padma Penumarthy
 Attachments: genAllTypesJSN.py

Flatten query fails due to IllegalArgumentException: the requested size must be 
non-negative.

Script to generate JSON data file is attached here.

{noformat}
0: jdbc:drill:schema=dfs.tmp> CREATE TABLE tbl_all_types_jsn_to_parquet AS 
. . . . . . . . . . . . . . > SELECT
. . . . . . . . . . . . . . > CAST( col_int AS INT) col_int, 
. . . . . . . . . . . . . . > CAST( col_bigint AS BIGINT) col_bigint, 
. . . . . . . . . . . . . . > CAST( col_char AS CHAR(10)) col_char, 
. . . . . . . . . . . . . . > CAST( col_fxdln_str AS VARCHAR(256)) 
col_fxdln_str, 
. . . . . . . . . . . . . . > CAST( col_varln_str AS VARCHAR(256)) 
col_varln_str, 
. . . . . . . . . . . . . . > CAST( col_float AS FLOAT) col_float, 
. . . . . . . . . . . . . . > CAST( col_double AS DOUBLE PRECISION) col_double, 
. . . . . . . . . . . . . . > CAST( col_date AS DATE) col_date, 
. . . . . . . . . . . . . . > CAST( col_time AS TIME) col_time, 
. . . . . . . . . . . . . . > CAST( col_tmstmp AS TIMESTAMP) col_tmstmp, 
. . . . . . . . . . . . . . > CAST( col_boolean AS BOOLEAN) col_boolean, 
. . . . . . . . . . . . . . > col_binary, 
. . . . . . . . . . . . . . > array_of_ints from `all_supported_types.json`;
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 9 |
+---++
1 row selected (0.29 seconds)
{noformat}

Reset all options and set slice_target=1
alter system reset all;
alter system set `planner.slice_target`=1;

output_batch_size was set to its default value
drill.exec.memory.operator.output_batch_size = 16777216

 

{noformat}
select *, flatten(array_of_ints) from tbl_all_types_jsn_to_parquet;

Error: SYSTEM ERROR: IllegalArgumentException: the requested size must be 
non-negative

Fragment 0:0

[Error Id: 480bae96-ae89-45a7-b937-011c0f87c14d on qa102-45.qa.lab:31010] 
(state=,code=0)
0: jdbc:drill:schema=dfs.tmp>
{noformat}

Stack trace from drillbit.log

{noformat}
2018-03-15 12:19:43,916 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO 
o.a.drill.exec.work.foreman.Foreman - Query text for query id 
255538af-bcd5-98ee-32e0-68d98fc4a6fa: select *, flatten(array_of_ints) from 
tbl_all_types_jsn_to_parquet
2018-03-15 12:19:43,952 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO 
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2018-03-15 12:19:43,953 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO 
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2018-03-15 12:19:43,966 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO 
o.a.d.exec.store.parquet.Metadata - Took 0 ms to get file statuses
2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO 
o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of 1 
using 1 threads. Time: 2ms total, 2.927366ms avg, 2ms max.
2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO 
o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of 1 
using 1 threads. Earliest start: 2.829000 μs, Latest start: 2.829000 μs, 
Average start: 2.829000 μs .
2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO 
o.a.d.exec.store.parquet.Metadata - Took 3 ms to read file metadata
2018-03-15 12:19:44,000 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO 
o.a.d.e.w.fragment.FragmentExecutor - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: 
State change requested AWAITING_ALLOCATION --> RUNNING
2018-03-15 12:19:44,000 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO 
o.a.d.e.w.f.FragmentStatusReporter - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: 
State to report: RUNNING
2018-03-15 12:19:44,905 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO 
o.a.d.e.w.fragment.FragmentExecutor - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: 
State change requested RUNNING --> FAILED
2018-03-15 12:19:44,927 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO 
o.a.d.e.w.fragment.FragmentExecutor - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: 
State change requested FAILED --> FINISHED
2018-03-15 12:19:44,928 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] ERROR 
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalArgumentException: 
the requested size must be non-negative

Fragment 0:0

[Error Id: 480bae96-ae89-45a7-b937-011c0f87c14d on qa102-45.qa.lab:31010]
org.a

Re: [ANNOUNCE] New Committer: Volodymyr Vysotskyi

[Khurram Faraaz]

Congratulations Volodymyr!


From: Arina Ielchiieva 
Sent: Thursday, March 15, 2018 10:16:51 AM
To: dev@drill.apache.org
Subject: [ANNOUNCE] New Committer: Volodymyr Vysotskyi

The Project Management Committee (PMC) for Apache Drill has
invited Volodymyr Vysotskyi to become a committer, and we are pleased to
announce that he has accepted.

Volodymyr has been contributing for Drill over a year. He contributed in
different areas, including code generation, json processing, function
implementations.
Also he actively participated in Calcite rebase and showed profound knowledge
in planning area.
Currently he is working on decimal's enhancement in Drill.

Congratulations Volodymyr and thank you for your contributions!

- Arina
(on behalf of the Apache Drill PMC)

[jira] [Created] (DRILL-6247) Minor fragments remain in CANCELLATION_REQUESTED state after query failure

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6247:
-

 Summary: Minor fragments remain in CANCELLATION_REQUESTED state 
after query failure
 Key: DRILL-6247
 URL: https://issues.apache.org/jira/browse/DRILL-6247
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.13.0
Reporter: Khurram Faraaz
 Attachments: cancellation_requested_march_14.png, 
drillbit_snippet.log, jstack_cancellation_requested.txt

Once a query fails, in this case due to an OOM in RPC we see many minor 
fragments are reported to be in CANCELLATION_REQUESTED state on Web UI after 
query has failed. The problem is reproducible. drillbit.log for this failure 
and jstack output are attached here.

To reproduce the problem on a 4 node cluster.

alter system reset all;
alter system set `planner.slice_target`=1;

Failing query => SELECT * , FLATTEN(arr) FROM many_json_files;

Drill 1.13.0-SNAPSHOT, commit id: 766315ea17377199897d685ab801edd38394fe01

Stack trace from output of jstack, fragment 0:0 is reported to be in 
CANCELLATION_REQUESTED state on Drill Web UI

jstack -l 13488 > jstack_DRILL_6235.txt

{noformat}
"25569e98-10f9-2fe2-9dec-0a42f3ad45fa:frag:0:0" #87 daemon prio=10 os_prio=0 
tid=0x7f9d01374360 nid=0x2ff5 waiting on condition [0x7f9cd5536000]
 java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for <0x0007a388b300> (a 
org.apache.drill.exec.rpc.ResettableBarrier$InternalSynchronizer)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
 at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
 at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
 at org.apache.drill.exec.rpc.ResettableBarrier.await(ResettableBarrier.java:70)
 at 
org.apache.drill.exec.rpc.AbstractRemoteConnection$WriteManager.waitForWritable(AbstractRemoteConnection.java:114)
 at 
org.apache.drill.exec.rpc.AbstractRemoteConnection.blockOnNotWritable(AbstractRemoteConnection.java:76)
 at org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:108)
 at 
org.apache.drill.exec.rpc.user.UserServer$BitToUserConnection.sendData(UserServer.java:275)
 at 
org.apache.drill.exec.ops.AccountingUserConnection.sendData(AccountingUserConnection.java:42)
 at 
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:120)
 at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95)
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:233)
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:226)
 at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
{noformat}

Stack trace from drillbit.log

{noformat}
2018-03-14 10:52:44,545 [25569e98-10f9-2fe2-9dec-0a42f3ad45fa:frag:1:49] INFO 
o.a.d.e.w.fragment.FragmentExecutor - User Error Occurred: One or more nodes 
ran out of memory while executing the query. (Failure allocating buffer.)
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more 
nodes ran out of memory while executing the query.

Failure allocating buffer.

[Error Id: b83884df-af31-411a-9b28-554c294a7357 ]
 at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
 ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:243)
 [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[na:1.8.0_161]
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[na:1.8.0_161]
 at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Failure 
allocating buffer.
 at 
io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:67)
 ~[drill-memory-base-1.13.0-SNA

[jira] [Created] (DRILL-6235) Flatten query leads to out of memory in RPC layer.

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6235:
-

 Summary: Flatten query leads to out of memory in RPC layer.
 Key: DRILL-6235
 URL: https://issues.apache.org/jira/browse/DRILL-6235
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Khurram Faraaz
Assignee: Padma Penumarthy
 Attachments: 25593391-512d-23ab-7c84-3651006931e2.sys.drill

Flatten query leads to out of memory in RPC layer. Query profile is attached 
here.

Total number of JSON files = 4095
Each JSON file has nine rows
And each row in the JSON has an array with 1024 integer values, and there are 
other string values outside of the array.
Two major fragments and eighty eight minor fragments were created

On a 4 node CentOS cluster
number of CPU cores
[root@qa102-45 ~]# grep -c ^processor /proc/cpuinfo
32

Details of memory
{noformat}
0: jdbc:drill:schema=dfs.tmp> select * from sys.memory;
+--++---+-+-+-+-+
| hostname | user_port | heap_current | heap_max | direct_current | 
jvm_direct_current | direct_max |
+--++---+-+-+-+-+
| qa102-45.qa.lab | 31010 | 1130364912 | 4294967296 | 0 | 170528 | 8589934592 |
| qa102-47.qa.lab | 31010 | 171823104 | 4294967296 | 0 | 21912 | 8589934592 |
| qa102-48.qa.lab | 31010 | 201326576 | 4294967296 | 0 | 21912 | 8589934592 |
| qa102-46.qa.lab | 31010 | 214780896 | 4294967296 | 0 | 21912 | 8589934592 |
+--++---+-+-+-+-+
4 rows selected (0.166 seconds)
{noformat}

Reset all options and set slice_target=1
alter system reset all;
alter system set `planner.slice_target`=1;

{noformat}
SELECT * , FLATTEN(arr) FROM many_json_files
...

Error: RESOURCE ERROR: One or more nodes ran out of memory while executing the 
query.

Failure allocating buffer.
Fragment 1:38

[Error Id: cf4fd273-d8a2-45e8-8d72-15c738e53b0f on qa102-45.qa.lab:31010] 
(state=,code=0)
{noformat}

Stack trace from drillbit.log fir above failing query.
{noformat}
2018-03-12 11:52:33,849 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO 
o.a.d.e.w.fragment.FragmentExecutor - 25593391-512d-23ab-7c84-3651006931e2:0:0: 
State change requested AWAITING_ALLOCATION --> RUNNING
2018-03-12 11:52:33,849 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO 
o.a.d.e.w.f.FragmentStatusReporter - 25593391-512d-23ab-7c84-3651006931e2:0:0: 
State to report: RUNNING
2018-03-12 11:52:33,854 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO 
o.a.d.e.c.ClassCompilerSelector - Java compiler policy: DEFAULT, Debug option: 
true
2018-03-12 11:52:35,929 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder 
- Failure allocating buffer on incoming stream due to memory limits. Current 
Allocation: 92340224.
2018-03-12 11:52:35,929 [BitServer-3] WARN o.a.d.exec.rpc.ProtobufLengthDecoder 
- Failure allocating buffer on incoming stream due to memory limits. Current 
Allocation: 92340224.
2018-03-12 11:52:35,930 [BitServer-3] ERROR o.a.drill.exec.rpc.data.DataServer 
- Out of memory in RPC layer.
2018-03-12 11:52:35,930 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer 
- Out of memory in RPC layer.
2018-03-12 11:52:35,930 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder 
- Failure allocating buffer on incoming stream due to memory limits. Current 
Allocation: 83886080.
2018-03-12 11:52:35,930 [BitServer-3] WARN o.a.d.exec.rpc.ProtobufLengthDecoder 
- Failure allocating buffer on incoming stream due to memory limits. Current 
Allocation: 83886080.
2018-03-12 11:52:35,930 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer 
- Out of memory in RPC layer.
2018-03-12 11:52:35,930 [BitServer-3] ERROR o.a.drill.exec.rpc.data.DataServer 
- Out of memory in RPC layer.
2018-03-12 11:52:35,931 [BitServer-3] WARN o.a.d.exec.rpc.ProtobufLengthDecoder 
- Failure allocating buffer on incoming stream due to memory limits. Current 
Allocation: 83886080.
2018-03-12 11:52:35,931 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder 
- Failure allocating buffer on incoming stream due to memory limits. Current 
Allocation: 83886080.
2018-03-12 11:52:35,931 [BitServer-3] ERROR o.a.drill.exec.rpc.data.DataServer 
- Out of memory in RPC layer.
2018-03-12 11:52:35,931 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer 
- Out of memory in RPC layer.
...
...
2018-03-12 11:52:35,939 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder 
- Failure allocating buffer on incoming stream due to memory limits. Current 
Allocation: 67174400.
2018-03-12 11:52:35,939 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer 
- Out of memory in RPC layer.
2018-03-12 11:52:35,939 [BitServer-2

[jira] [Created] (DRILL-6229) memory leak when output_batch_size is set to 4096

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6229:
-

 Summary: memory leak when output_batch_size is set to 4096
 Key: DRILL-6229
 URL: https://issues.apache.org/jira/browse/DRILL-6229
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Khurram Faraaz


When drill.exec.memory.operator.output_batch_size` = 4096 (4KB), we 
continuously see in each test run, the memory leak when tests are run from test 
framework.

Drill 1.13.0 git.commit.id=766315ea17377199897d685ab801edd38394fe01

alter system set `drill.exec.memory.operator.output_batch_size` = 4096;
To run flatten tests from drill test framework
bin/run_tests -s Functional/flatten_operators -g functional -d -t 500 -n 40

The below two tests time out and we see the memory leak
Timeouts:
/root/drillTEST_FRAMEWORK/drill-test-framework/framework/resources/Functional/flatten_operators/10rows/filter24.q

Timeouts:
/root/drillTEST_FRAMEWORK/drill-test-framework/framework/resources/Functional/flatten_operators/10rows/filter19.q

oadd.org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: 
Connection ... (user client) closed unexpectedly. Drillbit down?

Stack trace printed in drill test framework log
{noformat}
[Error Id: 54a16569-6555-4e92-aa5f-f76a8c11b31f ]
at 
oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
at 
oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$ChannelClosedHandler$1.operationComplete(QueryResultHandler.java:376)
at 
oadd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at 
oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
at 
oadd.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
at 
oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
at 
oadd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
at 
oadd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
at 
oadd.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1003)
at 
oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:632)
at 
oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:610)
at 
oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:553)
at 
oadd.io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1236)
at 
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618)
at 
oadd.io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:602)
at 
oadd.io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71)
at 
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618)
at 
oadd.io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:602)
at 
oadd.io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73)
at 
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618)
at 
oadd.io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:35)
at 
oadd.io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:607)
at 
oadd.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at 
oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at java.lang.Thread.run(Thread.java:748)
Channel is closed, discarding remaining 908 byte(s) in buffer.
Exception while closing client.
java.lang.RuntimeException: Exception while closing
at 
oadd.org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46)
at oadd.org.apache.drill.exec.client.DrillClient.close(DrillClient.java:459)
at 
org.apache.drill.jdbc.impl.DrillConnectionImpl.closeOrWarn(DrillConnectionImpl.java:810)
at 
org.apache.drill.jdbc.impl.DrillConnectionImpl.cleanup(DrillConnectionImpl.java:823)
at 
org.apache.drill.jdbc.impl.DrillHandler.onConnectionClose(DrillHandler.java:36)
at 
oadd.org.apache.calcite.avatica.AvaticaConnection.close(AvaticaConnection.java:205)
at org.apache.drill.test.framework.ConnectionPool.close(ConnectionPool.java:102)
at org.apache.drill.test.framework.TestDriver.runTests(TestDriver.java:562)
at org.apache.drill.test.framework.TestDriver.main(TestDriver.java:100)
Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory 
leaked: (4198400)
Allocator(ROOT) 0/4096/6713344/28631367680 (res/actual/peak/limit

[jira] [Created] (DRILL-6215) Use prepared statement instead of Statement in JdbcRecordReader class

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6215:
-

 Summary: Use prepared statement instead of Statement in 
JdbcRecordReader class
 Key: DRILL-6215
 URL: https://issues.apache.org/jira/browse/DRILL-6215
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - JDBC
Affects Versions: 1.12.0
Reporter: Khurram Faraaz


Use prepared statement instead of Statement in JdbcRecordReader class, which is 
more efficient and less vulnerable to SQL injection attacks.

Apache Drill 1.13.0-SNAPSHOT, commit : 9073aed67d89e8b2188870d6c812706085c9c41b

Findbugs reports the below bug and suggests that we use prepared statement 
instead of Statement.

{noformat}
In class org.apache.drill.exec.store.jdbc.JdbcRecordReader
In method 
org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(OperatorContext, 
OutputMutator)
At JdbcRecordReader.java:[line 170]
org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(OperatorContext, 
OutputMutator) passes a nonconstant String to an execute method on an SQL 
statement

The method invokes the execute method on an SQL statement with a String that 
seems to be dynamically generated. 
Consider using a prepared statement instead. 
It is more efficient and less vulnerable to SQL injection attacks.

{noformat}

LOC - 
https://github.com/apache/drill/blob/a9ea4ec1c5645ddab4b7aef9ac060ff5f109b696/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcRecordReader.java#L170

{noformat}
To run with findbugs:
mvn clean install -Pfindbugs -DskipTests

Findbugs will wirite the output to finbugsXml.html in the target directory of 
each module. 
For example the java-exec module report is located at: 
./exec/java-exec/target/findbugs/findbugsXml.html
Use 
find . -name "findbugsXml.html"
to locate the files.
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6214) DrillRuntimeException: Unexpected byte 0xfe at position 0 encountered while decoding UTF8 string.

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6214:
-

 Summary: DrillRuntimeException: Unexpected byte 0xfe at position 0 
encountered while decoding UTF8 string.
 Key: DRILL-6214
 URL: https://issues.apache.org/jira/browse/DRILL-6214
 Project: Apache Drill
  Issue Type: Bug
  Components: Functions - Drill
Affects Versions: 1.12.0
Reporter: Khurram Faraaz


binary_string function returns DrillRuntimeException.

Apache Drill 1.13.0-SNAPSHOT, commit id: 
9073aed67d89e8b2188870d6c812706085c9c41b

{noformat}
0: jdbc:drill:schema=dfs.tmp> select 
binary_string(convert_to('abcdefghi','UTF16')) binStr from (values(1));
Error: SYSTEM ERROR: DrillRuntimeException: Unexpected byte 0xfe at position 0 
encountered while decoding UTF8 string.

Fragment 1:0

[Error Id: 6233b0fe-19f2-40ff-9c9e-c4e317ff19d0 on qa102-48.qa.lab:31010] 
(state=,code=0)

Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Unexpected 
byte 0xfe at position 0 encountered while decoding UTF8 string.
 at 
org.apache.drill.exec.expr.fn.impl.StringFunctionUtil.utf8CharLen(StringFunctionUtil.java:88)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at 
org.apache.drill.exec.expr.fn.impl.StringFunctionUtil.getUTF8CharLength(StringFunctionUtil.java:32)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at 
org.apache.drill.exec.test.generated.ProjectorGen346.doSetup(ProjectorTemplate.java:116)
 ~[na:na]
 at 
org.apache.drill.exec.test.generated.ProjectorGen346.setup(ProjectorTemplate.java:101)
 ~[na:na]
 at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchemaFromInput(ProjectRecordBatch.java:490)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:498)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:82)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105) 
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at 
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95) 
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:233)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_131]
 at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_131]
 at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
 ~[hadoop-common-2.7.0-mapr-1707.jar:na]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:226)
 [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 ... 4 common frames omitted
{noformat}

{noformat}
0: jdbc:drill:schema=dfs.tmp> select 
binary_string(convert_to('123456','UTF16')) binStr from (values(1));
Error: SYSTEM ERROR: DrillRuntimeException: Unexpected byte 0xfe at position 0 
encountered while decoding UTF8 string.

Fragment 1:0

[Error Id: 8a1bb0f0-7a9a-46da-b5e9-46e184690e2b on qa102-46.qa.lab:31010] 
(state=,code=0)

Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Unexpected 
byte 0xfe at position 0 encountered while decoding UTF8 string.
 at 
org.apache.drill.exec.expr.fn.impl.StringFunctionUtil.utf8CharLen(StringFunctionUtil.java:88)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at 
org.apache.drill.exec.expr.fn.impl.StringFunctionUtil.getUTF8CharLength(StringFunctionUtil.java:32)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at 
org.apache.drill.exec.test.generated.ProjectorGen272.doSetup(ProjectorTemplate.java:116)
 ~[na:na]
 at 
org.apache.drill.exec.test.generated.ProjectorGen272.setup(ProjectorTemplate.java:101)
 ~[na:na]
 at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchemaFromInput(ProjectRecordBatch.java:490)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
 at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:498)
 ~[drill-java-exec-1.13.0-SNAPSHOT.jar:

Re: [ANNOUNCE] New Committer: Kunal Khatua

[Khurram Faraaz]

Congratulations Kunal.


From: Sorabh Hamirwasia 
Sent: Tuesday, February 27, 2018 8:56:12 AM
To: dev@drill.apache.org
Subject: Re: [ANNOUNCE] New Committer: Kunal Khatua

Congratulations Kunal!


Thanks,
Sorabh


From: Arina Yelchiyeva 
Sent: Tuesday, February 27, 2018 8:44:24 AM
To: dev@drill.apache.org
Subject: Re: [ANNOUNCE] New Committer: Kunal Khatua

Congrats, Kunal!

On Tue, Feb 27, 2018 at 6:42 PM, Aman Sinha  wrote:

> The Project Management Committee (PMC) for Apache Drill has invited Kunal
> Khatua  to become a committer, and we are pleased to announce that he
> has accepted.
>
> Over the last couple of years, Kunal has made substantial contributions to
> the process of creating and interpreting of query profiles, among other
> code contributions. He has led the efforts for Drill performance evaluation
> and benchmarking.  He is a prolific writer on the user mailing list,
> providing detailed responses.
>
> Welcome Kunal, and thank you for your contributions.  Keep up the good
> work !
>
> - Aman
> (on behalf of the Apache Drill PMC)
>

Re: [DISCUSS] 1.13.0 release

[Khurram Faraaz]

I believe we should move to JDK 8 in 1.13.0 release.


Thanks,

Khurram


From: Arina Yelchiyeva 
Sent: Monday, February 26, 2018 8:03:31 AM
To: dev@drill.apache.org
Subject: Re: [DISCUSS] 1.13.0 release

I remember that at the beginning of the year we discussed that Drill on
Yarn, transition to JDK 8 should be in 1.13 release.
Before doing the release we need to make sure all these are done and merged.
Drill on Yarn PR is reviewed but requires some fixes. Not sure about JDK 8.


On Mon, Feb 26, 2018 at 5:50 PM, Parth Chandra  wrote:

> Since there don't appear to be many PRs that folks want merged in, I'm
> thinking of rolling out the release candidate on March 1st. That should
> give folks who want to get stuff in at the last minute enough time. Note
> that I'm on Indian time so I'll be half a day ahead of most other folks.
> Charles, that gives you your  deadline :)
>
> Parth
>
>
>
> On Fri, Feb 23, 2018 at 4:20 PM, Charles Givre  wrote:
>
> > I agree and I’ll try to get the log file PR done for this release.
> >
> > Sent from my iPhone
> >
> > > On Feb 23, 2018, at 00:14, Parth Chandra  wrote:
> > >
> > > Bit of a tepid response from dev; but Aman's approval is all the
> > > encouragement I need to roll out a release :)
> > >
> > > Thoughts on pending PRs?
> > >
> > >
> > >
> > >
> > >> On Thu, Feb 22, 2018 at 9:54 PM, Aman Sinha 
> > wrote:
> > >>
> > >> Agreed...it would be good to get the ball rolling on the 1.13.0
> release.
> > >> Among other things, this release
> > >> has the long pending Calcite rebase changes and the sooner we get it
> it
> > out
> > >> for users, the better.
> > >>
> > >> Thanks for volunteering !
> > >>
> > >> -Aman
> > >>
> > >>> On Wed, Feb 21, 2018 at 9:03 PM, Parth Chandra 
> > wrote:
> > >>>
> > >>> Hello Drillers,
> > >>>
> > >>>  I feel we might benefit from a early release for 1.13.0. We took
> > longer
> > >>> to do the previous release so it would be nice to bring the release
> > train
> > >>> back on track.
> > >>>
> > >>>  I'll volunteer (!) to manage the release :)
> > >>>
> > >>>  What do you guys think?
> > >>>
> > >>>  If we are in agreement on starting the release cycle and there are
> any
> > >>> issues on which work is in progress, that you feel we *must* include
> in
> > >> the
> > >>> release, please post in reply to this thread. Let's at least get a
> head
> > >>> start on closing pending PRs since these are usually what delays
> > >> releases.
> > >>>
> > >>> Thanks
> > >>>
> > >>> Parth
> > >>>
> > >>
> >
>

[jira] [Created] (DRILL-6181) CTAS should support writing nested structures (nested lists) to parquet.

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6181:
-

 Summary: CTAS should support writing nested structures (nested 
lists) to parquet.
 Key: DRILL-6181
 URL: https://issues.apache.org/jira/browse/DRILL-6181
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - Parquet
Affects Versions: 1.12.0
Reporter: Khurram Faraaz


Both Parquet and Hive support writing nested structures into parquet

https://issues.apache.org/jira/browse/HIVE-8909
https://issues.apache.org/jira/browse/PARQUET-113

A CTAS from Drill fails when there is a nested list of lists, in one of the 
columns in the project.

JSON data used in the test, note that "arr" is a nested list of lists 
 
{noformat} 
[root@qa102-45 ~]# cat jsonToParquet_02.json
{"id":"123","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"3","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"13","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"12","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"2","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"1","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"230","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"1230","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"1123","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"2123","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"1523","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
[root@qa102-45 ~]#
{noformat}

CTAS fails with UnsupportedOperationException on Drill 1.12.0-mapr commit id 
bb07ebbb9ba8742f44689f8bd8efb5853c5edea0

{noformat}
 0: jdbc:drill:schema=dfs.tmp> CREATE TABLE tbl_prq_from_json_02 as select id, 
arr from `jsonToParquet_02.json`;
Error: SYSTEM ERROR: UnsupportedOperationException: Unsupported type LIST

Fragment 0:0

[Error Id: 7e5b3c2d-9cf1-4e87-96c8-e7e7e8055ddf on qa102-45.qa.lab:31010] 
(state=,code=0)
{noformat}

Stack trace from drillbit.log

{noformat}
2018-02-22 09:56:54,368 [2570fb99-62da-a516-2c1f-0381e21723ae:frag:0:0] ERROR 
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: 
UnsupportedOperationException: Unsupported type LIST

Fragment 0:0

[Error Id: 7e5b3c2d-9cf1-4e87-96c8-e7e7e8055ddf on qa102-45.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
UnsupportedOperationException: Unsupported type LIST

Fragment 0:0

[Error Id: 7e5b3c2d-9cf1-4e87-96c8-e7e7e8055ddf on qa102-45.qa.lab:31010]
 at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
 ~[drill-common-1.12.0-mapr.jar:1.12.0-mapr]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301)
 [drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
 [drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
 at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
 [drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
 at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.12.0-mapr.jar:1.12.0-mapr]
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[na:1.8.0_161]
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[na:1.8.0_161]
 at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.lang.UnsupportedOperationException: Unsupported type LIST
 at 
org.apache.drill.exec.store.parquet.ParquetRecordWriter.getType(ParquetRecordWriter.java:253)
 ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
 at 
org.apache.drill.exec.store.parquet.ParquetRecordWriter.newSchema(ParquetRecordWriter.java:205)
 ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
 at 
org.apache.drill.exec.store.parquet.ParquetRecordWriter.updateSchema(ParquetRecordWriter.java:190)
 ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
 at 
org.apache.drill.exec.physical.impl.WriterRecordBatch.setupNewSchema(WriterRecordBatch.java:157)
 ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
 at 
org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:103)
 ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
 ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
 at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)

[jira] [Created] (DRILL-6155) support for pagination of results on Drill's web UI

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6155:
-

 Summary: support for pagination of results on Drill's web UI
 Key: DRILL-6155
 URL: https://issues.apache.org/jira/browse/DRILL-6155
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - Java
Affects Versions: 1.12.0
Reporter: Khurram Faraaz


We need support for pagination of results on Drill's web UI. Today if a query 
returns say a million results, those results are not paginated on Drill's web 
UI. Such large number of results should be paginated as required on the web UI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6146) UNION with empty input on any one side returns incorrect results

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6146:
-

 Summary: UNION with empty input on any one side returns incorrect 
results
 Key: DRILL-6146
 URL: https://issues.apache.org/jira/browse/DRILL-6146
 Project: Apache Drill
  Issue Type: Bug
  Components: Query Planning  Optimization
Affects Versions: 1.12.0
Reporter: Khurram Faraaz


When any one side of the UNION has an empty file as input, Drill returns 
incorrect results.
 
table t3 does not have any data inserted into its rows. Postgress returns 1 as 
the result for both the queries, whereas Drill does not.
 
{noformat}
postgres=# create table t3(id int, name varchar(25));
CREATE TABLE 
postgres=# select * from (values(1)) t union select id from t3;
       1
 
postgres=# select id from t3 union select * from (values(1)) t;
  1
 {noformat}
 
 
Results from Drill 1.12.0-mapr, note we return result 1 as result after the 
union.
We have a directory named empty_JSON_f , and it has a single empty JSON file 
(that JSON file has no content in it, it is empty).
 
{noformat}
0: jdbc:drill:schema=dfs.tmp> select * from (values(1)) UNION select id from 
empty_JSON_f;
+-+
| EXPR$0  |
+-+
| 1       |
+-+
1 row selected (2.272 seconds){noformat}
However, in this query we return null and loose the value 1 from the right hand 
side, after the union, this doesn't seem correct 
{noformat}
0: jdbc:drill:schema=dfs.tmp> select id from empty_JSON_f UNION select * from 
(values(1));
+---+
|  id   |
+---+
| null  |
+---+
1 row selected (0.33 seconds){noformat}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6141) JOIN query that uses USING clause returns incorrect results

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6141:
-

 Summary: JOIN query that uses USING clause returns incorrect 
results
 Key: DRILL-6141
 URL: https://issues.apache.org/jira/browse/DRILL-6141
 Project: Apache Drill
  Issue Type: Bug
  Components: Query Planning  Optimization
Affects Versions: 1.12.0
Reporter: Khurram Faraaz


Join query that uses USING clause returns incorrect results.

Postgres 9.2.23 returns only one occurrence of the "id" column

{noformat}
postgres=# create table t1(id int, name varchar(30));
CREATE TABLE
postgres=# create table t2(id int, name varchar(30));
CREATE TABLE

postgres=# select * from t1;
 id | name
+---
 10 | John
 13 | Kevin
 15 | Susan
(3 rows)

postgres=# select * from t2;
 id | name
+---
 19 | Kyle
 13 | Kevin
 1 | Bob
 17 | Kumar
(4 rows)

postgres=# select * from t1 JOIN t2 USING(id);
 id | name | name
+---+---
 13 | Kevin | Kevin
(1 row)

{noformat}

results from Drill 1.12.0-mapr commit : 2de42491be795721bcb4059bd46e27fc33272309

{noformat}


0: jdbc:drill:schema=dfs.tmp> create table t1 as select cast(columns[0] as int) 
c1, cast(columns[1] as varchar(30)) c2 from `t1.csv`;
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 3 |
+---++
1 row selected (0.213 seconds)
0: jdbc:drill:schema=dfs.tmp> create table t2 as select cast(columns[0] as int) 
c1, cast(columns[1] as varchar(30)) c2 from `t2.csv`;
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 4 |
+---++
1 row selected (0.168 seconds)

0: jdbc:drill:schema=dfs.tmp> select * from t1;
+-++
| c1 | c2 |
+-++
| 10 | John |
| 13 | Kevin |
| 15 | Susan |
+-++
3 rows selected (0.15 seconds)
0: jdbc:drill:schema=dfs.tmp> select * from t2;
+-++
| c1 | c2 |
+-++
| 19 | Kyle |
| 13 | Kevin |
| 1 | Bob |
| 17 | Kumar |
+-++
4 rows selected (0.171 seconds)

## Note that Drill returns an extra column, unlike Postgres, for the same query 
over same data

0: jdbc:drill:schema=dfs.tmp> select * from t1 JOIN t2 USING(c1);
+-++--++
| c1 | c2 | c10 | c20 |
+-++--++
| 13 | Kevin | 13 | Kevin |
+-++--++
1 row selected (0.256 seconds)

## explain plan for above query

0: jdbc:drill:schema=dfs.tmp> explain plan for select * from t1 JOIN t2 
USING(c1);
+--+--+
| text | json |
+--+--+
| 00-00 Screen
00-01 ProjectAllowDup(*=[$0], *0=[$1])
00-02 Project(T49¦¦*=[$0], T48¦¦*=[$2])
00-03 Project(T49¦¦*=[$2], c10=[$3], T48¦¦*=[$0], c1=[$1])
00-04 HashJoin(condition=[=($3, $1)], joinType=[inner])
00-06 Project(T48¦¦*=[$0], c1=[$1])
00-08 Scan(table=[[dfs, tmp, t2]], groupscan=[ParquetGroupScan 
[entries=[ReadEntryWithPath [path=maprfs:///tmp/t2]], 
selectionRoot=maprfs:/tmp/t2, numFiles=1, numRowGroups=1, 
usedMetadataFile=false, columns=[`*`]]])
00-05 Project(T49¦¦*=[$0], c10=[$1])
00-07 Project(T49¦¦*=[$0], c1=[$1])
00-09 Scan(table=[[dfs, tmp, t1]], groupscan=[ParquetGroupScan 
[entries=[ReadEntryWithPath [path=maprfs:///tmp/t1]], 
selectionRoot=maprfs:/tmp/t1, numFiles=1, numRowGroups=1, 
usedMetadataFile=false, columns=[`*`]]])

{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] New PMC member: Paul Rogers

[Khurram Faraaz]

Congratulations Paul.


From: Vova Vysotskyi 
Sent: Wednesday, January 31, 2018 12:51:25 PM
To: dev@drill.apache.org
Subject: Re: [ANNOUNCE] New PMC member: Paul Rogers

Congrats, Paul!

2018-01-31 22:40 GMT+02:00 Rob Wu :

> Congratulations, Paul!
>
> Best regards,
>
> Rob
>
> Best regards,
>
> Rob
> 
> From: Charles Givre 
> Sent: Wednesday, January 31, 2018 9:47:25 AM
> To: dev@drill.apache.org
> Subject: Re: [ANNOUNCE] New PMC member: Paul Rogers
>
> Congrats Paul!  Very well deserved!
>
> > On Jan 30, 2018, at 16:50, Aman Sinha  wrote:
> >
> > I am pleased to announce that Drill PMC invited Paul Rogers to the PMC
> and
> > he has accepted the invitation.
> >
> > Congratulations Paul and thanks for your contributions !
> >
> > -Aman
> > (on behalf of Drill PMC)
>
>


--
Kind regards,
Volodymyr Vysotskyi

Re: [ANNOUNCE] New Committer: Boaz Ben-Zvi

[Khurram Faraaz]

Congratulations, Boaz!


From: Robert Wu 
Sent: Wednesday, December 13, 2017 2:38:42 PM
To: dev@drill.apache.org
Subject: RE: [ANNOUNCE] New Committer: Boaz Ben-Zvi

Congratulations, Boaz!

Rob

-Original Message-
From: Jinfeng Ni [mailto:j...@apache.org]
Sent: Wednesday, December 13, 2017 12:20 PM
To: dev 
Subject: Re: [ANNOUNCE] New Committer: Boaz Ben-Zvi

Congratulations and welcome, Boaz!


Jinfeng


On Wed, Dec 13, 2017 at 11:17 AM, Robert Hou  wrote:

> Congratulations, Boaz!
>
>
> --Robert
>
> 
> From: Paul Rogers 
> Sent: Wednesday, December 13, 2017 11:02 AM
> To: dev@drill.apache.org
> Subject: Re: [ANNOUNCE] New Committer: Boaz Ben-Zvi
>
> Congrats! Well deserved.
>
> - Paul
>
> > On Dec 13, 2017, at 11:00 AM, Timothy Farkas  wrote:
> >
> > Congrats!
> >
> > 
> > From: Kunal Khatua 
> > Sent: Wednesday, December 13, 2017 10:47:14 AM
> > To: dev@drill.apache.org
> > Subject: RE: [ANNOUNCE] New Committer: Boaz Ben-Zvi
> >
> > Congratulations, Boaz!!
> >
> > -Original Message-
> > From: Abhishek Girish [mailto:agir...@apache.org]
> > Sent: Wednesday, December 13, 2017 10:25 AM
> > To: dev@drill.apache.org
> > Subject: Re: [ANNOUNCE] New Committer: Boaz Ben-Zvi
> >
> > Congratulations Boaz!
> > On Wed, Dec 13, 2017 at 10:23 AM Aman Sinha  wrote:
> >
> >> The Project Management Committee (PMC) for Apache Drill has invited
> >> Boaz Ben-Zvi  to become a committer, and we are pleased to announce
> >> that he has accepted.
> >>
> >> Boaz has been an active contributor to Drill for more than a year.
> >> He designed and implemented the Hash Aggregate spilling and is
> >> leading the efforts for Hash Join spilling.
> >>
> >> Welcome Boaz, and thank you for your contributions.  Keep up the
> >> good work !
> >>
> >> - Aman
> >> (on behalf of the Apache Drill PMC)
> >>
>
>

Re: [ANNOUNCE] New Committer: Vitalii Diravka

[Khurram Faraaz]

Congratulations Vitalii.


Regards,

Khurram


From: Vlad Rozov 
Sent: Tuesday, December 12, 2017 1:10:09 PM
To: dev@drill.apache.org
Subject: Re: [ANNOUNCE] New Committer: Vitalii Diravka

Congrats!

Thank you,

Vlad

On 12/11/17 11:18, Timothy Farkas wrote:
> Congrats!
> 
> From: Gautam Parai 
> Sent: Monday, December 11, 2017 7:32:27 AM
> To: dev@drill.apache.org
> Subject: Re: [ANNOUNCE] New Committer: Vitalii Diravka
>
> Congratulations Vitalii!
>
>
> Gautam
>
> 
> From: Vitalii Diravka 
> Sent: Monday, December 11, 2017 2:36:00 AM
> To: dev@drill.apache.org
> Subject: Re: [ANNOUNCE] New Committer: Vitalii Diravka
>
> Thanks to everyone! I highly appreciate the opportunity to work with you to
> make Drill better!
>
> Kind regards
> Vitalii
>
> On Mon, Dec 11, 2017 at 11:30 AM, Saurabh Mahapatra <
> saurabhmahapatr...@gmail.com> wrote:
>
>> Yes and yes!!! Congrats!
>>
>> On Mon, Dec 11, 2017 at 12:28 AM, Arina Yelchiyeva <
>> arina.yelchiy...@gmail.com> wrote:
>>
>>> Congratulations!
>>>
>>> Kind regards
>>> Arina
>>>
>>> On Mon, Dec 11, 2017 at 8:06 AM, Prasad Nagaraj Subramanya <
>>> prasadn...@gmail.com> wrote:
>>>
 Congratulations Vitali!

 Thanks,
 Prasad

 On Sun, Dec 10, 2017 at 9:58 PM, Robert Hou  wrote:

> Congratulations!
>
>
> --Robert
>
> 
> From: Paul Rogers 
> Sent: Sunday, December 10, 2017 4:29 PM
> To: dev@drill.apache.org
> Subject: Re: [ANNOUNCE] New Committer: Vitalii Diravka
>
> Congrats! Well deserved.
>
> - Paul
>
>> On Dec 10, 2017, at 3:16 PM, AnilKumar B 
 wrote:
>> Congratulations Vitalii
>>
>> Thanks & Regards,
>> B Anil Kumar.
>>
>> On Sun, Dec 10, 2017 at 3:12 PM, rahul challapalli <
>> challapallira...@gmail.com> wrote:
>>
>>> Congratulations Vitalii!
>>>
>>> On Sun, Dec 10, 2017 at 3:05 PM, Kunal Khatua 
 wrote:
 Congratulations!!

 -Original Message-
 From: Aman Sinha [mailto:amansi...@apache.org]
 Sent: Sunday, December 10, 2017 11:06 AM
 To: dev@drill.apache.org
 Subject: [ANNOUNCE] New Committer: Vitalii Diravka

 The Project Management Committee (PMC) for Apache Drill has
>> invited
 Vitalii Diravka  to become a committer, and we are pleased to
 announce
>>> that
 he has accepted.

 Vitalii has been an active contributor to Drill over the last 1
>> 1/2
>>> years.
 His contributions have spanned areas such as: CASTing issues with
 Date/Timestamp, Parquet metadata and SQL enhancements, among
>>> others.
 Welcome Vitalii, and thank you for your contributions.  Keep up
>> the
> good
 work !

 - Aman
 (on behalf of the Apache Drill PMC)

>

[jira] [Created] (DRILL-5965) Array index access returns an empty array

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5965:
-

 Summary: Array index access returns an empty array
 Key: DRILL-5965
 URL: https://issues.apache.org/jira/browse/DRILL-5965
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - JSON
Affects Versions: 1.12.0
Reporter: Khurram Faraaz


Accessing an array with [] from JSON data, returns an empty 
string, whereas it should return the actual data from the array at the index 
and not an empty array. 

Drill 1.11.0-mapr  commit: 065d72ba48c7af6b389b763753ecb6bf7d229ce8

{noformat}
0: jdbc:drill:schema=dfs.tmp> select t.structured_rep[0] from 
`cornell_nlvr_train.json` t limit 1;
+-+
| EXPR$0  |
+-+
| []  |
+-+
1 row selected (0.249 seconds)
{noformat}

Where as accessing the elements of the array returns correct results.

{noformat}
0: jdbc:drill:schema=dfs.tmp> select t.structured_rep[0][0] from 
`cornell_nlvr_train.json` t limit 1;
+---+
|EXPR$0 |
+---+
| {"y_loc":21,"size":20,"type":"triangle","x_loc":27,"color":"Yellow"}  |
+---+
1 row selected (0.325 seconds)
0: jdbc:drill:schema=dfs.tmp> select t.structured_rep[0][1] from 
`cornell_nlvr_train.json` t limit 1;
+-+
|   EXPR$0|
+-+
| {"y_loc":60,"size":10,"type":"circle","x_loc":59,"color":"Yellow"}  |
+-+
1 row selected (0.247 seconds)
{noformat}

Data used in the test
{noformat}
{
"sentence": "There is a circle closely touching a corner of a box.",
"label": "true",
"identifier": "1304-0",
"directory": "74",
"evals": {
"r0": "true"
},
"structured_rep": [
[{
"y_loc": 21,
"size": 20,
"type": "triangle",
"x_loc": 27,
"color": "Yellow"
}, {
"y_loc": 60,
"size": 10,
"type": "circle",
"x_loc": 59,
"color": "Yellow"
}],
[{
"y_loc": 81,
"size": 10,
"type": "triangle",
"x_loc": 48,
"color": "Yellow"
}, {
"y_loc": 64,
"size": 20,
"type": "circle",
"x_loc": 77,
"color": "#0099ff"
}],
[{
"y_loc": 2,
"size": 20,
"type": "triangle",
"x_loc": 62,
"color": "Yellow"
}, {
"y_loc": 70,
"size": 30,
"type": "circle",
"x_loc": 70,
"color": "Black"
}, {
"y_loc": 51,
"size": 20,
"type": "circle",
"x_loc": 30,
"color": "#0099ff"
}, {
"y_loc": 42,
"size": 20,
"type": "circle",
"x_loc": 67,
"color": "Yellow"
}, {
"y_loc": 73,
"size": 20,
"type": "circle",
"x_loc": 37,
"color": "Black"
}, {
"y_loc": 14,
"size": 30,
"type": "triangle",
"x_loc": 7,
"color": "Yellow"
}, {
"y_loc": 27,
"size": 10,
"type": "circle",
"x_loc": 48,
"color": "Black"
}]
]
}
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5963:
-

 Summary: Canceling a query hung in planning state, leaves the 
query in ENQUEUED state for ever.
 Key: DRILL-5963
 URL: https://issues.apache.org/jira/browse/DRILL-5963
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.12.0
 Environment: Drill 1.12.0-SNAPSHOT, commit: 
4a718a0bd728ae02b502ac93620d132f0f6e1b6c
Reporter: Khurram Faraaz
Priority: Critical


Canceling the below query that is hung in planning state, leaves the query in 
ENQUEUED state for ever.

Here is the query that is hung in planning state
{noformat}
0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS 
CSV_DATA from (values(1));
+--+
|  |
+--+
+--+
No rows selected (304.291 seconds)
{noformat}

Explain plan for that query also just hangs.
{noformat}
explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
from (values(1));
...
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5947) COUNT() on empty JSON results in schema change error

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5947:
-

 Summary: COUNT() on empty JSON results in schema 
change error
 Key: DRILL-5947
 URL: https://issues.apache.org/jira/browse/DRILL-5947
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Khurram Faraaz


Count over a field, from a non-empty and another empty JSON file results in 
schema change error, "Streaming aggregate does not support schema changes"

{noformat}
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was 
removed in 8.0
apache drill 1.11.0-mapr 
"drill baby drill"
0: jdbc:drill:schema=dfs.tmp> select count(id) from `diff_types`;
Error: UNSUPPORTED_OPERATION ERROR: Streaming aggregate does not support schema 
changes
Prior schema : 
BatchSchema [fields=[id(BIGINT:OPTIONAL)], selectionVector=NONE]
New schema : 
BatchSchema [fields=[id(INT:OPTIONAL)], selectionVector=NONE]

Fragment 0:0

[Error Id: c1c77a0e-c7cf-494e-b8d2-ef2cc7e816e0 on centos-01.qa.lab:31010] 
(state=,code=0)
0: jdbc:drill:schema=dfs.tmp> select id from `dif`Display all 872 
possibilities? (y or n)
0: jdbc:drill:schema=dfs.tmp> select id from `diff_types/m1.json`;
+---+
|  id   |
+---+
| 1234  |
+---+
1 row selected (0.157 seconds)
0: jdbc:drill:schema=dfs.tmp> select id from `diff_types/m2.json`;
+---+
|  id   |
+---+
| null  |
+---+
1 row selected (0.169 seconds)
0: jdbc:drill:schema=dfs.tmp> select * from `diff_types`;
+---+
|  id   |
+---+
| 1234  |
| null  |
+---+
2 rows selected (0.209 seconds)
0: jdbc:drill:schema=dfs.tmp> select count(*) from `diff_types`;
+-+
| EXPR$0  |
+-+
| 2   |
+-+
1 row selected (0.208 seconds)

{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5945) Streaming aggregate does not support schema changes

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5945:
-

 Summary: Streaming aggregate does not support schema changes
 Key: DRILL-5945
 URL: https://issues.apache.org/jira/browse/DRILL-5945
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Khurram Faraaz


There are two JSON files, one of the arrays "arr" in f2.json is empty, whereas 
array "arr" in f1.json is not empty, a count on arr results in, "Streaming 
aggregate does not support schema changes"

{noformat}
0: jdbc:drill:schema=dfs.tmp> select count(arr) from `f*.json`;
Error: UNSUPPORTED_OPERATION ERROR: Streaming aggregate does not support schema 
changes
Prior schema : 
BatchSchema [fields=[arr(VARCHAR:REPEATED) [$data$(VARCHAR:REQUIRED)]], 
selectionVector=NONE]
New schema : 
BatchSchema [fields=[arr(INT:REPEATED) [$data$(INT:REQUIRED)]], 
selectionVector=NONE]

Fragment 0:0

[Error Id: 9081b02f-0de8-486a-b58b-f28ec3b84d98 on centos-01.qa.lab:31010] 
(state=,code=0)

{noformat}

Stack trace from drillbit.log

{noformat}
2017-11-08 13:36:48,159 [25fc870f-e4d6-7107-c6cc-6e68ba92c4ba:frag:0:0] INFO  
o.a.d.e.p.i.a.StreamingAggBatch - User Error Occurred: Streaming aggregate does 
not support schema changes
Prior schema :
BatchSchema [fields=[arr(VARCHAR:REPEATED) [$data$(VARCHAR:REQUIRED)]], 
selectionVector=NONE]
New schema :
BatchSchema [fields=[arr(INT:REPEATED) [$data$(INT:REQUIRED)]], 
selectionVector=NONE]
org.apache.drill.common.exceptions.UserException: UNSUPPORTED_OPERATION ERROR: 
Streaming aggregate does not support schema changes
Prior schema :
BatchSchema [fields=[arr(VARCHAR:REPEATED) [$data$(VARCHAR:REQUIRED)]], 
selectionVector=NONE]
New schema :
BatchSchema [fields=[arr(INT:REPEATED) [$data$(INT:REQUIRED)]], 
selectionVector=NONE]


[Error Id: 9081b02f-0de8-486a-b58b-f28ec3b84d98 ]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
 ~[drill-common-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext(StreamingAggBatch.java:207)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105) 
[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95) 
[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:234)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:227)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at java.security.AccessController.doPrivileged(Native Method) 
[na:1.8.0_144]
at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_144]
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
 [hadoop-common-2.7.0-mapr-1607.jar:na]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:227)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.11.0-mapr.jar:1.11.0-mapr]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[na:1.8.0_144]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[na:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]

{noformat}

Data used in the query

{noformat}

[root@centos-01 json_data]# cat f1.json 
{
"id":1,
"arr":["str1","str2","str3","str4"]
}
[root@centos-01 json_data]# cat f2.json 
{
&qu

[jira] [Created] (DRILL-5909) need new JMX metrics for (FAILED and CANCELED) queries

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5909:
-

 Summary: need new JMX metrics for (FAILED and CANCELED) queries
 Key: DRILL-5909
 URL: https://issues.apache.org/jira/browse/DRILL-5909
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Monitoring
Affects Versions: 1.11.0, 1.12.0
Reporter: Khurram Faraaz


we have these JMX metrics today

{noformat}
drill.queries.running
drill.queries.completed
{noformat}

we need these new JMX metrics

{noformat}
drill.queries.failed
drill.queries.canceled
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

DMV appointment at 9:00 AM today, will be in by 11:00 AM after test, depending on when everything completes @DMV.

[Khurram Faraaz]

I have a DMV appointment at 9:00 AM today, I should be in by 11:00 AM after 
test, depending on when everything completes.


Thanks,

Khurram

[jira] [Created] (DRILL-5806) DrillRuntimeException: Interrupted but context.shouldContinue() is true

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5806:
-

 Summary: DrillRuntimeException: Interrupted but 
context.shouldContinue() is true
 Key: DRILL-5806
 URL: https://issues.apache.org/jira/browse/DRILL-5806
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.12.0
 Environment: Drill 1.12.0 commit : 
aaff1b35b7339fb4e6ab480dd517994ff9f0a5c5
Reporter: Khurram Faraaz



On a three node cluster
1. run concurrent queries (TPC-DS query 11) from a Java program.
2. stop the drillbit (foreman drillbit) this way, 
/opt/mapr/drill/drill-1.12.0/bin/drillbit.sh stop
3. InterruptedException: null, is written to the drillbit.log

Stack trace from drillbit.log
{noformat}
2017-09-19 21:49:20,867 [263e6f48-0ace-0c0d-4f90-55ae2f0d778b:frag:5:0] ERROR 
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: InterruptedException

Fragment 5:0

[Error Id: 63ce8c18-040a-47f9-9643-e826de9a1a27 on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
InterruptedException

Fragment 5:0

[Error Id: 63ce8c18-040a-47f9-9643-e826de9a1a27 on centos-01.qa.lab:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
 ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:298)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264)
 [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_91]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: 
Interrupted but context.shouldContinue() is true
at 
org.apache.drill.exec.work.batch.BaseRawBatchBuffer.getNext(BaseRawBatchBuffer.java:178)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.unorderedreceiver.UnorderedReceiverBatch.getNextBatch(UnorderedReceiverBatch.java:141)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.unorderedreceiver.UnorderedReceiverBatch.next(UnorderedReceiverBatch.java:164)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:141)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.test.generated.HashAggregatorGen498.doWork(HashAggTemplate.java:581)
 ~[na:na]
at 
org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext(HashAggBatch.java:168)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar

[jira] [Resolved] (DRILL-3241) Query with window function runs out of direct memory and does not report back to client that it did

[Khurram Faraaz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Khurram Faraaz resolved DRILL-3241.
---
Resolution: Fixed

> Query with window function runs out of direct memory and does not report back 
> to client that it did
> ---
>
> Key: DRILL-3241
> URL: https://issues.apache.org/jira/browse/DRILL-3241
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Deneche A. Hakim
>Priority: Critical
> Fix For: 1.12.0
>
>
> Even though query run out of memory and was cancelled on the server, client 
> (sqlline) was never notified of the event and it appears to the user that 
> query is hung. 
> Configuration:
> Single drillbit configured with:
> DRILL_MAX_DIRECT_MEMORY="2G"
> DRILL_HEAP="1G"
> TPCDS100 parquet files
> Query:
> {code}
> select 
>   sum(ss_quantity) over(partition by ss_store_sk order by ss_sold_date_sk) 
> from store_sales;
> {code}
> drillbit.log
> {code}
> 2015-06-01 21:42:29,514 [BitServer-5] ERROR 
> o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  
> Connection: /10.10.88.133:31012 <--> /10.10.88.133:38887 (data server).  
> Closing connection.
> io.netty.handler.codec.DecoderException: java.lang.OutOfMemoryError: Direct 
> buffer memory
> at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:233)
>  ~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>  [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>  [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
>  [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>  [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>  [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
>  [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:618)
>  [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> at 
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:329) 
> [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:250) 
> [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>  [netty-common-4.0.27.Final.jar:4.0.27.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_71]
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) 
> ~[na:1.7.0_71]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) 
> ~[na:1.7.0_71]
> at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:437) 
> ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
> at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:179) 
> ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
> at io.netty.buffer.PoolArena.allocate(PoolArena.java:168) 
> ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
> at io.netty.buffer.PoolArena.reallocate(PoolArena.java:280) 
> ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
> at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:110) 
> ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:251) 
> ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
> at 
> io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:849) 
> ~[netty

[jira] [Created] (DRILL-5793) NPE on close

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5793:
-

 Summary: NPE on close
 Key: DRILL-5793
 URL: https://issues.apache.org/jira/browse/DRILL-5793
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.12.0
 Environment: Drill 1.12.0 commit : 
aaff1b35b7339fb4e6ab480dd517994ff9f0a5c5
Reporter: Khurram Faraaz


The code looks wrong:
{noformat}
 @Override
 public void close() throws Exception {
   options.close();
 }
If the shutdown occurs to early, options is not yet assigned and an NPE results.
{noformat}

{noformat}
2017-09-14 20:16:39,551 [main] DEBUG o.apache.drill.exec.server.Drillbit - 
Shutdown begun.
2017-09-14 20:16:41,560 [pool-5-thread-1] INFO  
o.a.drill.exec.rpc.user.UserServer - closed eventLoopGroup 
io.netty.channel.nio.NioEventLoopGroup@71a84ff4 in 1006 ms
2017-09-14 20:16:41,560 [pool-5-thread-2] INFO  
o.a.drill.exec.rpc.data.DataServer - closed eventLoopGroup 
io.netty.channel.nio.NioEventLoopGroup@f711283 in 1005 ms
2017-09-14 20:16:41,561 [pool-5-thread-1] INFO  
o.a.drill.exec.service.ServiceEngine - closed userServer in 1007 ms
2017-09-14 20:16:41,562 [pool-5-thread-2] DEBUG 
o.a.drill.exec.memory.BaseAllocator - closed allocator[rpc:bit-data].
2017-09-14 20:16:41,562 [pool-5-thread-2] INFO  
o.a.drill.exec.service.ServiceEngine - closed dataPool in 1008 ms
2017-09-14 20:16:41,563 [main] DEBUG o.a.drill.exec.memory.BaseAllocator - 
closed allocator[rpc:user].
2017-09-14 20:16:41,563 [main] DEBUG o.a.drill.exec.memory.BaseAllocator - 
closed allocator[rpc:bit-control].
2017-09-14 20:16:41,593 [main] DEBUG o.a.drill.exec.memory.BaseAllocator - 
closed allocator[ROOT].
2017-09-14 20:16:41,593 [main] WARN  o.apache.drill.exec.server.Drillbit - 
Failure on close()
java.lang.NullPointerException: null
at 
org.apache.drill.exec.server.options.SystemOptionManager.close(SystemOptionManager.java:369)
 ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at 
org.apache.drill.exec.server.DrillbitContext.close(DrillbitContext.java:241) 
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at org.apache.drill.exec.work.WorkManager.close(WorkManager.java:154) 
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) 
~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) 
~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:173) 
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:314) 
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:290) 
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:286) 
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (DRILL-4296) Query hangs in CANCELLATION_REQUESTED when cancelled after it starts returning results

[Khurram Faraaz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Khurram Faraaz resolved DRILL-4296.
---
Resolution: Fixed

> Query hangs in CANCELLATION_REQUESTED when cancelled after it starts 
> returning results
> --
>
> Key: DRILL-4296
> URL: https://issues.apache.org/jira/browse/DRILL-4296
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.5.0
> Environment: commit.id=c9dbfbd
> 2 nodes with 32 cores and 32GB of max direct memory for drill
>Reporter: Deneche A. Hakim
> Attachments: 295eefc3-8d15-d63b-a721-3fde365b639c.sys.drill, 
> data.tar.gz, node1_jstack.txt, node2_jstack.txt
>
>
> After running the following queries (it's the same reproduction from 
> DRILL-2274):
> {noformat}
> set planner.memory.max_query_memory_per_node=8589934592;
> select sub1.uid from `all2274.json` sub1 inner join `all2274.json` sub2 on 
> sub1.uid = sub2.uid order by sub1.uid;
> {noformat}
> After the query starts returning results, I cancelled the query from sqlline. 
> This caused the query to hang in a CANCELLATION_REQUESTED state.
> Looking at jstack (attached) the root fragment is blocked waiting for Ack 
> from the client.
> The the foreman node (which also runs Zookeeper) runs out of disk space once 
> the query finishes spilling, which seems to contribute to the issue. Once I 
> changed the spill directory to nfs I no longer so the issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [ANNOUNCE] New PMC member: Arina Ielchiieva

[Khurram Faraaz]

Congratulations, Arina!


From: Jinfeng Ni 
Sent: Thursday, August 3, 2017 12:06:31 AM
To: dev
Subject: Re: [ANNOUNCE] New PMC member: Arina Ielchiieva

Congratulations, Arina!



On Wed, Aug 2, 2017 at 11:27 AM, Kunal Khatua  wrote:

> Congratulations, Arina!!
>
>
> Thank you for your contributions to Drill !
>
>
> ~ Kunal
>
> 
> From: Aman Sinha 
> Sent: Wednesday, August 2, 2017 11:23:23 AM
> To: dev@drill.apache.org
> Subject: [ANNOUNCE] New PMC member: Arina Ielchiieva
>
> I am pleased to announce that Drill PMC invited Arina Ielchiieva to the PMC
> and she has accepted the invitation.
>
> Congratulations Arina and thanks for your contributions !
>
> -Aman
> (on behalf of Drill PMC)
>

Re: [VOTE] Release Apache Drill 1.11.0 - rc0

[Khurram Faraaz]

+1 (non-binding)


Built from source and ran few of the Functional tests, looks good.


From: Sorabh Hamirwasia 
Sent: Friday, July 28, 2017 6:24:29 AM
To: dev@drill.apache.org
Subject: Re: [VOTE] Release Apache Drill 1.11.0 - rc0


LGTM +1


- Built from source on CentOS using [4], deployed on a one node cluster

- Downloaded and deployed a one node cluster using [2]

- Ran some queries against employee.json & regions.json file in classpath using 
sqlline

- Enabled plain and kerberos authentication and was successfully able to 
connect sqlline and run queries.

- Enabled SASL encryption (DRILL-4335) and was successfully able to connect 
sqlline and run queries.

- Verified Web UI working with encryption enabled/disabled.

- Built C++ client on CentOS

- Enabled SASL encryption (DRILL-4335) and was successfully able to connect 
querySubmitter and run queries.

- Successfully ran Laurent's query for issue DRILL-5659 with encryption enabled.

- Verified bunch of negative cases in context of security.


Thanks,
Sorabh



From: Abhishek Girish 
Sent: Thursday, July 27, 2017 3:24:50 PM
To: dev@drill.apache.org
Subject: Re: [VOTE] Release Apache Drill 1.11.0 - rc0

+1 (non-binding)

- Built from source using [4], deployed on a multi-node cluster
- Ran tests from Drill Test Framework [5]

Looks good!

[5] https://github.com/mapr/drill-test-framework
[https://avatars3.githubusercontent.com/u/1226517?v=3=400]

GitHub - mapr/drill-test-framework: Test Framework for 
...
github.com
drill-test-framework - Test Framework for Apache Drill




On Thu, Jul 27, 2017 at 3:17 PM, Aman Sinha  wrote:

> LGTM  +1
>
> Downloaded the source on my mac and built using Java 1.8
> Started in embedded mode. Created a view on TPC-DS data and ran a bunch of
> queries with joins, aggregations, order-by
> Did a CTAS with partition-by,  Created metadata cache.
> Checked Explain plans
> Checked Web UI for query profile.
>
> RC0 looks good !
>
>
> On Wed, Jul 26, 2017 at 4:46 PM, Padma Penumarthy 
> wrote:
>
> > +1 (non-binding)
> >
> > Tried in embedded mode on my mac. Ran some queries.
> > Downloaded and built on CentOS VM.
> > Installed the build on the 4 node cluster. Ran some queries on parquet
> > files.
> >
> > Thanks,
> > Padma
> >
> >
> > > On Jul 26, 2017, at 2:54 PM, Kunal Khatua  wrote:
> > >
> > > +1 (non-binding)
> > >
> > > Downloaded and built in CentOS. Tried a handful of queries and it
> worked
> > fine.
> > >
> > > Tried repro for DRILL-5420 (CPU churn) and no issues. Ran a few 100
> > query cancellations and limit queries too, but didn't see any issues.
> > (guessing this resolves DRILL-5435 and DRILL-5609 as well).
> > > Tested the above with DRILL-5481 (in-memory profile persistence) and
> > that worked fine too.
> > >
> > >
> > >
> > > -Original Message-
> > > From: Parth Chandra [mailto:par...@apache.org]
> > > Sent: Tuesday, July 25, 2017 4:26 PM
> > > To: dev 
> > > Subject: Re: [VOTE] Release Apache Drill 1.11.0 - rc0
> > >
> > > +1
> > >
> > > Downloaded src and checked all signatures.
> > > Built on MacOS. Built C++ client on MacOS.
> > > Verified fix for DRILL-5659
> > > Ran a couple of test queries in sqlline.
> > >
> > > Looks good.
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Jul 25, 2017 at 3:36 AM, Arina Yelchiyeva <
> > arina.yelchiy...@gmail.com> wrote:
> > >
> > >> Hi all,
> > >>
> > >> I'd like to propose the first release candidate (rc0) of Apache Drill,
> > >> version 1.11.0.
> > >>
> > >> The release candidate covers a total of 126 resolved JIRAs [1]. Thanks
> > >> to everyone who contributed to this release.
> > >>
> > >> The tarball artifacts are hosted at [2] and the maven artifacts are
> > >> hosted at [3].
> > >>
> > >> This release candidate is based on commit
> > >> 4220fb2fffbc81883df3e5fea575fa0a584852b3 located at [4].
> > >>
> > >> The vote ends at 1:00 PM UTC (5:00 AM PT), July 28, 2017.
> > >>
> > >> [ ] +1
> > >> [ ] +0
> > >> [ ] -1
> > >>
> > >> Here's my vote: +1 (non-binding)
> > >>
> > >>
> > >> [1]
> > >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > >> projectId=12313820=12339943
> > >> [2] http://home.apache.org/~arina/drill/releases/1.11.0/rc0/
> > >> [3] https://repository.apache.org/content/repositories/
> > >> orgapachedrill-1042/
> > >> [4] https://github.com/arina-ielchiieva/drill/commits/drill-1.11.0
> > >>
> > >> Kind regards
> > >> Arina
> > >>
> >
> >
>

Re: FindHardDistributionScans throws a NPE while visiting a TableScan

[Khurram Faraaz]

Muhammad, please create a pull request and someone will review you code, ensure 
that existing unit tests don't fail due to your changes.


Thanks,

Khurram


From: Muhammad Gelbana 
Sent: Wednesday, June 21, 2017 4:11:41 PM
To: dev@drill.apache.org
Subject: Re: FindHardDistributionScans throws a NPE while visiting a TableScan

This has been bugging me for sometime, and I've only solved it after
starting this thread !

I solved this by overriding the
*org.apache.calcite.rel.AbstractRelNode.accept(RelShuttle)* method for the
relational node(s) containing *JdbcTableScan* to avoid this.

@Override
> public RelNode accept(RelShuttle shuttle) {
>
> if(shuttle.getClass().getName().equals("org.apache.drill.exec.planner.sql.handlers.FindHardDistributionScans")){
> return this;
> }
> return super.accept(shuttle);
> }


If someone finds this introducing another bug, please tell me about it.


*-*
*Muhammad Gelbana*
http://www.linkedin.com/in/mgelbana

On Tue, Jun 20, 2017 at 2:13 AM, Jinfeng Ni  wrote:

> unwrap() essentially is doing a cast.  If it returns null for
> unwrap(DrillTranslatableTable.class) or unwrap(DrillTable.class), it means
> the table associate with this TableScan does not implement either
> interface. My suspicion is  JDBC storage plugin returns JdbcTable [1],
> unlikely other storage plugin which returns an instance implementing
> DrillTable.
>
> This seems to indicate FindHardDistributionScans could not be used to
> non-DrillTable. I'm not sure if that's the intention of that code, though.
>
> 1.
> https://github.com/apache/calcite/blob/master/core/src/
> main/java/org/apache/calcite/adapter/jdbc/JdbcSchema.java#L233-L234
>
> On Mon, Jun 19, 2017 at 2:19 PM, Muhammad Gelbana 
> wrote:
>
> > Everyone,
> >
> > I made a copy of the Jdbc plugin and made modifications to it by adding a
> > few rules. None of the modification I made or the rules I wrote should
> have
> > anything extra to do with handling the following SQL query
> >
> > SELECT * FROM incorta.SALES.SALES SALES WHERE 1 = 2 LIMIT 1
> >
> >
> > I know the query is useless, but I need to to know how to fix the
> following
> > error thrown while handling this query. This is the final query plan:
> >
> > DEBUG o.a.d.e.p.s.h.DefaultSqlHandler - HEP_BOTTOM_UP:Convert SUM to
> $SUM0
> > > (0ms):
> > > DrillLimitRel(*fetch=[1]*): rowcount = 1.0, cumulative cost = {201.0
> > > rows, 205.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 2653
> > >   DrillLimitRel(*offset=[0], fetch=[0]*): rowcount = 1.0, cumulative
> cost
> > > = {200.0 rows, 201.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 2651
> > > GelbanaJdbcDrel: rowcount = 100.0, cumulative cost = {200.0 rows,
> > > 201.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 2649
> > >   JdbcTableScan(table=[[gelbana, SALES, SALES]]): rowcount = 100.0,
> > > cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0
> > memory},
> > > id = 2572
> >
> >
> > This is the throw error stacktrace
> >
> > [Error Id: 83ea094a-db24-4d6d-bf0d-271db26db933 on 640fb7ebbd1a:31010]
> > at
> > org.apache.drill.common.exceptions.UserException$
> > Builder.build(UserException.java:543)
> > ~[drill-common-1.9.0.jar:1.9.0]
> > at
> > org.apache.drill.exec.work.foreman.Foreman$ForemanResult.
> > close(Foreman.java:825)
> > [drill-java-exec-1.9.0.jar:1.9.0]
> > at org.apache.drill.exec.work.foreman.Foreman.moveToState(
> > Foreman.java:935)
> > [drill-java-exec-1.9.0.jar:1.9.0]
> > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:281)
> > [drill-java-exec-1.9.0.jar:1.9.0]
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > [na:1.8.0_131]
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > [na:1.8.0_131]
> > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> > Caused by: org.apache.drill.exec.work.foreman.ForemanException:
> Unexpected
> > exception during fragment initialization: null
> > ... 4 common frames omitted
> > *Caused by: java.lang.NullPointerException: null*
> > at
> > org.apache.drill.exec.planner.sql.handlers.FindHardDistributionScans.
> > visit(FindHardDistributionScans.java:55)
> > ~[drill-java-exec-1.9.0.jar:1.9.0]
> > at org.apache.calcite.rel.core.TableScan.accept(TableScan.java:166)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.RelShuttleImpl.visitChild(
> > RelShuttleImpl.java:53)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at
> > org.apache.calcite.rel.RelShuttleImpl.visitChildren(
> > RelShuttleImpl.java:68)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:126)
> > ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
> > at org.apache.calcite.rel.AbstractRelNode.accept(
> AbstractRelNode.java:256)
> > 

[jira] [Created] (DRILL-5591) non-ASCII characters in text file result in MalformedInputException

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5591:
-

 Summary: non-ASCII characters in text file result in 
MalformedInputException
 Key: DRILL-5591
 URL: https://issues.apache.org/jira/browse/DRILL-5591
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - Text & CSV
Affects Versions: 1.11.0
Reporter: Khurram Faraaz


I am on Drill 1.11.0 commit id: 874bf62

To repro the issue:
wget http://cfdisat.blob.core.windows.net/lco/l_RFC_2017_05_11_2.txt.gz
gunzip l_RFC_2017_05_11_2.txt.gz
hadoop fs -put l_RFC_2017_05_11_2.txt /tmp

There are some non-ASCII characters at the beginning and end of the file used 
in the test.
{noformat}
[root@centos-01 drill_5590]# head l_RFC_2017_05_11_2.txt
����0���1��
��� ��RFC|SNCF|SUBCONTRATACION
CUBB910321AC1|0|0
CUBB9104187K9|0|0
CUBB910709KD0|0|0
CUBB910817CE8|0|0
CUBB9111286YA|0|0
CUBB920408J69|0|0
{noformat}

Failing query
{noformat}
0: jdbc:drill:schema=dfs.tmp> select count(1) from `l_RFC_2017_05_11_2.txt` t 
where columns[0] like 'CUBA7706%';
Error: SYSTEM ERROR: MalformedInputException: Input length = 1

Fragment 0:0

[Error Id: cdfa704c-0bc8-4791-95ae-d05b4c63beab on centos-01.qa.lab:31010] 
(state=,code=0)
{noformat}

Stack trace from drillbit.log
{noformat}
Caused by: java.lang.RuntimeException: 
java.nio.charset.MalformedInputException: Input length = 1
at 
org.apache.drill.exec.expr.fn.impl.CharSequenceWrapper.decodeUT8(CharSequenceWrapper.java:185)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.expr.fn.impl.CharSequenceWrapper.setBuffer(CharSequenceWrapper.java:119)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.test.generated.FiltererGen15.doEval(FilterTemplate2.java:50)
 ~[na:na]
at 
org.apache.drill.exec.test.generated.FiltererGen15.filterBatchNoSV(FilterTemplate2.java:100)
 ~[na:na]
at 
org.apache.drill.exec.test.generated.FiltererGen15.filterBatch(FilterTemplate2.java:73)
 ~[na:na]
at 
org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.doWork(FilterRecordBatch.java:81)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:93)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:133)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 

[jira] [Created] (DRILL-5581) Query with CASE statement returns wrong results

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5581:
-

 Summary: Query with CASE statement returns wrong results
 Key: DRILL-5581
 URL: https://issues.apache.org/jira/browse/DRILL-5581
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Khurram Faraaz


A query that uses case statement, returns wrong results.

{noformat}
Apache Drill 1.11.0-SNAPSHOT, commit id: 874bf629

[test@centos-101 ~]# cat order_sample.csv
202634342,2101,20160301

apache drill 1.11.0-SNAPSHOT
"this isn't your grandfather's sql"
0: jdbc:drill:schema=dfs.tmp> ALTER SESSION SET `store.format`='csv';
+---++
|  ok   |summary |
+---++
| true  | store.format updated.  |
+---++
1 row selected (0.245 seconds)
0: jdbc:drill:schema=dfs.tmp> CREATE VIEW  `vw_order_sample_csv` as
. . . . . . . . . . . . . . > SELECT
. . . . . . . . . . . . . . > `columns`[0] AS `ND`,
. . . . . . . . . . . . . . > CAST(`columns`[1] AS BIGINT) AS `col1`,
. . . . . . . . . . . . . . > CAST(`columns`[2] AS BIGINT) AS `col2`
. . . . . . . . . . . . . . > FROM `order_sample.csv`;
+---+--+
|  ok   |   summary|
+---+--+
| true  | View 'vw_order_sample_csv' created successfully in 'dfs.tmp' schema  |
+---+--+
1 row selected (0.253 seconds)
0: jdbc:drill:schema=dfs.tmp> select
. . . . . . . . . . . . . . > case
. . . . . . . . . . . . . . > when col1 > col2 then col1
. . . . . . . . . . . . . . > else col2
. . . . . . . . . . . . . . > end as temp_col,
. . . . . . . . . . . . . . > case
. . . . . . . . . . . . . . > when col1 = 2101 and (20170302 - col2) > 
1 then 'D'
. . . . . . . . . . . . . . > when col2 = 2101 then 'P'
. . . . . . . . . . . . . . > when col1 - col2 > 1 then '0'
. . . . . . . . . . . . . . > else 'A'
. . . . . . . . . . . . . . > end as status
. . . . . . . . . . . . . . > from  `vw_order_sample_csv`;
+---+-+
| temp_col  | status  |
+---+-+
| 20160301  | A   |
+---+-+
1 row selected (0.318 seconds)

0: jdbc:drill:schema=dfs.tmp> explain plan for
. . . . . . . . . . . . . . > select
. . . . . . . . . . . . . . > case
. . . . . . . . . . . . . . > when col1 > col2 then col1
. . . . . . . . . . . . . . > else col2
. . . . . . . . . . . . . . > end as temp_col,
. . . . . . . . . . . . . . > case
. . . . . . . . . . . . . . > when col1 = 2101 and (20170302 - col2) > 
1 then 'D'
. . . . . . . . . . . . . . > when col2 = 2101 then 'P'
. . . . . . . . . . . . . . > when col1 - col2 > 1 then '0'
. . . . . . . . . . . . . . > else 'A'
. . . . . . . . . . . . . . > end as status
. . . . . . . . . . . . . . > from  `vw_order_sample_csv`;
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01  Project(temp_col=[CASE(>(CAST(ITEM($0, 1)):BIGINT, CAST(ITEM($0, 
2)):BIGINT), CAST(ITEM($0, 1)):BIGINT, CAST(ITEM($0, 2)):BIGINT)], 
status=[CASE(AND(=(CAST(ITEM($0, 1)):BIGINT, 2101), >(-(20170302, 
CAST(ITEM($0, 2)):BIGINT), 1)), 'D', =(CAST(ITEM($0, 2)):BIGINT, 2101), 
'P', >(-(CAST(ITEM($0, 1)):BIGINT, CAST(ITEM($0, 2)):BIGINT), 1), '0', 
'A')])
00-02Scan(groupscan=[EasyGroupScan 
[selectionRoot=maprfs:/tmp/order_sample.csv, numFiles=1, columns=[`columns`[1], 
`columns`[2]], files=[maprfs:///tmp/order_sample.csv]]])

// Details of Java compiler from sys.options
0: jdbc:drill:schema=dfs.tmp> select name, status from sys.options where name 
like '%java_compiler%';
++--+
|  name  |  status  |
++--+
| exec.java.compiler.exp_in_method_size  | DEFAULT  |
| exec.java_compiler | DEFAULT  |
| exec.java_compiler_debug   | DEFAULT  |
| exec.java_compiler_janino_maxsize  | DEFAULT  |
++--+
4 rows selected (0.21 seconds)

{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [ANNOUNCE] New Committer: Laurent Goujon

[Khurram Faraaz]

Congratulations Laurent.


From: Parth Chandra 
Sent: Friday, June 9, 2017 3:14:00 AM
To: dev@drill.apache.org
Subject: [ANNOUNCE] New Committer: Laurent Goujon

The Project Management Committee (PMC) for Apache Drill has invited Laurent
Goujon to become a committer, and we are pleased to announce that he has
accepted.

Laurent has a long list of contributions many in the client side interfaces
and metadata queries.

Welcome Laurent, and thank you for your contributions.  Keep up the good
work !

- Parth
(on behalf of the Apache Drill PMC)

[jira] [Created] (DRILL-5576) OutOfMemoryException when some CPU cores are taken offline while concurrent queries are under execution

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5576:
-

 Summary: OutOfMemoryException when some CPU cores are taken 
offline while concurrent queries are under execution
 Key: DRILL-5576
 URL: https://issues.apache.org/jira/browse/DRILL-5576
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.11.0
 Environment: 3 nodes CentOS cluster
Reporter: Khurram Faraaz


When we reduce the number of available CPU cores while concurrent queries are 
under execution we see an OOM.

Drill 1.11.0 commit ID: d11aba2
three node CentOS 6.8 cluster
On each of the nodes Drill's direct memory was set to
export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-"16G"}

There are 24 cores on the node where foreman Drillbit is under execution.
{noformat}
[root@centos-01 logs]# lscpu
Architecture:  x86_64
CPU op-mode(s):32-bit, 64-bit
Byte Order:Little Endian
CPU(s):24
On-line CPU(s) list:   0,2,4,5,8,9,12,14,15,18,20,22
Off-line CPU(s) list:  1,3,6,7,10,11,13,16,17,19,21,23
Thread(s) per core:1
Core(s) per socket:4
Socket(s): 2
NUMA node(s):  2
Vendor ID: GenuineIntel
CPU family:6
Model: 44
Model name:Intel(R) Xeon(R) CPU   E5645  @ 2.40GHz
Stepping:  2
CPU MHz:   1600.000
BogoMIPS:  4799.86
Virtualization:VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache:  256K
L3 cache:  12288K
NUMA node0 CPU(s): 0,2,4,5,12,14,15
NUMA node1 CPU(s): 8,9,18,20,22
{noformat}

Java code snippet that creates threads and executes TPC-DS query 11 concurrently
{noformat}
ExecutorService executor = Executors.newFixedThreadPool(48);
try {
for (int i = 1; i <= 48; i++) {
executor.submit(new ConcurrentQuery(conn));
}
} catch (Exception e) {
System.out.println(e.getMessage());
e.printStackTrace();
}
{noformat}

While the TPC-DS Query 11 is under execution using above program, we take half 
of the available CPU cores offline
{noformat}
[root@centos-01 ~]# sh turnCPUCoresOffline.sh
OFFLINE cores are :
1,3,6-7,10-11,13,16-17,19,21,23
ONLINE cores are :
0,2,4-5,8-9,12,14-15,18,20,22
{noformat}

The result is we see an OutOfMemoryException, drillbit.log files are attached.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5573) Query profiles missing from Web UI and physical *.sys.drill file missing for query that was in CANCELLATION_REQUESTED state

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5573:
-

 Summary: Query profiles missing from Web UI and physical 
*.sys.drill file missing for query that was in CANCELLATION_REQUESTED state
 Key: DRILL-5573
 URL: https://issues.apache.org/jira/browse/DRILL-5573
 Project: Apache Drill
  Issue Type: Bug
  Components: Web Server
Affects Versions: 1.11.0
 Environment: 3 node CentOS cluster
Reporter: Khurram Faraaz


A query that was in CANCELLATION_REQUESTED state after the Java concurrent 
program was terminated by issuing Ctrl-C from the command line.
Next, stop and start Drillbits (Drill 1.11.0 commit id: d11aba2) on all nodes, 
and we see the below error message on Drill's Web UI, and we do not find the 
physical file .sys.drill under drill-1.11.0/logs/profiles folder
clush -a /opt/mapr/drill/drill-1.11.0/bin/drillbit.sh stop
clush -a /opt/mapr/drill/drill-1.11.0/bin/drillbit.sh start

http://:8047/profiles/26c9073d-a8ff-17b0-5a86-34042bdfb494

{
  "errorMessage" : "VALIDATION ERROR: No profile with given query id 
'26c9073d-a8ff-17b0-5a86-34042bdfb494' exists. Please verify the query 
id.\n\n\n[Error Id: c8f1ed5c-d93f-42b7-840d-54a1ed8da132 ]"
}

Stack trace from drillbit.log from foreman node, for the above error id.
{noformat}
2017-06-07 10:18:37,612 [main] INFO  o.a.drill.exec.server.rest.WebServer - 
Setting up HTTP connector for web server
2017-06-07 10:18:38,601 [main] INFO  o.apache.drill.exec.server.Drillbit - 
Startup completed (4149 ms).
2017-06-07 10:20:03,718 [qtp1575043618-89] INFO  
o.a.d.e.s.r.profile.ProfileResources - User Error Occurred: No profile with 
given query id '26c9073d-a8ff-17b0-5a86-34042bdfb494' exists. Please verify the 
query id.
org.apache.drill.common.exceptions.UserException: VALIDATION ERROR: No profile 
with given query id '26c9073d-a8ff-17b0-5a86-34042bdfb494' exists. Please 
verify the query id.


[Error Id: c8f1ed5c-d93f-42b7-840d-54a1ed8da132 ]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
 ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.server.rest.profile.ProfileResources.getQueryProfile(ProfileResources.java:288)
 [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.server.rest.profile.ProfileResources.getProfile(ProfileResources.java:310)
 [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[na:1.8.0_91]
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[na:1.8.0_91]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[na:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
at 
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
 [jersey-server-2.8.jar:na]
at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
 [jersey-server-2.8.jar:na]
at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
 [jersey-server-2.8.jar:na]
at 
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
 [jersey-server-2.8.jar:na]
at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
 [jersey-server-2.8.jar:na]
at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387)
 [jersey-server-2.8.jar:na]
at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331)
 [jersey-server-2.8.jar:na]
at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103)
 [jersey-server-2.8.jar:na]
at 
org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) 
[jersey-server-2.8.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) 
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) 
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) 
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) 
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) 
[jersey-common-2.8.jar:na]
at 
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
 [jers

[jira] [Created] (DRILL-5569) NullPointerException

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5569:
-

 Summary: NullPointerException
 Key: DRILL-5569
 URL: https://issues.apache.org/jira/browse/DRILL-5569
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - Parquet
Affects Versions: 1.11.0
Reporter: Khurram Faraaz


The below Exception was seen when TPC-DS Query 4 was executed against Drill 
1.11.0

Drill 1.11.0 git commit ID: d11aba2
[root@centos-01 mapr]# cat MapRBuildVersion
5.2.1.42646.GA

Stack trace from drillbit.log
{noformat}
2017-06-06 07:46:43,160 [Drillbit-ShutdownHook#0] WARN  
o.apache.drill.exec.work.WorkManager - Closing WorkManager but there are 80 
running fragments.
2017-06-06 07:46:43,207 [Drillbit-ShutdownHook#0] INFO  
o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 959, cache miss 
count: 12, hit rate: 99%
2017-06-06 07:46:43,504 [scan-3] ERROR o.a.d.e.u.f.BufferedDirectBufInputStream 
- Error reading from stream 1_1_0.parquet. Error was : Error reading out of an 
FSDataInputStream using the Hadoop 2 ByteBuffer based read method.
2017-06-06 07:46:43,510 [scan-8] ERROR o.a.d.e.u.f.BufferedDirectBufInputStream 
- Error reading from stream 1_1_0.parquet. Error was : Error reading out of an 
FSDataInputStream using the Hadoop 2 ByteBuffer based read method.
2017-06-06 07:46:43,514 [scan-8] INFO  o.a.d.e.s.p.c.AsyncPageReader - User 
Error Occurred: Exception occurred while reading from disk. 
(java.io.IOException: Error reading out of an FSDataInputStream using the 
Hadoop 2 ByteBuffer based read method.)
org.apache.drill.common.exceptions.UserException: DATA_READ ERROR: Exception 
occurred while reading from disk.

File:  /drill/testdata/tpcds_sf1/parquet/store_sales/1_1_0.parquet
Column:  ss_ext_list_price
Row Group Start:  75660513

[Error Id: 3a758095-fcc4-4364-a50b-33a027c1beb6 ]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
 ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.handleAndThrowException(AsyncPageReader.java:199)
 [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.access$600(AsyncPageReader.java:81)
 [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$AsyncPageReaderTask.call(AsyncPageReader.java:483)
 [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$AsyncPageReaderTask.call(AsyncPageReader.java:392)
 [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[na:1.8.0_65]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_65]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: java.io.IOException: java.io.IOException: Error reading out of an 
FSDataInputStream using the Hadoop 2 ByteBuffer based read method.
at 
org.apache.drill.exec.util.filereader.BufferedDirectBufInputStream.getNextBlock(BufferedDirectBufInputStream.java:185)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.util.filereader.BufferedDirectBufInputStream.readInternal(BufferedDirectBufInputStream.java:212)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.util.filereader.BufferedDirectBufInputStream.read(BufferedDirectBufInputStream.java:277)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.util.filereader.DirectBufInputStream.getNext(DirectBufInputStream.java:111)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$AsyncPageReaderTask.call(AsyncPageReader.java:437)
 [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
... 5 common frames omitted
Caused by: java.io.IOException: Error reading out of an FSDataInputStream using 
the Hadoop 2 ByteBuffer based read method.
at 
org.apache.parquet.hadoop.util.CompatibilityUtil.getBuf(CompatibilityUtil.java:99)
 ~[parquet-hadoop-1.8.1-drill-r0.jar:1.8.1-drill-r0]
at 
org.apache.drill.exec.util.filereader.BufferedDirectBufInputStream.getNextBlock(BufferedDirectBufInputStream.java:182)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
... 9 common frames omitted
Caused by: java.lang.NullPointerException: null
at 
com.mapr.fs.MapRFsInStream.readIntoDirectByteBuffer(MapRFsInStream.java:219) 
~[maprfs-5.2.1-mapr.jar:5.2.1-mapr]
at com.mapr.fs.MapRFsInStream.read(MapRFsInStream.java:333) 
~[maprfs-5.2.1-mapr.jar

[jira] [Created] (DRILL-5570) InterruptedException: null

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5570:
-

 Summary: InterruptedException: null
 Key: DRILL-5570
 URL: https://issues.apache.org/jira/browse/DRILL-5570
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.11.0
 Environment: 3 node CentOS cluster
Reporter: Khurram Faraaz


When TPC-DS query11 was executed concurrently and one of the non-foreman 
Drillbits was stopped (./bin/drillbit.sh stop)
we see the below system error InterruptedException in the drillbit.log of the 
non-foreman node

Drill 1.11.0 git commit ID: d11aba2
[root@centos-01 mapr]# cat MapRBuildVersion
5.2.1.42646.GA

{noformat}
2017-06-06 07:46:44,288 [26c9a242-dfa1-35be-b5f1-ff6b4fa66086:frag:11:0] ERROR 
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: InterruptedException

Fragment 11:0

[Error Id: 40723399-8983-4777-a2bb-dc9d55ae338e on centos-02.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
InterruptedException

Fragment 11:0

[Error Id: 40723399-8983-4777-a2bb-dc9d55ae338e on centos-02.qa.lab:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
 ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:295)
 [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
 [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264)
 [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_65]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: 
Interrupted but context.shouldContinue() is true
at 
org.apache.drill.exec.work.batch.BaseRawBatchBuffer.getNext(BaseRawBatchBuffer.java:178)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.unorderedreceiver.UnorderedReceiverBatch.getNextBatch(UnorderedReceiverBatch.java:141)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.unorderedreceiver.UnorderedReceiverBatch.next(UnorderedReceiverBatch.java:159)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105) 
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext(PartitionSenderRootExec.java:144)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95) 
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:234)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:227)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method) 
~[na:1.8.0_65]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_65]
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
 ~[hadoop-common-2.7.0-mapr-1607.jar:na]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:227)
 [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
... 4 common frames omitted
Caused by: java.lang.InterruptedException: null
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
 ~[na:1.8.0_65]
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
 ~[na:1.8.0_65]
at 
java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
 ~[na:1.8.0_65]
at 
java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680) 
~[na:1.8.0_65]
at 
org.apache.drill.exec.work.batch.UnlimitedRawBatchBuffer$UnlimitedBufferQueue.take

[jira] [Created] (DRILL-5566) AssertionError: Internal error: invariant violated: call to wrong operator

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5566:
-

 Summary: AssertionError: Internal error: invariant violated: call 
to wrong operator
 Key: DRILL-5566
 URL: https://issues.apache.org/jira/browse/DRILL-5566
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Khurram Faraaz


CHARACTER_LENGTH is a non-reserved keyword as per the SQL specification. It is 
a monadic function that accepts exactly one operand or parameter.

{noformat}
 ::=

  | 
  | 
  | 
  | 
  ...
  ...

 ::=

  | 
 ::=
  { CHAR_LENGTH | CHARACTER_LENGTH }  
  [ USING  ] 
...
...
 ::=
CHARACTERS
  | OCTETS  
{noformat}

Drill reports an assertion error in drillbit.log when character_length function 
is used in a SQL query.
{noformat}
0: jdbc:drill:schema=dfs.tmp> select character_length(cast('hello' as 
varchar(10))) col1 from (values(1));
Error: SYSTEM ERROR: AssertionError: Internal error: invariant violated: call 
to wrong operator


[Error Id: 49198839-5a1b-4786-9257-59739b27d2a8 on centos-01.qa.lab:31010]

  (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception 
during fragment initialization: Internal error: invariant violated: call to 
wrong operator
org.apache.drill.exec.work.foreman.Foreman.run():297
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745
Caused By (java.lang.AssertionError) Internal error: invariant violated: call 
to wrong operator
org.apache.calcite.util.Util.newInternal():777
org.apache.calcite.util.Util.permAssert():885
org.apache.calcite.sql2rel.ReflectiveConvertletTable$3.convertCall():219
org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall():59
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit():4148
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit():3581
org.apache.calcite.sql.SqlCall.accept():130

org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression():4040
org.apache.calcite.sql2rel.StandardConvertletTable$8.convertCall():185
org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall():59
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit():4148
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit():3581
org.apache.calcite.sql.SqlCall.accept():130

org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression():4040
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectList():3411
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl():612
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect():568
org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive():2773
org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery():522
org.apache.drill.exec.planner.sql.SqlConverter.toRel():269

org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRel():623

org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert():195
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():164
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():131
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():79
org.apache.drill.exec.work.foreman.Foreman.runSQL():1050
org.apache.drill.exec.work.foreman.Foreman.run():280
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745 (state=,code=0)
{noformat}

Calcite supports character_length function
{noformat}
[root@centos-0170 csv]# ./sqlline
sqlline version 1.1.9
sqlline> !connect jdbc:calcite:model=target/test-classes/model.json admin admin
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further 
details.
0: jdbc:calcite:model=target/test-classes/mod> select 
character_length(cast('hello' as varchar(10))) col1 from (values(1));
++
|COL1|
++
| 5  |
++
1 row selected (1.379 seconds)
{noformat}

Postgres 9.3 also supports character_length function
{noformat}
postgres=# select character_length(cast('hello' as varchar(10))) col1 from 
(values(1)) foo;
 col1 
--
5
(1 row)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5564) IllegalStateException: allocator[op:21:1:5:HashJoinPOP]: buffer space (16674816) + prealloc space (0) + child space (0) != allocated (16740352)

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5564:
-

 Summary: IllegalStateException: allocator[op:21:1:5:HashJoinPOP]: 
buffer space (16674816) + prealloc space (0) + child space (0) != allocated 
(16740352)
 Key: DRILL-5564
 URL: https://issues.apache.org/jira/browse/DRILL-5564
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.11.0
 Environment: 3 node CentOS cluster
Reporter: Khurram Faraaz


Run a concurrent Java program that executes TPCDS query11
while the above concurrent java program is under execution
stop foreman Drillbit (from another shell, using below command)
./bin/drillbit.sh stop
and you will see the IllegalStateException: allocator[op:21:1:5:HashJoinPOP]:  
and another assertion error, in the drillbit.log
AssertionError: Failure while stopping processing for operator id 10. Currently 
have states of processing:false, setup:false, waiting:true.   

Drill 1.11.0 git commit ID: d11aba2 (with assertions enabled)
 
details from drillbit.log from the foreman Drillbit node.
{noformat}
2017-06-05 18:38:33,838 [26ca5afa-7f6d-991b-1fdf-6196faddc229:frag:23:1] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 
26ca5afa-7f6d-991b-1fdf-6196faddc229:23:1: State change requested RUNNING --> 
FAILED
2017-06-05 18:38:33,849 [26ca5afa-7f6d-991b-1fdf-6196faddc229:frag:23:1] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 
26ca5afa-7f6d-991b-1fdf-6196faddc229:23:1: State change requested FAILED --> 
FINISHED
2017-06-05 18:38:33,852 [26ca5afa-7f6d-991b-1fdf-6196faddc229:frag:23:1] ERROR 
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: AssertionError: Failure 
while stopping processing for operator id 10. Currently have states of 
processing:false, setup:false, waiting:true.

Fragment 23:1

[Error Id: a116b326-43ed-4569-a20e-a10ba03d215e on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: AssertionError: 
Failure while stopping processing for operator id 10. Currently have states of 
processing:false, setup:false, waiting:true.

Fragment 23:1

[Error Id: a116b326-43ed-4569-a20e-a10ba03d215e on centos-01.qa.lab:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
 ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:295)
 [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
 [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264)
 [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_91]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: java.lang.RuntimeException: java.lang.AssertionError: Failure while 
stopping processing for operator id 10. Currently have states of 
processing:false, setup:false, waiting:true.
at 
org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:101)
 ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:409)
 [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:250)
 [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
... 4 common frames omitted
Caused by: java.lang.AssertionError: Failure while stopping processing for 
operator id 10. Currently have states of processing:false, setup:false, 
waiting:true.
at 
org.apache.drill.exec.ops.OperatorStats.stopProcessing(OperatorStats.java:167) 
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:255) 
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.inn

[jira] [Created] (DRILL-5563) Stop non foreman Drillbit results in IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5563:
-

 Summary: Stop non foreman Drillbit results in 
IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.
 Key: DRILL-5563
 URL: https://issues.apache.org/jira/browse/DRILL-5563
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.11.0
 Environment: 3 node CentOS cluster
Reporter: Khurram Faraaz


Stopping the non-foreman Drillbit normally (as shown below) results in 
IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.

/opt/mapr/drill/drill-1.11.0/bin/drillbit.sh stop

Drill 1.11.0 commit ID: d11aba2

Details from drillbit.log
{noformat}
Mon Jun  5 09:29:09 UTC 2017 Terminating drillbit pid 28182
2017-06-05 09:29:09,651 [Drillbit-ShutdownHook#0] INFO  
o.apache.drill.exec.server.Drillbit - Received shutdown request.
2017-06-05 09:29:11,691 [pool-6-thread-1] INFO  
o.a.drill.exec.rpc.user.UserServer - closed eventLoopGroup 
io.netty.channel.nio.NioEventLoopGroup@55511dc2 in 1004 ms
2017-06-05 09:29:11,691 [pool-6-thread-2] INFO  
o.a.drill.exec.rpc.data.DataServer - closed eventLoopGroup 
io.netty.channel.nio.NioEventLoopGroup@4078d750 in 1004 ms
2017-06-05 09:29:11,692 [pool-6-thread-1] INFO  
o.a.drill.exec.service.ServiceEngine - closed userServer in 1005 ms
2017-06-05 09:29:11,692 [pool-6-thread-2] INFO  
o.a.drill.exec.service.ServiceEngine - closed dataPool in 1005 ms
2017-06-05 09:29:11,701 [Drillbit-ShutdownHook#0] INFO  
o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 21, cache miss 
count: 7, hit rate: 67%
2017-06-05 09:29:11,709 [Drillbit-ShutdownHook#0] ERROR 
o.a.d.exec.server.BootStrapContext - Error while closing
java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding child 
allocators.
Allocator(ROOT) 0/800/201359872/17179869184 (res/actual/peak/limit)
  child allocators: 4
Allocator(frag:3:2) 200/0/0/200 (res/actual/peak/limit)
  child allocators: 0
  ledgers: 0
  reservations: 0
Allocator(frag:4:2) 200/0/0/200 (res/actual/peak/limit)
  child allocators: 0
  ledgers: 0
  reservations: 0
Allocator(frag:1:2) 200/0/0/200 (res/actual/peak/limit)
  child allocators: 0
  ledgers: 0
  reservations: 0
Allocator(frag:2:2) 200/0/0/200 (res/actual/peak/limit)
  child allocators: 0
  ledgers: 0
  reservations: 0
  ledgers: 0
  reservations: 0

at 
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:492) 
~[drill-memory-base-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) 
[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) 
[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:247) 
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) 
[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) 
[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:159) 
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:253) 
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
2017-06-05 09:29:11,709 [Drillbit-ShutdownHook#0] INFO  
o.apache.drill.exec.server.Drillbit - Shutdown completed (2057 ms).
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5561) IllegalArgumentException: Self-suppression not permitted, seen when foreman Drillbit is killed

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5561:
-

 Summary: IllegalArgumentException: Self-suppression not permitted, 
seen when foreman Drillbit is killed
 Key: DRILL-5561
 URL: https://issues.apache.org/jira/browse/DRILL-5561
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.11.0
 Environment: 3 node Centos cluster
Reporter: Khurram Faraaz
Priority: Minor



Drill 1.11.0 running on three nodes

kill foreman Drillbit, [root@centos-01 ~]# kill -9 15852; while a long running 
SQL query (TPCDS Query 11) is under execution

drillbit.log from centos-01.qa.lab (foreman Drillbit) has no Exceptions or 
Errors logged in it.

snippet of drillbit.log from centos-02.qa.lab ( non foreman Drillbit )

{noformat}
2017-05-30 08:21:32,834 [BitClient-1] INFO  o.a.d.e.w.fragment.FragmentExecutor 
- 26d2d481-d949-1de8-9497-c1492f086501:12:0: State change requested RUNNING --> 
FAILED
2017-05-30 08:21:32,834 [BitServer-9] WARN  o.a.d.exec.rpc.RpcExceptionHandler 
- Exception occurred with closed channel.  Connection: /10.10.100.202:31012 
<--> /10.10.100.201:50773 (data server)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_65]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) 
~[na:1.8.0_65]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) 
~[na:1.8.0_65]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_65]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) 
~[na:1.8.0_65]
at 
io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311)
 ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
at io.netty.buffer.WrappedByteBuf.setBytes(WrappedByteBuf.java:407) 
~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.buffer.UnsafeDirectLittleEndian.setBytes(UnsafeDirectLittleEndian.java:30)
 ~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at io.netty.buffer.DrillBuf.setBytes(DrillBuf.java:770) 
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at 
io.netty.buffer.MutableWrappedByteBuf.setBytes(MutableWrappedByteBuf.java:280) 
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at 
io.netty.buffer.ExpandableByteBuf.setBytes(ExpandableByteBuf.java:26) 
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881) 
~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241)
 ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
 ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) 
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
 [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) 
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) 
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
 [netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
2017-05-30 08:21:32,848 [BitClient-1] INFO  o.a.d.e.w.fragment.FragmentExecutor 
- 26d2d481-d949-1de8-9497-c1492f086501:16:0: State change requested RUNNING --> 
FAILED
2017-05-30 08:21:32,850 [BitClient-1] INFO  o.a.d.e.w.fragment.FragmentExecutor 
- 26d2d481-d949-1de8-9497-c1492f086501:14:0: State change requested RUNNING --> 
FAILED
2017-05-30 08:21:32,851 [BitClient-1] WARN  
o.apache.drill.exec.rpc.RequestIdMap - Failure while attempting to fail rpc 
response.
java.lang.IllegalArgumentException: Self-suppression not permitted
at java.lang.Throwable.addSuppressed(Throwable.java:1043) ~[na:1.8.0_65]
at 
org.apache.drill.common.DeferredException.addException(DeferredException.java:88)
 ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:97)
 ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:409)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.access$700(FragmentExecutor.java:55)
 ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.F

[jira] [Created] (DRILL-5543) Inner join query results in IndexOutOfBoundsException

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5543:
-

 Summary: Inner join query results in IndexOutOfBoundsException
 Key: DRILL-5543
 URL: https://issues.apache.org/jira/browse/DRILL-5543
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Khurram Faraaz


Inner join query results in IndexOutOfBoundsException

Steps to reproduce the IOB.
Copy data from drill test framework../framework/resources/Datasources/joins 
directory to /drill/testdata/joins directory
Run the below SQL.

Apache Drill 1.11.0 commit ID: d11aba2e
MapR version : 5.2.1.42646.GA
[test@centos-ab xyzb]# uname -r
2.6.32-642.el6.x86_64

Three node CentOS cluster

{noformat}
0: jdbc:drill:schema=dfs.tmp> use dfs.joins;
+---++
|  ok   |summary |
+---++
| true  | Default schema changed to [dfs.joins]  |
+---++
1 row selected (0.279 seconds)

0: jdbc:drill:schema=dfs.tmp> select
. . . . . . . . . . . . . . > count(*)
. . . . . . . . . . . . . . > from
. . . . . . . . . . . . . . > j1 INNER JOIN j4 ON
. . . . . . . . . . . . . . > (
. . . . . . . . . . . . . . >  j1.c_date = j4.c_date
. . . . . . . . . . . . . . > and j1.c_time = j4.c_time
. . . . . . . . . . . . . . > and j1.c_timestamp = j4.c_timestamp
. . . . . . . . . . . . . . > and j1.c_boolean = j4.c_boolean
. . . . . . . . . . . . . . > and j1.c_varchar = j4.c_varchar
. . . . . . . . . . . . . . > and j1.c_float = j4.c_float
. . . . . . . . . . . . . . > and j1.c_double = j4.c_double
. . . . . . . . . . . . . . > and j1.d9 = j4.d9
. . . . . . . . . . . . . . > and j1.d18 = j4.d18
. . . . . . . . . . . . . . >
. . . . . . . . . . . . . . > )
. . . . . . . . . . . . . . > where j1.c_date IN (select c_date from j2);
Error: SYSTEM ERROR: IndexOutOfBoundsException: index: 9, length: 1 (expected: 
range(0, 0))

Fragment 0:0

[Error Id: f8323164-581e-4377-87e0-f44e0bc1d037 on centos-01.qa.lab:31010] 
(state=,code=0)
{noformat}

Stack trace from drillbit.log

{noformat}
2017-05-26 08:47:05,151 [26d8147b-394f-3fea-f1e1-ece75d9b3f41:frag:0:0] ERROR 
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IndexOutOfBoundsException: 
index: 9, length: 1 (expected: range(0, 0))

Fragment 0:0

[Error Id: f8323164-581e-4377-87e0-f44e0bc1d037 on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
IndexOutOfBoundsException: index: 9, length: 1 (expected: range(0, 0))

Fragment 0:0

[Error Id: f8323164-581e-4377-87e0-f44e0bc1d037 on centos-01.qa.lab:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
 ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:295)
 [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
 [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264)
 [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_91]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: java.lang.IndexOutOfBoundsException: index: 9, length: 1 (expected: 
range(0, 0))
at io.netty.buffer.DrillBuf.checkIndexD(DrillBuf.java:123) 
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at io.netty.buffer.DrillBuf.chk(DrillBuf.java:147) 
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at io.netty.buffer.DrillBuf.getByte(DrillBuf.java:775) 
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at 
org.apache.drill.exec.vector.UInt1Vector$Accessor.get(UInt1Vector.java:354) 
~[vector-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.vector.NullableDateVector$Accessor.isSet(NullableDateVector.java:394)
 ~[vector-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at 
org.apache.drill.exec.test.generated.StreamingAggregatorGen383.outputRecordKeysPrev(StreamingAggTemplate.java:186)
 ~[na:na]
at 
org.apache.drill.exec.test.generated.StreamingAggregatorGen383.outputToBatchPrev(StreamingAggTemplate.java:322)
 ~[na:na]
at 
org.apache.dril

Re: [ANNOUNCE] New Committer: Paul Rogers

[Khurram Faraaz]

Congratulations, Paul!


From: Bridget Bevens 
Sent: Friday, May 19, 2017 10:29:29 PM
To: dev
Subject: Re: [ANNOUNCE] New Committer: Paul Rogers

Congratulations, Paul!


From: Jinfeng Ni 
Sent: Friday, May 19, 2017 9:57:35 AM
To: dev
Subject: Re: [ANNOUNCE] New Committer: Paul Rogers

Congratulations, Paul!


On Fri, May 19, 2017 at 9:36 AM, Aman Bawa  wrote:

> Congratulations, Paul!
>
> On 5/19/17, 8:22 AM, "Aman Sinha"  wrote:
>
> The Project Management Committee (PMC) for Apache Drill has invited
> Paul
> Rogers to become a committer, and we are pleased to announce that he
> has
> accepted.
>
> Paul has a long list of contributions that have touched many aspects
> of the
> product.
>
> Welcome Paul, and thank you for your contributions.  Keep up the good
> work !
>
> - Aman
>
> (on behalf of the Apache Drill PMC)
>
>
>