[jira] [Created] (DRILL-7275) CTAS + CTE query fails with IllegalStateException: Read batch count [%d] should be greater than zero [0]
Khurram Faraaz created DRILL-7275:
-
Summary: CTAS + CTE query fails with IllegalStateException: Read
batch count [%d] should be greater than zero [0]
Key: DRILL-7275
URL: https://issues.apache.org/jira/browse/DRILL-7275
Project: Apache Drill
Issue Type: Bug
Components: Storage - Parquet
Affects Versions: 1.15.0
Reporter: Khurram Faraaz
CTAS + CTE query fails with IllegalStateException: Read batch count [%d] should
be greater than zero [0]
Precondition check fails on line 47 in VarLenFixedEntryReader.java
{noformat}
44 final int expectedDataLen = columnPrecInfo.precision;
45 final int entrySz = 4 + columnPrecInfo.precision;
46 final int readBatch = getFixedLengthMaxRecordsToRead(valuesToRead, entrySz);
47 Preconditions.checkState(readBatch > 0, "Read batch count [%d] should be
greater than zero", readBatch);
{noformat}
Stack trace from drillbit.log, also has the failing query.
{noformat}
2019-05-13 14:40:14,090 [23268c40-ef3a-6349-5901-5762f6888971:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query with id
23268c40-ef3a-6349-5901-5762f6888971 issued by scoop_stc: CREATE TABLE
TEST_TEMPLATE_SCHEMA_creid.tbl_c_EquityProxyDailyReturn AS
WITH
che AS (
SELECT * FROM
TEST_TEMPLATE_SCHEMA_creid.tbl_c_CompositeHierarchyEntry_TimeVarying
WHERE CompositeHierarchyName = 'AxiomaRegion/AxiomaSector/VectorUniverse'
AND state = 'DupesRemoved'
AND CompositeLevel = 'AxiomaRegion_1/AxiomaSector_1/VectorUniverse_0'
),
ef AS (SELECT * FROM
TEST_TEMPLATE_SCHEMA_creid.tbl_c_EquityDailyReturn_FXAdjusted WHERE Status =
'PresentInRawData'),
d AS (SELECT * FROM TEST_TEMPLATE_SCHEMA_creid.tbl_r_BusinessDate WHERE
IsWeekday),
x AS
(
SELECT
che.CompositeHierarchyName,
che.State,
che.CompositeNodeName,
d.`Date` AS RecordDate,
COUNT(che.CompositeNodeName) AS countDistinctConstituents,
COUNT(ef.VectorListingId) AS countDataPoints,
AVG(ef.DailyReturn) AS AvgReturn,
AVG(ef.DailyReturnUSD) AS AvgReturnUSD,
AVG(ef.NotionalReturnUSD) AS AvgNotionalReturnUSD
FROM d
INNER JOIN che ON d.`Date` BETWEEN che.CompositeUltimateChildStartDate AND
che.CompositeUltimateChildEndDate
LEFT OUTER JOIN ef ON d.`Date` = ef.RecordDate AND 'VectorListingId_' ||
CAST(ef.VectorListingId AS VARCHAR(100)) = che.UltimateChild
GROUP BY che.CompositeHierarchyName, che.State, che.CompositeNodeName,
d.`Date`, d.IsWeekday, d.IsHoliday
)
SELECT * FROM x
2019-05-13 14:40:16,971 [23268c40-ef3a-6349-5901-5762f6888971:foreman] INFO
o.a.d.e.p.s.h.CreateTableHandler - Creating persistent table
[tbl_c_EquityProxyDailyReturn].
...
...
2019-05-13 14:40:20,036 [23268c40-ef3a-6349-5901-5762f6888971:frag:6:10] INFO
o.a.d.exec.physical.impl.ScanBatch - User Error Occurred: Error in parquet
record reader.
Message:
Hadoop path: /DEV/tbl_c_EquityDailyReturn_FXAdjusted/1_32_32.parquet
Total records read: 0
Row group index: 0
Records in row group: 3243
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {
optional int64 VectorListingId;
optional int32 RecordDate (DATE);
required binary Status (UTF8);
required binary CurrencyISO (UTF8);
optional double DailyReturn;
optional double DailyReturnUSD;
optional double NotionalReturnUSD;
}
, metadata: \{drill-writer.version=2, drill.version=1.15.0.0-mapr}}, blocks:
[BlockMetaData\{3243, 204762 [ColumnMetaData{UNCOMPRESSED [VectorListingId]
optional int64 VectorListingId [RLE, BIT_PACKED, PLAIN], 4},
ColumnMetaData\{UNCOMPRESSED [RecordDate] optional int32 RecordDate (DATE)
[RLE, BIT_PACKED, PLAIN], 26021}, ColumnMetaData\{UNCOMPRESSED [Status]
required binary Status (UTF8) [BIT_PACKED, PLAIN], 39050},
ColumnMetaData\{UNCOMPRESSED [CurrencyISO] required binary CurrencyISO (UTF8)
[BIT_PACKED, PLAIN], 103968}, ColumnMetaData\{UNCOMPRESSED [DailyReturn]
optional double DailyReturn [RLE, BIT_PACKED, PLAIN], 126715},
ColumnMetaData\{UNCOMPRESSED [DailyReturnUSD] optional double DailyReturnUSD
[RLE, BIT_PACKED, PLAIN], 152732}, ColumnMetaData\{UNCOMPRESSED
[NotionalReturnUSD] optional double NotionalReturnUSD [RLE, BIT_PACKED, PLAIN],
178749}]}]} (Error in parquet record reader.
...
...
Hadoop path: /DEV/tbl_c_EquityDailyReturn_FXAdjusted/1_32_32.parquet
Total records read: 0
Row group index: 0
Records in row group: 3243
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {
optional int64 VectorListingId;
optional int32 RecordDate (DATE);
required binary Status (UTF8);
required binary CurrencyISO (UTF8);
optional double DailyReturn;
optional double DailyReturnUSD;
optional double NotionalReturnUSD;
}
, metadata: \{drill-writer.version=2, drill.version=1.15.0.0-mapr}}, blocks:
[BlockMetaData\{3243, 204762 [ColumnMetaData{UNCOMPRESSED [VectorListingId]
optional int64 VectorListingId [RLE, BIT_PACKED, PLAIN], 4},
ColumnMetaData\{UNCOMPRESSED [RecordDate] optional int32 RecordDate (DATE)
[RLE,
[jira] [Created] (DRILL-7256) Query over empty Hive tables fails, we will need to print heap usagePercent details in error message
Khurram Faraaz created DRILL-7256:
-
Summary: Query over empty Hive tables fails, we will need to print
heap usagePercent details in error message
Key: DRILL-7256
URL: https://issues.apache.org/jira/browse/DRILL-7256
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.15.0
Reporter: Khurram Faraaz
The below query from Drill's web UI on Hive tables failed due to not enough
heap memory to run this query.
It fails intermittently from Drill web UI, and note that the two Hive tables
used in the query are empty, meaning they have no data in them. The query does
not fail when run from sqlline.
The error message does not provide information about the usagePercent of heap.
It will be useful to provide heap usagePercent information as part of the error
message in QueryWrapper.java when usagePercent > HEAP_MEMORY_FAILURE_THRESHOLD
Drill 1.15.0
Failing query.
{noformat}
SELECT a.event_id
FROM hive.cust_bhsf_ce_blob a, hive.t_fct_clinical_event b
where
a.event_id=b.event_id
and a.blob_contents not like '%dd:contenttype="TESTS"%'
and b.EVENT_RELATIONSHIP_CD='B'
and b.EVENT_CLASS_CD in ('DOC')
and b.entry_mode_cd='Web'
and b.RECORD_STATUS_CD='Active'
and b.RESULT_STATUS_CD ='Auth (Verified)'
and substring(b.valid_until_dt_tm,1,10) >='2017-12-30'
and substring(b.event_end_date,1,10) >='2018-01-01'
{noformat}
Stack trace from drillbit.log
{noformat}
2019-05-09 16:25:58,472 [qtp1934687-790] ERROR
o.a.d.e.server.rest.QueryResources - Query from Web UI Failed
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: There is not
enough heap memory to run this query using the web interface.
Please try a query with fewer columns or with a filter or limit condition to
limit the data returned.
You can also try an ODBC/JDBC client.
[Error Id: 91668f42-d88e-426b-b1fe-c0d042700500 ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.15.0.5-mapr.jar:1.15.0.5-mapr]
at org.apache.drill.exec.server.rest.QueryWrapper.run(QueryWrapper.java:103)
~[drill-java-exec-1.15.0.5-mapr.jar:1.15.0.5-mapr]
at
org.apache.drill.exec.server.rest.QueryResources.submitQueryJSON(QueryResources.java:72)
~[drill-java-exec-1.15.0.5-mapr.jar:1.15.0.5-mapr]
at
org.apache.drill.exec.server.rest.QueryResources.submitQuery(QueryResources.java:87)
~[drill-java-exec-1.15.0.5-mapr.jar:1.15.0.5-mapr]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[na:1.8.0_151]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103)
[jersey-server-2.8.jar:na]
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269)
[jersey-server-2.8.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
[jersey-common-2.8.jar:na]
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:252)
[jersey-server-2.8.jar:na]
at
[jira] [Created] (DRILL-7163) Join query fails with java.lang.IllegalArgumentException: null
Khurram Faraaz created DRILL-7163:
-
Summary: Join query fails with java.lang.IllegalArgumentException:
null
Key: DRILL-7163
URL: https://issues.apache.org/jira/browse/DRILL-7163
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.15.0
Reporter: Khurram Faraaz
Join query fails with java.lang.IllegalArgumentException: null
Drill : 1.15.0
Failing query is
{noformat}
Select *
From
(
select
convert_from(t.itm.iUUID, 'UTF8') iUUID,
convert_from(t.UPC.UPC14, 'UTF8') UPC14,
convert_from(t.itm.upcDesc, 'UTF8') upcDesc,
convert_from(t.ris.mstBrdOid, 'UTF8') mstBrdOid,
convert_from(t.ris.vrfLgyMtch, 'UTF8') vrfLgyMtch,
convert_from(t.itm.mtch.cfdMtch, 'UTF8') cfdMtch,
convert_from(t.itm.uoM, 'UTF8') uoM,
convert_from(t.uomRec.valVal, 'UTF8') uomVal,
case when a.iUUID is null then 0 else 1 end as keyind
from hbase.`/mapr/tables/item-master` t
left outer join
(
select distinct
convert_from(t.m.iUUID, 'UTF8') iUUID
from hbase.`/mapr/tables/items` t
) a
on t.itm.iUUID = a.iUUID
) i
where (i.mstBrdOid is null
or i.vrfLgyMtch is null)
and i.keyind=1
{noformat}
Stack trace from drillbit.log
{noformat}
2019-03-27 11:45:44,563 [23646564-3d23-f32b-6f68-11d7c4dd7a19:frag:1:0] ERROR
o.a.d.e.physical.impl.BaseRootExec - Batch dump started: dumping last 2 failed
batches
2019-03-27 11:45:44,564 [23646564-3d23-f32b-6f68-11d7c4dd7a19:frag:1:0] ERROR
o.a.d.e.p.i.p.ProjectRecordBatch -
ProjectRecordBatch[projector=Projector[vector2=null, selectionVectorMode=NONE],
hasRemainder=false, remainderIndex=0, recordCount=0,
container=org.apache.drill.exec.record.VectorContainer@2133fd0e[recordCount =
0, schemaChanged = false, schema = BatchSchema [fields=[[`row_key`
(VARBINARY:REQUIRED)], [`clnDesc` (MAP:REQUIRED), children=([`bndlCnt`
(VARBINARY:OPTIONAL)], [`by` (VARBINARY:OPTIONAL)], [`desc`
(VARBINARY:OPTIONAL)], [`dt` (VARBINARY:OPTIONAL)], [`descExt`
(VARBINARY:OPTIONAL)])], [`dup` (MAP:REQUIRED), children=([`dupBy`
(VARBINARY:OPTIONAL)], [`dupDt` (VARBINARY:OPTIONAL)], [`duplicate`
(VARBINARY:OPTIONAL)], [`preferred` (VARBINARY:OPTIONAL)])], [`itm`
(MAP:REQUIRED), children=([`iUUID` (VARBINARY:OPTIONAL)], [`cfdLgyMtch`
(VARBINARY:OPTIONAL)], [`uoM` (VARBINARY:OPTIONAL)], [`upcCd`
(VARBINARY:OPTIONAL)], [`upcDesc` (VARBINARY:OPTIONAL)], [`promo`
(VARBINARY:OPTIONAL)])], [`lckSts` (MAP:REQUIRED), children=([`lckBy`
(VARBINARY:OPTIONAL)], [`lckDt` (VARBINARY:OPTIONAL)])], [`lgy` (MAP:REQUIRED),
children=([`lgyBr` (VARBINARY:OPTIONAL)])], [`obs` (MAP:REQUIRED),
children=([`POSFile` (VARBINARY:OPTIONAL)])], [`prmRec` (MAP:REQUIRED)], [`ris`
(MAP:REQUIRED), children=([`UPC` (VARBINARY:OPTIONAL)], [`brdDesc`
(VARBINARY:OPTIONAL)], [`brdExtDesc` (VARBINARY:OPTIONAL)], [`brdFamDesc`
(VARBINARY:OPTIONAL)], [`brdTypeCd` (VARBINARY:OPTIONAL)], [`flvDesc`
(VARBINARY:OPTIONAL)], [`mfgDesc` (VARBINARY:OPTIONAL)], [`modBy`
(VARBINARY:OPTIONAL)], [`modDt` (VARBINARY:OPTIONAL)], [`msaCatCd`
(VARBINARY:OPTIONAL)])], [`rjr` (MAP:REQUIRED)], [`uomRec` (MAP:REQUIRED),
children=([`valBy` (VARBINARY:OPTIONAL)], [`valDt` (VARBINARY:OPTIONAL)],
[`valVal` (VARBINARY:OPTIONAL)], [`recBy` (VARBINARY:OPTIONAL)], [`recDt`
(VARBINARY:OPTIONAL)], [`recRat` (VARBINARY:OPTIONAL)], [`recVal`
(VARBINARY:OPTIONAL)])], [`upc` (MAP:REQUIRED), children=([`UPC14`
(VARBINARY:OPTIONAL)], [`allUPCVar` (VARBINARY:OPTIONAL)])], [`$f12`
(VARBINARY:OPTIONAL)], [`iUUID` (VARCHAR:OPTIONAL)]], selectionVector=NONE],
wrappers = [org.apache.drill.exec.vector.VarBinaryVector@b23a384[field =
[`row_key` (VARBINARY:REQUIRED)], ...],
org.apache.drill.exec.vector.complex.MapVector@61c779ff,
org.apache.drill.exec.vector.complex.MapVector@575c0f96,
org.apache.drill.exec.vector.complex.MapVector@69b943fe,
org.apache.drill.exec.vector.complex.MapVector@7f90e2ce,
org.apache.drill.exec.vector.complex.MapVector@25c27442,
org.apache.drill.exec.vector.complex.MapVector@12d5ffd3,
org.apache.drill.exec.vector.complex.MapVector@3150f8c4,
org.apache.drill.exec.vector.complex.MapVector@49aefab2,
org.apache.drill.exec.vector.complex.MapVector@7f78e7a1,
org.apache.drill.exec.vector.complex.MapVector@426ea4fa,
org.apache.drill.exec.vector.complex.MapVector@74cee2ab,
org.apache.drill.exec.vector.NullableVarBinaryVector@4a0bfdea[field = [`$f12`
(VARBINARY:OPTIONAL)], ...],
org.apache.drill.exec.vector.NullableVarCharVector@72f64ee5[field = [`iUUID`
(VARCHAR:OPTIONAL)], ...]], ...]]
2019-03-27 11:45:44,565 [23646564-3d23-f32b-6f68-11d7c4dd7a19:frag:1:0] ERROR
o.a.d.e.p.impl.join.HashJoinBatch -
HashJoinBatch[container=org.apache.drill.exec.record.VectorContainer@45887d35[recordCount
= 0, schemaChanged = false, schema = BatchSchema [fields=[[`row_key`
(VARBINARY:REQUIRED)], [`clnDesc` (MAP:REQUIRED), children=([`bndlCnt`
(VARBINARY:OPTIONAL)], [`by` (VARBINARY:OPTIONAL)], [`desc`
(VARBINARY:OPTIONAL)],
[jira] [Created] (DRILL-7144) sqlline option : !set useLineContinuation false, fails with ParseException
Khurram Faraaz created DRILL-7144:
-
Summary: sqlline option : !set useLineContinuation false, fails
with ParseException
Key: DRILL-7144
URL: https://issues.apache.org/jira/browse/DRILL-7144
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.15.0, 1.13.0
Reporter: Khurram Faraaz
Assignee: Arina Ielchiieva
sqlline option does not work as intended. Returns ParseException instead.
!set useLineContinuation false
On mapr-drill-1.13.0 we hit the below Exception.
{noformat}
0: jdbc:drill:drillbit=drill-abcd-dev.dev.schw> !set useLineContinuation false
Error setting configuration: useLineContinuation:
java.lang.IllegalArgumentException: No method matching "setuseLineContinuation"
was found in sqlline.SqlLineOpts.
{noformat}
It does not work on drill-1.15.0-mapr-r1
git.branch=drill-1.15.0-mapr-r1
git.commit.id=ebc9fe49d4477b04701fdd81884d5a0b748a13ae
{noformat}
[test@test-ab bin]# ./sqlline -u
"jdbc:drill:schema=dfs.tmp;auth=MAPRSASL;drillbit=test-ab.qa.lab" -n mapr -p
mapr
Apache Drill 1.15.0.3-mapr
"Start your SQL engine."
0: jdbc:drill:schema=dfs.tmp> !set useLineContinuation false
0: jdbc:drill:schema=dfs.tmp> select * from sys.version
> select * from sys.memory
Error: PARSE ERROR: Encountered "select" at line 2, column 1.
Was expecting one of:
"ORDER" ...
"LIMIT" ...
"OFFSET" ...
"FETCH" ...
"NATURAL" ...
"JOIN" ...
"INNER" ...
"LEFT" ...
"RIGHT" ...
"FULL" ...
"CROSS" ...
"," ...
"OUTER" ...
"EXTEND" ...
"(" ...
"MATCH_RECOGNIZE" ...
"AS" ...
...
...
...
...
...
"TABLESAMPLE" ...
"WHERE" ...
"GROUP" ...
"HAVING" ...
"WINDOW" ...
"UNION" ...
"INTERSECT" ...
"EXCEPT" ...
"MINUS" ...
"." ...
"[" ...
SQL Query select * from sys.version
select * from sys.memory
^
[Error Id: 067d5402-b965-4660-8981-34491ab5a051 on test-ab.qa.lab:31010]
(state=,code=0)
{noformat}
{noformat}
[Error Id: 067d5402-b965-4660-8981-34491ab5a051 ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at org.apache.drill.exec.planner.sql.SqlConverter.parse(SqlConverter.java:185)
[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:138)
[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:110)
[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:76)
[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:584)
[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:272)
[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_151]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
Caused by: org.apache.calcite.sql.parser.SqlParseException: Encountered
"select" at line 2, column 1.
Was expecting one of:
"ORDER" ...
"LIMIT" ...
"OFFSET" ...
"FETCH" ...
...
"[" ...
at
org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.convertException(DrillParserImpl.java:350)
~[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at
org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.normalizeException(DrillParserImpl.java:131)
~[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:137)
~[calcite-core-1.17.0-drill-r2.jar:1.17.0-drill-r2]
at org.apache.calcite.sql.parser.SqlParser.parseStmt(SqlParser.java:162)
~[calcite-core-1.17.0-drill-r2.jar:1.17.0-drill-r2]
at org.apache.drill.exec.planner.sql.SqlConverter.parse(SqlConverter.java:177)
[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
... 8 common frames omitted
Caused by: org.apache.drill.exec.planner.sql.parser.impl.ParseException:
Encountered "select" at line 2, column 1.
Was expecting one of:
"ORDER" ...
"LIMIT" ...
"OFFSET" ...
"FETCH" ...
"NATURAL" ...
...
...
"[" ...
at
org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.generateParseException(DrillParserImpl.java:24076)
~[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at
org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.jj_consume_token(DrillParserImpl.java:23893)
~[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at
org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.SqlStmtEof(DrillParserImpl.java:899)
~[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at
[jira] [Created] (DRILL-7128) IllegalStateException: Read batch count [0] should be greater than zero
Khurram Faraaz created DRILL-7128:
-
Summary: IllegalStateException: Read batch count [0] should be
greater than zero
Key: DRILL-7128
URL: https://issues.apache.org/jira/browse/DRILL-7128
Project: Apache Drill
Issue Type: Bug
Components: Storage - Parquet
Affects Versions: 1.15.0
Reporter: Khurram Faraaz
Source table is a Hive table stored as parquet.
Issue is seen only when querying datacapturekey column, which is of VARCHAR
type.
Hive 2.3
MapR Drill : 1.15.0.0-mapr
commit id : 951ef156fb1025677a2ca2dcf84e11002bf4b513
{noformat}
0: jdbc:drill:drillbit=test.a.node1> describe bt_br_cc_invalid_leads ;
+-++--+
| COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
+-++--+
| wrapup | CHARACTER VARYING | YES |
| datacapturekey | CHARACTER VARYING | YES |
| leadgendate | CHARACTER VARYING | YES |
| crla1 | CHARACTER VARYING | YES |
| crla2 | CHARACTER VARYING | YES |
| invalid_lead | INTEGER | YES |
| destination_advertiser_vendor_name | CHARACTER VARYING | YES |
| source_program_key | CHARACTER VARYING | YES |
| publisher_publisher | CHARACTER VARYING | YES |
| areaname | CHARACTER VARYING | YES |
| data_abertura_ficha | CHARACTER VARYING | YES |
+-++--+
11 rows selected (1.85 seconds)
0: jdbc:drill:drillbit=test.a.node1>
// from the view definition, note that column datacapturekey is of type
VARVCHAR with precision 2000
{
"name" : "bt_br_cc_invalid_leads",
"sql" : "SELECT CAST(`wrapup` AS VARCHAR(2000)) AS `wrapup`,
CAST(`datacapturekey` AS VARCHAR(2000)) AS `datacapturekey`, CAST(`leadgendate`
AS VARCHAR(2000)) AS `leadgendate`, CAST(`crla1` AS VARCHAR(2000)) AS `crla1`,
CAST(`crla2` AS VARCHAR(2000)) AS `crla2`, CAST(`invalid_lead` AS INTEGER) AS
`invalid_lead`, CAST(`destination_advertiser_vendor_name` AS VARCHAR(2000)) AS
`destination_advertiser_vendor_name`, CAST(`source_program_key` AS
VARCHAR(2000)) AS `source_program_key`, CAST(`publisher_publisher` AS
VARCHAR(2000)) AS `publisher_publisher`, CAST(`areaname` AS VARCHAR(2000)) AS
`areaname`, CAST(`data_abertura_ficha` AS VARCHAR(2000)) AS
`data_abertura_ficha`\nFROM
`dfs`.`root`.`/user/bigtable/logs/hive/warehouse/bt_br_cc_invalid_leads`",
"fields" : [ {
"name" : "wrapup",
"type" : "VARCHAR",
"precision" : 2000,
"isNullable" : true
}, {
"name" : "datacapturekey",
"type" : "VARCHAR",
"precision" : 2000,
"isNullable" : true
...
...
// total number of rows in bt_br_cc_invalid_leads
0: jdbc:drill:drillbit=test.a.node1> select count(*) from
bt_br_cc_invalid_leads ;
+-+
| EXPR$0 |
+-+
| 20599 |
+-+
1 row selected (0.173 seconds)
{noformat}
Stack trace from drillbit.log
{noformat}
2019-03-18 12:19:01,610 [237010da-6eda-a913-0424-32f63fbe01be:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query with id
237010da-6eda-a913-0424-32f63fbe01be issued by bigtable: SELECT
`bt_br_cc_invalid_leads`.`datacapturekey` AS `datacapturekey`
FROM `dfs.drill_views`.`bt_br_cc_invalid_leads` `bt_br_cc_invalid_leads`
GROUP BY `bt_br_cc_invalid_leads`.`datacapturekey`
2019-03-18 12:19:02,495 [237010da-6eda-a913-0424-32f63fbe01be:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 237010da-6eda-a913-0424-32f63fbe01be:0:0:
State change requested AWAITING_ALLOCATION --> RUNNING
2019-03-18 12:19:02,495 [237010da-6eda-a913-0424-32f63fbe01be:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 237010da-6eda-a913-0424-32f63fbe01be:0:0:
State to report: RUNNING
2019-03-18 12:19:02,502 [237010da-6eda-a913-0424-32f63fbe01be:frag:0:0] INFO
o.a.d.exec.physical.impl.ScanBatch - User Error Occurred: Error in parquet
record reader.
Message:
Hadoop path: /user/bigtable/logs/hive/warehouse/bt_br_cc_invalid_leads/08_0
Total records read: 0
Row group index: 0
Records in row group: 1551
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message hive_schema {
optional binary wrapup (UTF8);
optional binary datacapturekey (UTF8);
optional binary leadgendate (UTF8);
optional binary crla1 (UTF8);
optional binary crla2 (UTF8);
optional binary invalid_lead (UTF8);
optional binary destination_advertiser_vendor_name (UTF8);
optional binary source_program_key (UTF8);
optional binary publisher_publisher (UTF8);
optional binary areaname (UTF8);
optional binary data_abertura_ficha (UTF8);
}
, metadata: {}}, blocks: [BlockMetaData\{1551, 139906
[ColumnMetaData{UNCOMPRESSED [wrapup] optional binary wrapup (UTF8)
[PLAIN_DICTIONARY, RLE, BIT_PACKED], 4}, ColumnMetaData\{UNCOMPRESSED
[datacapturekey] optional binary datacapturekey (UTF8) [RLE, PLAIN,
BIT_PACKED], 656}, ColumnMetaData\{UNCOMPRESSED [leadgendate] optional binary
leadgendate (UTF8) [PLAIN_DICTIONARY, RLE, BIT_PACKED], 23978},
[jira] [Created] (DRILL-7100) parquet RecordBatchSizerManager : IllegalArgumentException: the requested size must be non-negative
Khurram Faraaz created DRILL-7100:
-
Summary: parquet RecordBatchSizerManager :
IllegalArgumentException: the requested size must be non-negative
Key: DRILL-7100
URL: https://issues.apache.org/jira/browse/DRILL-7100
Project: Apache Drill
Issue Type: Bug
Components: Storage - Parquet
Affects Versions: 1.15.0
Reporter: Khurram Faraaz
Table has string columns that can range from 1024 bytes to 32MB in length, we
should be able to handle such wide string columns in parquet, when querying
from Drill.
Hive Version 2.3.3
Drill Version 1.15
{noformat}
CREATE TABLE temp.cust_bhsf_ce_blob_parquet (
event_id DECIMAL,
valid_until_dt_tm string,
blob_seq_num DECIMAL,
valid_from_dt_tm string,
blob_length DECIMAL,
compression_cd DECIMAL,
blob_contents string,
updt_dt_tm string,
updt_id DECIMAL,
updt_task DECIMAL,
updt_cnt DECIMAL,
updt_applctx DECIMAL,
last_utc_ts string,
ccl_load_dt_tm string,
ccl_updt_dt_tm string )
STORED AS PARQUET;
{noformat}
The source table is stored as ORC format.
Failing query.
{noformat}
SELECT event_id, BLOB_CONTENTS FROM hive.temp.cust_bhsf_ce_blob_parquet WHERE
event_id = 3443236037
2019-03-07 14:40:17,886 [237e8c79-0e9b-45d6-9134-0da95dba462f:frag:1:269] INFO
o.a.d.exec.physical.impl.ScanBatch - User Error Occurred: the requested size
must be non-negative (the requested size must be non-negative)
org.apache.drill.common.exceptions.UserException: INTERNAL_ERROR ERROR: the
requested size must be non-negative
{noformat}
Snippet from drillbit.log file
{noformat}
[Error Id: 41a4d597-f54d-42a6-be6d-5dbeb7f642ba ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:293)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:126)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:116)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:126)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:116)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:69)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:297)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:284)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_181]
at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_181]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
[hadoop-common-2.7.0-mapr-1808.jar:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:284)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: java.lang.IllegalArgumentException: the requested size must be
non-negative
at
org.apache.drill.shaded.guava.com.google.common.base.Preconditions.checkArgument(Preconditions.java:135)
~[drill-shaded-guava-23.0.jar:23.0]
at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:224)
[jira] [Created] (DRILL-7061) Selecting option to limit results to 1000 on web UI causes parse error
Khurram Faraaz created DRILL-7061:
-
Summary: Selecting option to limit results to 1000 on web UI
causes parse error
Key: DRILL-7061
URL: https://issues.apache.org/jira/browse/DRILL-7061
Project: Apache Drill
Issue Type: Bug
Components: Web Server
Affects Versions: 1.16.0
Reporter: Khurram Faraaz
Attachments: image-2019-02-27-14-17-24-348.png
Selecting option to Limit results to 1,000 causes a parse error on web UI,
screen shot is attached. Browser used was Chrome.
Drill version => 1.16.0-SNAPSHOT
commit = e342ff5
Error reported on web UI when we press Submit button on web UI
{noformat}
Query Failed: An Error Occurred
org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR: 'LIMIT
start, count' is not allowed under the current SQL conformance level SQL Query
-- [autoLimit: 1,000 rows] select * from ( select length(varStr) from
dfs.`/root/many_json_files` ) limit 1,000 [Error Id:
e252d1cc-54d4-4530-837c-a1726a5be89f on qa102-45.qa.lab:31010]{noformat}
!image-2019-02-27-14-17-24-348.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-7023) Query fails with IndexOutOfBoundsException after upgrade from drill 1.13.0 to drill 1.14.0
Khurram Faraaz created DRILL-7023:
-
Summary: Query fails with IndexOutOfBoundsException after upgrade
from drill 1.13.0 to drill 1.14.0
Key: DRILL-7023
URL: https://issues.apache.org/jira/browse/DRILL-7023
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Query fails with IndexOutOfBoundsException after upgrade from drill 1.13.0 to
drill 1.14.0
{noformat}
2018-12-06 21:43:00,538 [23f5f79c-3777-eb37-ee46-f73be74381ef:frag:2:1] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IndexOutOfBoundsException
Fragment 2:1
[Error Id: 3b653503-b6da-4853-a395-317a169468ce on am1397.test.net:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IndexOutOfBoundsException
Fragment 2:1
[Error Id: 3b653503-b6da-4853-a395-317a169468ce on am1397.test.net:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361)
[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216)
[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327)
[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.14.0-mapr.jar:1.14.0-mapr]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_152]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
Caused by: java.lang.IndexOutOfBoundsException: null
at io.netty.buffer.DrillBuf.getBytes(DrillBuf.java:677)
~[drill-memory-base-1.14.0-mapr.jar:4.0.48.Final]
at org.apache.drill.exec.vector.BigIntVector.copyEntry(BigIntVector.java:389)
~[vector-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.test.generated.HashJoinProbeGen480.appendProbe(HashJoinProbeTemplate.java:190)
~[na:na]
at
org.apache.drill.exec.test.generated.HashJoinProbeGen480.outputOuterRow(HashJoinProbeTemplate.java:223)
~[na:na]
at
org.apache.drill.exec.test.generated.HashJoinProbeGen480.executeProbePhase(HashJoinProbeTemplate.java:357)
~[na:na]
at
org.apache.drill.exec.test.generated.HashJoinProbeGen480.probeAndProject(HashJoinProbeTemplate.java:400)
~[na:na]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.innerNext(HashJoinBatch.java:465)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:69)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:142)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
[jira] [Created] (DRILL-6994) TIMESTAMP type DOB column in Spark parquet is treated as VARBINARY in Drill
Khurram Faraaz created DRILL-6994:
-
Summary: TIMESTAMP type DOB column in Spark parquet is treated as
VARBINARY in Drill
Key: DRILL-6994
URL: https://issues.apache.org/jira/browse/DRILL-6994
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
A timestamp type column in a parquet file created from Spark is treated as
VARBINARY by Drill 1.14.0., Trying to cast DOB column to DATE results in an
Exception, although the monthOfYear field is in the allowed range.
Data used in the test
{noformat}
[test@md123 spark_data]# cat inferSchema_example.csv
Name,Department,years_of_experience,DOB
Sam,Software,5,1990-10-10
Alex,Data Analytics,3,1992-10-10
{noformat}
Create the parquet file using the above CSV file
{noformat}
[test@md123 bin]# ./spark-shell
19/01/22 21:21:34 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://md123.qa.lab:4040
Spark context available as 'sc' (master = local[*], app id =
local-1548192099796).
Spark session available as 'spark'.
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.3.1-mapr-SNAPSHOT
/_/
Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_191)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import org.apache.spark.sql.\{DataFrame, SQLContext}
import org.apache.spark.sql.\{DataFrame, SQLContext}
scala> import org.apache.spark.\{SparkConf, SparkContext}
import org.apache.spark.\{SparkConf, SparkContext}
scala> val sqlContext: SQLContext = new SQLContext(sc)
warning: there was one deprecation warning; re-run with -deprecation for details
sqlContext: org.apache.spark.sql.SQLContext =
org.apache.spark.sql.SQLContext@2e0163cb
scala> val df =
sqlContext.read.format("com.databricks.spark.csv").option("header",
"true").option("inferSchema", "true").load("/apps/inferSchema_example.csv")
df: org.apache.spark.sql.DataFrame = [Name: string, Department: string ... 2
more fields]
scala> df.printSchema
test
|-- Name: string (nullable = true)
|-- Department: string (nullable = true)
|-- years_of_experience: integer (nullable = true)
|-- DOB: timestamp (nullable = true)
scala> df.write.parquet("/apps/infer_schema_example.parquet")
// Read the parquet file
scala> val data = sqlContext.read.parquet("/apps/infer_schema_example.parquet")
data: org.apache.spark.sql.DataFrame = [Name: string, Department: string ... 2
more fields]
// Print the schema of the parquet file from Spark
scala> data.printSchema
test
|-- Name: string (nullable = true)
|-- Department: string (nullable = true)
|-- years_of_experience: integer (nullable = true)
|-- DOB: timestamp (nullable = true)
// Display the contents of parquet file on spark-shell
// register temp table and do a show on all records,to display.
scala> data.registerTempTable("employee")
warning: there was one deprecation warning; re-run with -deprecation for details
scala> val allrecords = sqlContext.sql("SELeCT * FROM employee")
allrecords: org.apache.spark.sql.DataFrame = [Name: string, Department: string
... 2 more fields]
scala> allrecords.show()
++--+---+---+
|Name| Department|years_of_experience| DOB|
++--+---+---+
| Sam| Software| 5|1990-10-10 00:00:00|
|Alex|Data Analytics| 3|1992-10-10 00:00:00|
++--+---+---+
{noformat}
Querying the parquet file from Drill 1.14.0-mapr, results in the DOB column
(timestamp type in Spark) being treated as VARBINARY.
{noformat}
apache drill 1.14.0-mapr
"a little sql for your nosql"
0: jdbc:drill:schema=dfs.tmp> select * from
dfs.`/apps/infer_schema_example.parquet`;
+---+-+--+--+
| Name | Department | years_of_experience | DOB |
+---+-+--+--+
| Sam | Software | 5 | [B@2bef51f2 |
| Alex | Data Analytics | 3 | [B@650eab8 |
+---+-+--+--+
2 rows selected (0.229 seconds)
// typeof(DOB) column returns a VARBINARY type, whereas the parquet schema in
Spark for DOB: timestamp (nullable = true)
0: jdbc:drill:schema=dfs.tmp> select typeof(DOB) from
dfs.`/apps/infer_schema_example.parquet`;
++
| EXPR$0 |
++
| VARBINARY |
| VARBINARY |
++
2 rows selected (0.199 seconds)
{noformat}
// CAST to DATE type results in Exception, though the monthOfYear is in the
range [1,12]
{noformat}
0: jdbc:drill:schema=dfs.tmp> select cast(DOB as DATE) from
dfs.`/apps/infer_schema_example.parquet`;
Error: SYSTEM ERROR: IllegalFieldValueException: Value 0 for monthOfYear must
be in the range [1,12]
Fragment
[jira] [Created] (DRILL-6990) IllegalStateException: The current reader doesn't support getting next information
Khurram Faraaz created DRILL-6990:
-
Summary: IllegalStateException: The current reader doesn't support
getting next information
Key: DRILL-6990
URL: https://issues.apache.org/jira/browse/DRILL-6990
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Attachments: parqt_nestedArray.parquet.tar
Reading a parquet file created from Spark, returns IllegalStateException: The
current reader doesn't support getting next information
Drill 1.14.0, parquet file created from Spark is attached here.
//Steps to create parquet file from Spark 2.3.1
[root@ba102-495 ~]# cd /opt/mapr/spark/spark-2.3.1
[root@ba102-495 spark-2.3.1]# cd bin
[root@ba102-495 bin]# ./spark-shell
19/01/21 22:57:05 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://qa102-45.qa.lab:4040
Spark context available as 'sc' (master = local[*], app id =
local-1548111430809).
Spark session available as 'spark'.
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.3.1-mapr-SNAPSHOT
/_/
Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_191)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import spark.implicits._
import spark.implicits._
scala> val df = spark.read.json("/apps/nestedDataJson.json")
df: org.apache.spark.sql.DataFrame = [id: bigint, nested_array:
array>]
scala> df.write.parquet("/apps/parqt_nestedArray.parquet")
Data used in test
{noformat}
[root@ba102-495 ~]# cat nestedDataJson.json
{"id":19,"nested_array":[[1,2,3,4],[5,6,7,8],[9,10,12]]}
{"id":14121,"nested_array":[[1,3,4],[5,6,8],[9,11,12]]}
{"id":18894,"nested_array":[[1,3,4],[5,6,7,8],[9,10,11,12]]}
{"id":12499,"nested_array":[[1,4],[5,7,8],[9,11,12]]}
{"id":120,"nested_array":[[1,4],[5,7,8],[9,10,11,12]]}
{"id":12,"nested_array":[[1,2,3,4],[5,6,7,8],[11,12]]}
{"id":13,"nested_array":[[1,2,3,4],[5,8],[9,10,11,12]]}
{"id":14,"nested_array":[[1,2,3,4],[5,68],[9,10,11,12]]}
{"id":123,"nested_array":[[1,2,3,4],[5,8],[9,10,11,12]]}
{"id":124,"nested_array":[[1,2,4],[5,6,7,8],[9,10,11,12]]}
{"id":134,"nested_array":[[1,4],[5,8],[9,12]]}
{noformat}
>From drillbit.log
{noformat}
Query Failed: An Error Occurred
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
IllegalStateException: The current reader doesn't support getting next
information. Fragment 0:0 [Error Id: c16c70dd-6565-463f-83a7-118ccd8442e2 on
ba102-495.qa.lab:31010]
...
...
2019-01-21 23:08:11,268 [23b9af24-10b9-ad11-5583-ecc3e0c562e6:frag:0:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: The
current reader doesn't support getting next information.
Fragment 0:0
[Error Id: c16c70dd-6565-463f-83a7-118ccd8442e2 on ba102-495.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: The current reader doesn't support getting next
information.
Fragment 0:0
[Error Id: c16c70dd-6565-463f-83a7-118ccd8442e2 on ba102-495.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361)
[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216)
[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327)
[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.14.0-mapr.jar:1.14.0-mapr]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: java.lang.IllegalStateException: The current reader doesn't support
getting next information.
at
org.apache.drill.exec.vector.complex.impl.AbstractBaseReader.next(AbstractBaseReader.java:64)
~[vector-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.vector.complex.impl.SingleMapReaderImpl.next(SingleMapReaderImpl.java:31)
~[vector-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.test.generated.ProjectorGen971.doEval(ProjectorTemplate.java:35)
~[na:na]
at
org.apache.drill.exec.test.generated.ProjectorGen971.projectRecords(ProjectorTemplate.java:67)
~[na:na]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:231)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
[jira] [Created] (DRILL-6979) Add autofocus attribute to username on login page, and to query textbox on Query tab.
Khurram Faraaz created DRILL-6979: - Summary: Add autofocus attribute to username on login page, and to query textbox on Query tab. Key: DRILL-6979 URL: https://issues.apache.org/jira/browse/DRILL-6979 Project: Apache Drill Issue Type: Improvement Components: Web Server Affects Versions: 1.16.0 Reporter: Khurram Faraaz Assignee: Khurram Faraaz Add autofocus attribute to username on login page, and to query textbox on Query tab. The two text boxes that need the change are in these files ./exec/java-exec/src/main/resources/rest/query/query.ftl ./exec/java-exec/src/main/resources/rest/login.ftl -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6961) Error Occurred: Cannot connect to the db. query INFORMATION_SCHEMA.VIEWS : Maybe you have incorrect connection params or db unavailable now (timeout)
Khurram Faraaz created DRILL-6961:
-
Summary: Error Occurred: Cannot connect to the db. query
INFORMATION_SCHEMA.VIEWS : Maybe you have incorrect connection params or db
unavailable now (timeout)
Key: DRILL-6961
URL: https://issues.apache.org/jira/browse/DRILL-6961
Project: Apache Drill
Issue Type: Improvement
Components: Storage - Information Schema
Affects Versions: 1.13.0
Reporter: Khurram Faraaz
Trying to query drill information_schema.views table returns error. Disabling
openTSDB plugin resolves the problem.
Drill 1.13.0
Failing query :
{noformat}
SELECT TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, VIEW_DEFINITION FROM
INFORMATION_SCHEMA.`VIEWS` where VIEW_DEFINITION not like 'kraken';
{noformat}
Stack Trace from drillbit.log
{noformat}
2019-01-07 15:36:21,975 [23cc39aa-2618-e9f0-e77e-4fafa6edc314:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query id
23cc39aa-2618-e9f0-e77e-4fafa6edc314: SELECT TABLE_CATALOG, TABLE_SCHEMA,
TABLE_NAME, VIEW_DEFINITION FROM INFORMATION_SCHEMA.`VIEWS` where
VIEW_DEFINITION not like 'kraken'
2019-01-07 15:36:35,221 [23cc39aa-2618-e9f0-e77e-4fafa6edc314:frag:0:0] INFO
o.a.d.e.s.o.c.services.ServiceImpl - User Error Occurred: Cannot connect to the
db. Maybe you have incorrect connection params or db unavailable now (timeout)
org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Cannot
connect to the db. Maybe you have incorrect connection params or db unavailable
now
[Error Id: f8b4c074-ba62-4691-b142-a8ea6e4f6b2a ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl.getTableNames(ServiceImpl.java:107)
[drill-opentsdb-storage-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl.getAllMetricNames(ServiceImpl.java:70)
[drill-opentsdb-storage-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory$OpenTSDBSchema.getTableNames(OpenTSDBSchemaFactory.java:78)
[drill-opentsdb-storage-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.calcite.jdbc.SimpleCalciteSchema.addImplicitTableToBuilder(SimpleCalciteSchema.java:106)
[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
at org.apache.calcite.jdbc.CalciteSchema.getTableNames(CalciteSchema.java:318)
[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
at
org.apache.calcite.jdbc.CalciteSchema$SchemaPlusImpl.getTableNames(CalciteSchema.java:587)
[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
at
org.apache.calcite.jdbc.CalciteSchema$SchemaPlusImpl.getTableNames(CalciteSchema.java:548)
[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
at
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.visitTables(InfoSchemaRecordGenerator.java:227)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:216)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:209)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:196)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.store.ischema.InfoSchemaTableType.getRecordReader(InfoSchemaTableType.java:58)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:34)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:30)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at org.apache.drill.exec.physical.impl.ImplCreator$2.run(ImplCreator.java:146)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at org.apache.drill.exec.physical.impl.ImplCreator$2.run(ImplCreator.java:142)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_144]
at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_144]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1633)
[hadoop-common-2.7.0-mapr-1710.jar:na]
at
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:142)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:182)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:137)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:182)
[jira] [Created] (DRILL-6942) Provide ability to sort list of profiles on Drill web UI
Khurram Faraaz created DRILL-6942: - Summary: Provide ability to sort list of profiles on Drill web UI Key: DRILL-6942 URL: https://issues.apache.org/jira/browse/DRILL-6942 Project: Apache Drill Issue Type: Improvement Components: Web Server Affects Versions: 1.15.0 Reporter: Khurram Faraaz We need to provide a way to sort the query profiles, on the profiles tab on Drill web UI. The use case is when users run many queries (several hundred queries), they want a way to list the queries that have taken the longest time (i.e. duration) to complete. All queries that completed, failed or were canceled and took very long, all such queries should be listed on the top in the profiles tab page on Drill web ui. An option to list the query profiles based on total duration should be provided to user. That way user can easily identify such long running queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6937) sys.functions table needs a fix in the names column
Khurram Faraaz created DRILL-6937:
-
Summary: sys.functions table needs a fix in the names column
Key: DRILL-6937
URL: https://issues.apache.org/jira/browse/DRILL-6937
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.15.0
Reporter: Khurram Faraaz
The function names in the name column of sys.functions in some cases, are the
operators, this is not the expected behavior, the name column should have
actual names and not the operators.
I am on Drill 1.15.0 commit : 8743e8f1e8d5bca4d67c94d07a8560ad356ff2b6
{noformat}
Apache Drill 1.15.0
"Data is the new oil. Ready to Drill some?"
0: jdbc:drill:schema=dfs.tmp> select count(*) from sys.functions;
+-+
| EXPR$0 |
+-+
| 2846 |
+-+
1 row selected (0.327 seconds)
0: jdbc:drill:schema=dfs.tmp>
{noformat}
{noformat}
0: jdbc:drill:schema=dfs.tmp> select distinct name from sys.functions limit 12;
++
| name |
++
| != |
| $sum0 |
| && |
| - |
| /int |
| < |
| <= |
| <> |
| = |
| == |
| > |
| >= |
++
12 rows selected (0.175 seconds)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6816) NPE - Concurrent query execution using PreparedStatement
Khurram Faraaz created DRILL-6816:
-
Summary: NPE - Concurrent query execution using PreparedStatement
Key: DRILL-6816
URL: https://issues.apache.org/jira/browse/DRILL-6816
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Concurrent query execution from JDBC program using PreparedStatement results in
NPE.
Queries that were executed concurrently are (part of a query file),
{noformat}
select id from `test_tbl.json`
select count(id) from `test_tbl.json`
select count(*) from `test_tbl.json`
select * from `test_tbl.json`
{noformat}
Drill 1.14.0
git.commit.id=35a1ae23c9b280b9e73cb0f6f01808c996515454
MapR version => 6.1.0.20180911143226.GA (secure cluster)
Executing the above queries concurrently using a Statement object results in
successful query execution.
{noformat}
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(query);
{noformat}
However, when the same queries listed above are executed using a
PreparedStatement object we see an NPE
{noformat}
PreparedStatement prdstmnt = conn.prepareStatement(query);
prdstmnt.executeUpdate();
{noformat}
Stack trace from drillbit.log
{noformat}
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1143)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1150)
at
org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:511)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:146)
at
org.apache.drill.jdbc.impl.DrillPreparedStatementImpl.executeLargeUpdate(DrillPreparedStatementImpl.java:512)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:142)
at RunQuery.executeQuery(RunQuery.java:61)
at RunQuery.run(RunQuery.java:30)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (DRILL-6563) TPCDS query 10 has regressed
[ https://issues.apache.org/jira/browse/DRILL-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Khurram Faraaz resolved DRILL-6563. --- Resolution: Fixed > TPCDS query 10 has regressed > - > > Key: DRILL-6563 > URL: https://issues.apache.org/jira/browse/DRILL-6563 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning Optimization >Affects Versions: 1.14.0 >Reporter: Khurram Faraaz >Assignee: Pritesh Maker >Priority: Major > Fix For: 1.15.0 > > Attachments: 24ca3c6c-90e1-a4bf-6c6f-3f981fa2d043.sys.drill, > query10.fast_plan_old_commit, tpcds_query_10_plan_slow_140d09e.pdf, > tpcds_query_plan_10_140d09e.txt > > > TPC-DS query 10 has regressed in performance from taking 3.5 seconds to > execute on Apache Drill 1.14.0 commit b92f599 , to 07 min 51.851 sec to > complete execution on Apache Drill 1.14.0 commit 140d09e. Query was executed > over SF1 parquet views on a 4 node cluster. > Query plan from old and newer commit is attached here, with the query profile > from newer commit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6602) Drill query cancellation should log the originator
Khurram Faraaz created DRILL-6602: - Summary: Drill query cancellation should log the originator Key: DRILL-6602 URL: https://issues.apache.org/jira/browse/DRILL-6602 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Reporter: Khurram Faraaz The Drill query Cancel functionality does not log the origin of the Cancel request. (1). security issue, we don't know who (which user) canceled the query. (2). debugging such a canceled query can be a pain to find the origin and the root cause of why a query was canceled. On Apache Drill 1.14.0 git.commit.id.abbrev=b0314a3, we have observed this problem, and it can be consistently reproduced. There is no information at all about the origin of why the query was canceled in the drillbit.log -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6595) IllegalAccessError: tried to access field org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files from class org.apache.drill.exec.store.pa
Khurram Faraaz created DRILL-6595:
-
Summary: IllegalAccessError: tried to access field
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files
from class org.apache.drill.exec.store.parquet.metadata.Metadata_V3
Key: DRILL-6595
URL: https://issues.apache.org/jira/browse/DRILL-6595
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
java.lang.IllegalAccessError reported in drillbit.out file
Apache Drill 1.14.0 git.commit.id.abbrev=b0314a3
{noformat}
Jul 11, 2018 1:33:39 PM
com.fasterxml.jackson.module.afterburner.deser.OptimizedSettableBeanProperty
_reportProblem
WARNING: Disabling Afterburner deserialization for class
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3
(field #1; mutator
com.fasterxml.jackson.module.afterburner.deser.SettableObjectFieldProperty),
due to access error (type java.lang.IllegalAccessError, message=tried to access
field
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files
from class
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer48257508)
java.lang.IllegalAccessError: tried to access field
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files
from class
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer48257508
at
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer48257508.objectField(org/apache/drill/exec/store/parquet/metadata/Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer.java)
at
com.fasterxml.jackson.module.afterburner.deser.SettableObjectFieldProperty.deserializeAndSet(SettableObjectFieldProperty.java:50)
at
com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288)
at
com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:189)
at
com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:120)
at
com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:130)
at
com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:97)
at
com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:254)
at
com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:68)
at
com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4001)
at
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3058)
at
org.apache.drill.exec.store.parquet.metadata.Metadata.readBlockMeta(Metadata.java:617)
at
org.apache.drill.exec.store.parquet.metadata.Metadata.readBlockMeta(Metadata.java:156)
at
org.apache.drill.exec.store.parquet.ParquetGroupScan.expandSelectionFromMetadataCache(ParquetGroupScan.java:374)
at
org.apache.drill.exec.store.parquet.ParquetGroupScan.expandIfNecessary(ParquetGroupScan.java:337)
at
org.apache.drill.exec.store.parquet.ParquetGroupScan.(ParquetGroupScan.java:121)
at
org.apache.drill.exec.store.parquet.ParquetGroupScan.(ParquetGroupScan.java:102)
at
org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(ParquetFormatPlugin.java:180)
at
org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(ParquetFormatPlugin.java:70)
at
org.apache.drill.exec.store.dfs.FileSystemPlugin.getPhysicalScan(FileSystemPlugin.java:136)
at
org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:114)
at
org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:109)
at
org.apache.drill.exec.planner.logical.DrillTable.getGroupScan(DrillTable.java:99)
at
org.apache.drill.exec.planner.logical.DrillPushProjectIntoScanRule.canPushProjectIntoScan(DrillPushProjectIntoScanRule.java:125)
at
org.apache.drill.exec.planner.logical.DrillPushProjectIntoScanRule.onMatch(DrillPushProjectIntoScanRule.java:81)
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:652)
at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:368)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:426)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:366)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRawDrel(DefaultSqlHandler.java:255)
at
[jira] [Created] (DRILL-6590) DATA_WRITE ERROR: Hash Join failed to write to output file: /tmp/drill/spill/24bac407
Khurram Faraaz created DRILL-6590:
-
Summary: DATA_WRITE ERROR: Hash Join failed to write to output
file: /tmp/drill/spill/24bac407
Key: DRILL-6590
URL: https://issues.apache.org/jira/browse/DRILL-6590
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Apache Drill 1.14.0 git.commit.id.abbrev=eb946b0
There was enough space on /tmp, however Hash Join failed to write to spill file
[test@qa102-45 drill-1.14.0]# clush -a df -h /tmp
: Filesystem Size Used Avail Use% Mounted on
: /dev/mapper/vg_root-lv_root 500G 150G 351G 30% /
: Filesystem Size Used Avail Use% Mounted on
: /dev/mapper/vg_root-lv_root 500G 17G 484G 4% /
: Filesystem Size Used Avail Use% Mounted on
: /dev/mapper/vg_root-lv_root 500G 14G 487G 3% /
: Filesystem Size Used Avail Use% Mounted on
: /dev/mapper/vg_root-lv_root 500G 13G 488G 3% /
Stack trace from drillbit.log
{noformat}
2018-07-10 18:17:51,953 [BitServer-10] WARN o.a.d.exec.rpc.control.WorkEventBus
- A fragment message arrived but there was no registered listener for that
message: profile {
state: FAILED
error {
error_id: "6e258de2-2d4f-4b48-967d-df1b329955cd"
endpoint {
address: "qa102-48.qa.lab"
user_port: 31010
control_port: 31011
data_port: 31012
version: "1.14.0-SNAPSHOT"
state: STARTUP
}
error_type: DATA_WRITE
message: "DATA_WRITE ERROR: Hash Join failed to write to output file:
/tmp/drill/spill/24bac407-2adb-5763-ed08-cb5714dca2c0_HashJoin_4-22-53/spill15_outer\n\nFragment
4:53\n\n[Error Id: 6e258de2-2d4f-4b48-967d-df1b329955cd on
qa102-48.qa.lab:31010]"
exception {
exception_class: "java.nio.channels.ClosedByInterruptException"
stack_trace {
class_name: "..."
line_number: 0
method_name: "..."
is_native_method: false
}
stack_trace {
class_name: "com.google.protobuf.CodedOutputStream"
file_name: "CodedOutputStream.java"
line_number: 833
method_name: "refreshBuffer"
is_native_method: false
}
stack_trace {
class_name: "com.google.protobuf.CodedOutputStream"
file_name: "CodedOutputStream.java"
line_number: 843
method_name: "flush"
is_native_method: false
}
stack_trace {
class_name: "com.google.protobuf.AbstractMessageLite"
file_name: "AbstractMessageLite.java"
line_number: 91
method_name: "writeDelimitedTo"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.cache.VectorSerializer$Writer"
file_name: "VectorSerializer.java"
line_number: 97
method_name: "write"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.common.HashPartition"
file_name: "HashPartition.java"
line_number: 346
method_name: "spillThisPartition"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.common.HashPartition"
file_name: "HashPartition.java"
line_number: 263
method_name: "completeABatch"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.common.HashPartition"
file_name: "HashPartition.java"
line_number: 237
method_name: "completeAnOuterBatch"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.common.HashPartition"
file_name: "HashPartition.java"
line_number: 232
method_name: "appendOuterRow"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.test.generated.HashJoinProbeGen49"
file_name: "HashJoinProbeTemplate.java"
line_number: 306
method_name: "executeProbePhase"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.test.generated.HashJoinProbeGen49"
file_name: "HashJoinProbeTemplate.java"
line_number: 393
method_name: "probeAndProject"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.join.HashJoinBatch"
file_name: "HashJoinBatch.java"
line_number: 357
method_name: "innerNext"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.record.AbstractRecordBatch"
file_name: "AbstractRecordBatch.java"
line_number: 172
method_name: "next"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.record.AbstractRecordBatch"
file_name: "AbstractRecordBatch.java"
line_number: 119
method_name: "next"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.join.HashJoinBatch"
file_name: "HashJoinBatch.java"
line_number: 274
method_name: "sniffNonEmptyBatch"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.join.HashJoinBatch"
file_name: "HashJoinBatch.java"
line_number: 236
method_name: "prefetchFirstBatchFromBothSides"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.join.HashJoinBatch"
file_name: "HashJoinBatch.java"
line_number: 216
method_name: "buildSchema"
is_native_method: false
[jira] [Created] (DRILL-6583) UI usability issue
Khurram Faraaz created DRILL-6583: - Summary: UI usability issue Key: DRILL-6583 URL: https://issues.apache.org/jira/browse/DRILL-6583 Project: Apache Drill Issue Type: Bug Components: Web Server Affects Versions: 1.14.0 Reporter: Khurram Faraaz Attachments: UI_usability_issue_AD_1_14_0.png When a query is under execution, on the web UI we see this text which is actually a set of different links that help navigate to different pages on the UI, below that query's profile. Apache Drill 1.14.0 git.commit.id.abbrev=f481a7c They all appear on a single line with no spacing and a typo, the formatting of the text for those links needs to be changed / improved. Attached is a screenshot for the issue, look for "FirstPrevious1NextLast" on the bottom left of the screenshot. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6563) TPCDS query 10 has regressed
Khurram Faraaz created DRILL-6563: - Summary: TPCDS query 10 has regressed Key: DRILL-6563 URL: https://issues.apache.org/jira/browse/DRILL-6563 Project: Apache Drill Issue Type: Bug Components: Query Planning Optimization Affects Versions: 1.14.0 Reporter: Khurram Faraaz Attachments: 24ca3c6c-90e1-a4bf-6c6f-3f981fa2d043.sys.drill, query10.fast_plan_old_commit, tpcds_query_10_plan_slow_140d09e.pdf, tpcds_query_plan_10_140d09e.txt TPC-DS query 10 has regressed in performance from taking 3.5 seconds to execute on Apache Drill 1.14.0 commit b92f599 , to 07 min 51.851 sec to complete execution on Apache Drill 1.14.0 commit 140d09e. Query was executed over SF1 parquet views on a 4 node cluster. Query plan from old and newer commit is attached here, with the query profile from newer commit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6548) IllegalStateException: Unexpected EMIT outcome received in buildSchema phase
Khurram Faraaz created DRILL-6548:
-
Summary: IllegalStateException: Unexpected EMIT outcome received
in buildSchema phase
Key: DRILL-6548
URL: https://issues.apache.org/jira/browse/DRILL-6548
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Assignee: Sorabh Hamirwasia
On a four node Apache Drill 1.14.0 master branch against TPC-DS SF1 parquet
data (parquet views)
git.commit.id.abbrev=b92f599
TPC-DS query 69 fails with IllegalStateException: Unexpected EMIT outcome
received in buildSchema phase
Failing query is,
{noformat}
2018-06-27 15:24:39,493 [24cbf157-e95c-42ab-7307-f75f5943a277:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query id
24cbf157-e95c-42ab-7307-f75f5943a277: SELECT cd_gender,
cd_marital_status,
cd_education_status,
Count(*) cnt1,
cd_purchase_estimate,
Count(*) cnt2,
cd_credit_rating,
FROM customer c,
customer_address ca,
customer_demographics
WHERE c.c_current_addr_sk = ca.ca_address_sk
AND ca_state IN ( 'KS', 'AZ', 'NE' )
AND cd_demo_sk = c.c_current_cdemo_sk
AND EXISTS (SELECT *
FROM store_sales,
date_dim
WHERE c.c_customer_sk = ss_customer_sk
AND ss_sold_date_sk = d_date_sk
AND d_year = 2004
AND d_moy BETWEEN 3 AND 3 + 2)
AND ( NOT EXISTS (SELECT *
FROM web_sales,
date_dim
WHERE c.c_customer_sk = ws_bill_customer_sk
AND ws_sold_date_sk = d_date_sk
AND d_year = 2004
AND d_moy BETWEEN 3 AND 3 + 2)
AND NOT EXISTS (SELECT *
FROM catalog_sales,
date_dim
WHERE c.c_customer_sk = cs_ship_customer_sk
AND cs_sold_date_sk = d_date_sk
AND d_year = 2004
AND d_moy BETWEEN 3 AND 3 + 2) )
GROUP BY cd_gender,
cd_marital_status,
cd_education_status,
cd_purchase_estimate,
cd_credit_rating
ORDER BY cd_gender,
cd_marital_status,
cd_education_status,
cd_purchase_estimate,
cd_credit_rating
cd_credit_rating
LIMIT 100
{noformat}
Stack trace from drillbit.log
{noformat}
2018-06-27 15:24:42,130 [24cbf157-e95c-42ab-7307-f75f5943a277:frag:0:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException:
Unexpected EMIT outcome received in buildSchema phase
Fragment 0:0
[Error Id: ba1a35e0-807e-4bab-b820-8aa6aad80e87 on qa102-45.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Unexpected EMIT outcome received in buildSchema phase
Fragment 0:0
[Error Id: ba1a35e0-807e-4bab-b820-8aa6aad80e87 on qa102-45.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.lang.IllegalStateException: Unexpected EMIT outcome received in
buildSchema phase
at
org.apache.drill.exec.physical.impl.TopN.TopNBatch.buildSchema(TopNBatch.java:178)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:152)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
[jira] [Created] (DRILL-6518) DESCRIBE command on Drill created parquet table does not return results
Khurram Faraaz created DRILL-6518:
-
Summary: DESCRIBE command on Drill created parquet table does not
return results
Key: DRILL-6518
URL: https://issues.apache.org/jira/browse/DRILL-6518
Project: Apache Drill
Issue Type: Bug
Components: Storage - Parquet
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Attachments: 0_0_0.parquet, item.drill.parquet_metadata
Describe command on a Drill (1.14.0) created parquet table, does not return the
table description.
Parquet file and parquet metadata cache file for item table is attached here,
it has the details of column types in the metadata cache file.
{noformat}
Apache Drill 1.14.0-SNAPSHOT
commit : b447260e49dc4a8c906f5b310c037fe6dd77166f
{noformat}
{noformat}
// DESCRIBE commands returns no information about the table.
0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> describe
dfs.`/drill/testdata/tpcds_sf1/parquet/item`;
+--++--+
| COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
+--++--+
+--++--+
No rows selected (0.221 seconds)
0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> refresh table metadata
dfs.`/drill/testdata/tpcds_sf1/parquet/item`;
+---+--+
| ok | summary |
+---+--+
| true | Successfully updated metadata for table
/drill/testdata/tpcds_sf1/parquet/item. |
+---+--+
1 row selected (0.173 seconds)
0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> describe
dfs.`/drill/testdata/tpcds_sf1/parquet/item`;
+--++--+
| COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
+--++--+
+--++--+
No rows selected (0.229 seconds)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6517) IllegalStateException: Record count not set for this vector container
Khurram Faraaz created DRILL-6517:
-
Summary: IllegalStateException: Record count not set for this
vector container
Key: DRILL-6517
URL: https://issues.apache.org/jira/browse/DRILL-6517
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Assignee: Padma Penumarthy
Attachments: 24d7b377-7589-7928-f34f-57d02061acef.sys.drill
TPC-DS query is Canceled after 2 hrs and 47 mins and we see an
IllegalStateException: Record count not set for this vector container, in
drillbit.log
Steps to reproduce the problem, query profile
(24d7b377-7589-7928-f34f-57d02061acef) is attached here.
{noformat}
In drill-env.sh set max direct memory to 12G on all 4 nodes in cluster
export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-"12G"}
and set these options from sqlline,
alter system set `planner.memory.max_query_memory_per_node` = 10737418240;
alter system set `drill.exec.hashagg.fallback.enabled` = true;
To run the query (replace IP-ADDRESS with your foreman node's IP address)
cd /opt/mapr/drill/drill-1.14.0/bin
./sqlline -u
"jdbc:drill:schema=dfs.tpcds_sf1_parquet_views;drillbit=" -f
/root/query72.sql
{noformat}
Stack trace from drillbit.log
{noformat}
2018-06-18 20:08:51,912 [24d7b377-7589-7928-f34f-57d02061acef:frag:4:49] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException:
Record count not set for this vector container
Fragment 4:49
[Error Id: 73177a1c-f7aa-4c9e-99e1-d6e1280e3f27 on qa102-45.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Record count not set for this vector container
Fragment 4:49
[Error Id: 73177a1c-f7aa-4c9e-99e1-d6e1280e3f27 on qa102-45.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.lang.IllegalStateException: Record count not set for this
vector container
at com.google.common.base.Preconditions.checkState(Preconditions.java:173)
~[guava-18.0.jar:na]
at
org.apache.drill.exec.record.VectorContainer.getRecordCount(VectorContainer.java:394)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.getRecordCount(RemovingRecordBatch.java:49)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.RecordBatchSizer.(RecordBatchSizer.java:690)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.RecordBatchSizer.(RecordBatchSizer.java:662)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.JoinBatchMemoryManager.update(JoinBatchMemoryManager.java:73)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.JoinBatchMemoryManager.update(JoinBatchMemoryManager.java:79)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.prefetchFirstBatchFromBothSides(HashJoinBatch.java:242)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.buildSchema(HashJoinBatch.java:218)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:152)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.sniffNonEmptyBatch(HashJoinBatch.java:276)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.prefetchFirstBatchFromBothSides(HashJoinBatch.java:238)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
[jira] [Created] (DRILL-6453) TPC-DS query 72 has regressed
Khurram Faraaz created DRILL-6453:
-
Summary: TPC-DS query 72 has regressed
Key: DRILL-6453
URL: https://issues.apache.org/jira/browse/DRILL-6453
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill
TPC-DS query 72 seems to have regressed, query profile for the case where it
Canceled after 2 hours on Drill 1.14.0 is attached here.
{noformat}
On, Drill 1.14.0-SNAPSHOT
commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took
around 55 seconds to execute)
SF1 parquet data on 4 nodes;
planner.memory.max_query_memory_per_node = 10737418240.
drill.exec.hashagg.fallback.enabled = true
TPC-DS query 72 executed successfully & took 47 seconds to complete execution.
{noformat}
{noformat}
TPC-DS data in the below run has date values stored as DATE datatype and not
VARCHAR type
On, Drill 1.14.0-SNAPSHOT
commit : 82e1a12
SF1 parquet data on 4 nodes;
planner.memory.max_query_memory_per_node = 10737418240.
drill.exec.hashagg.fallback.enabled = true
and
alter system set `exec.hashjoin.num_partitions` = 1;
TPC-DS query 72 executed for 2 hrs and 11 mins and did not complete, I had to
Cancel it by stopping the Foreman drillbit.
As a result several minor fragments are reported to be in
CANCELLATION_REQUESTED state on UI.
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6452) document steps to execute SQL queries from Postman (chrome extension) on Drill
Khurram Faraaz created DRILL-6452: - Summary: document steps to execute SQL queries from Postman (chrome extension) on Drill Key: DRILL-6452 URL: https://issues.apache.org/jira/browse/DRILL-6452 Project: Apache Drill Issue Type: Bug Components: Documentation Affects Versions: 1.14.0 Reporter: Khurram Faraaz We need documentation to list the steps with screen shots about executing SQL queries from Postman (chrome extension) on Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6441) IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.
Khurram Faraaz created DRILL-6441:
-
Summary: IllegalStateException: Allocator[ROOT] closed with
outstanding child allocators.
Key: DRILL-6441
URL: https://issues.apache.org/jira/browse/DRILL-6441
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
This is seen when the drillbit is shutdown using $DRILL_HOME/bin/drillbit.sh
stop, it happens when BaseAllocator tries to close.
Apache Drill 1.14.0 commit c6c5d27d91468a29656bee2acba55d3321978aab
{noformat}
2018-05-11 15:19:44,510 [2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:0:0:
State to report: FINISHED
Wed May 16 10:17:13 PDT 2018 Terminating drillbit pid 32076
2018-05-16 10:17:13,793 [Drillbit-ShutdownHook#0] INFO
o.apache.drill.exec.server.Drillbit - Received shutdown request.
2018-05-16 10:17:14,876 [Drillbit-ShutdownHook#0] INFO
o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 20, cache miss
count: 6, hit rate: 70%
2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] ERROR
o.a.d.exec.server.BootStrapContext - Error while closing
java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding child
allocators.
Allocator(ROOT) 0/0/9577600/34359738368 (res/actual/peak/limit)
child allocators: 8
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
ledgers: 0
reservations: 0
at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:496)
~[drill-memory-base-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:259)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:263)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:363)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] INFO
o.apache.drill.exec.server.Drillbit - Shutdown completed (1095 ms).
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6427) outputBatchSize is missing from the DEBUG output for HashJoinBatch operator
Khurram Faraaz created DRILL-6427:
-
Summary: outputBatchSize is missing from the DEBUG output for
HashJoinBatch operator
Key: DRILL-6427
URL: https://issues.apache.org/jira/browse/DRILL-6427
Project: Apache Drill
Issue Type: Bug
Reporter: Khurram Faraaz
Assignee: Padma Penumarthy
Drill 1.14.0-SNAPSHOT commit : f99d1f1323c0a5bac99842d6283d3025f3cb527f
outputBatchSize is missing from the DEBUG output for HashJoinBatch operator
Query used in test,
{noformat}
select count(*) from `twovarchar_asc_128MB.parquet` t1,
`twovarchar_asc_16MB.parquet` t2 WHERE t1.Varbinaryvalue = t2.Varbinaryvalue
{noformat}
Snippet from drillbit.log
{noformat}
2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG
o.a.d.e.p.impl.join.HashJoinBatch - left input: batch count : 1, avg batch
bytes : 90145920, avg row bytes : 8166, record count : 11040
2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG
o.a.d.e.p.impl.join.HashJoinBatch - right input: batch count : 1, avg batch
bytes : 10567808, avg row bytes : 7506, record count : 1408
2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG
o.a.d.e.p.impl.join.HashJoinBatch - output: batch count : 166, avg batch bytes
: 15951453, avg row bytes : 15672, record count : 168960
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6330) TPC-DS query 95 failed to plan
Khurram Faraaz created DRILL-6330:
-
Summary: TPC-DS query 95 failed to plan
Key: DRILL-6330
URL: https://issues.apache.org/jira/browse/DRILL-6330
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
TPC-DS query 95 planned and executed successfully on Drill 1.10.0 (git.commit
id : bbcf4b76), it fails to plan on 1.14.0 master on 4 node cluster, SF1
parquet data.
TPC-DS query 95 fails to plan on Drill 1.14.0
git.commit.id.abbrev=da24113
git.branch=master
{noformat}
[root@qa102-45 bin]# ./sqlline -u
"jdbc:drill:schema=dfs.tpcds_sf1_parquet_views;drillbit=" -f
~/tpcds_drill_1_14_0/query95.sql
1/2 WITH ws_wh AS
(
SELECT ws1.ws_order_number,
ws1.ws_warehouse_sk wh1,
ws2.ws_warehouse_sk wh2
FROM web_sales ws1,
web_sales ws2
WHERE ws1.ws_order_number = ws2.ws_order_number
AND ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk)
SELECT
Count(DISTINCT ws_order_number) AS `order count` ,
Sum(ws_ext_ship_cost) AS `total shipping cost` ,
Sum(ws_net_profit) AS `total net profit`
FROM web_sales ws1 ,
date_dim ,
customer_address ,
web_site
WHERE d_date BETWEEN '2000-4-01' AND (
Cast('2000-4-01' AS DATE) + INTERVAL '60' day)
AND ws1.ws_ship_date_sk = d_date_sk
AND ws1.ws_ship_addr_sk = ca_address_sk
AND ca_state = 'IN'
AND ws1.ws_web_site_sk = web_site_sk
AND web_company_name = 'pri'
AND ws1.ws_order_number IN
(
SELECT ws_order_number
FROM ws_wh)
AND ws1.ws_order_number IN
(
SELECT wr_order_number
FROM web_returns,
ws_wh
WHERE wr_order_number = ws_wh.ws_order_number)
ORDER BY count(DISTINCT ws_order_number)
LIMIT 100;
Error: UNSUPPORTED_OPERATION ERROR: This query cannot be planned possibly due
to either a cartesian join or an inequality join
[Error Id: d2ec3faf-1253-492e-bc56-80689e62335c on qa102-45.qa.lab:31010]
(org.apache.drill.exec.work.foreman.UnsupportedRelOperatorException) This query
cannot be planned possibly due to either a cartesian join or an inequality join
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToPrel():450
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():179
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():146
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():84
org.apache.drill.exec.work.foreman.Foreman.runSQL():567
org.apache.drill.exec.work.foreman.Foreman.run():264
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748 (state=,code=0)
{noformat}
>From the profile file under log/profiles directory, for TPC-DS query 95.
{noformat}
{"id":\{"part1":2679297541583898541,"part2":5048058602276109542},"type":1,"start":1523661076935,"end":1523661077389,"query":"WITH
ws_wh AS \n( \nSELECT ws1.ws_order_number, \nws1.ws_warehouse_sk wh1,
\nws2.ws_warehouse_sk wh2 \nFROM web_sales ws1, \nweb_sales ws2 \nWHERE
ws1.ws_order_number = ws2.ws_order_number \nAND ws1.ws_warehouse_sk <>
ws2.ws_warehouse_sk) \nSELECT \nCount(DISTINCT ws_order_number) AS `order
count` , \nSum(ws_ext_ship_cost) AS `total shipping cost` ,
\nSum(ws_net_profit) AS `total net profit` \nFROM web_sales ws1 , \ndate_dim ,
\ncustomer_address , \nweb_site \nWHERE d_date BETWEEN '2000-4-01' AND (
\nCast('2000-4-01' AS DATE) + INTERVAL '60' day) \nAND ws1.ws_ship_date_sk =
d_date_sk \nAND ws1.ws_ship_addr_sk = ca_address_sk \nAND ca_state = 'IN' \nAND
ws1.ws_web_site_sk = web_site_sk \nAND web_company_name = 'pri' \nAND
ws1.ws_order_number IN \n( \nSELECT ws_order_number \nFROM ws_wh) \nAND
ws1.ws_order_number IN \n( \nSELECT wr_order_number \nFROM web_returns, \nws_wh
\nWHERE wr_order_number = ws_wh.ws_order_number) \nORDER BY count(DISTINCT
ws_order_number) \nLIMIT
100","foreman":\{"address":"qa102-45.qa.lab","userPort":31010,"controlPort":31011,"dataPort":31012,"version":"1.14.0-SNAPSHOT","state":0},"state":4,"totalFragments":0,"finishedFragments":0,"user":"anonymous","error":"UNSUPPORTED_OPERATION
ERROR: This query cannot be planned possibly due to either a cartesian join or
an inequality join\n\n","verboseError":"UNSUPPORTED_OPERATION ERROR: This query
cannot be planned possibly due to either a cartesian join or an inequality
join\n\n\n\n
(org.apache.drill.exec.work.foreman.UnsupportedRelOperatorException) This query
cannot be planned possibly due to either a cartesian join or an inequality
join\n
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToPrel():450\n
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():179\n
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():146\n
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():84\n
org.apache.drill.exec.work.foreman.Foreman.runSQL():567\n
org.apache.drill.exec.work.foreman.Foreman.run():264\n
[jira] [Created] (DRILL-6329) TPC-DS Query 66 failed due to OOM
Khurram Faraaz created DRILL-6329:
-
Summary: TPC-DS Query 66 failed due to OOM
Key: DRILL-6329
URL: https://issues.apache.org/jira/browse/DRILL-6329
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Attachments: 252f0f20-2774-43d7-ec31-911ee0f5f330.sys.drill,
TPCDS_Query_66.sql, TPCDS_Query_66_PLAN.txt
TPC-DS Query 66 failed after 27 minutes on Drill 1.14.0 on a 4 node cluster
against SF1 parquet data (dfs.tpcds_sf1_parquet_views). Query 66 and the query
profile and the query plan are attached here.
This seems to be a regression, the same query worked fine on 1.10.0
On Drill 1.10.0 ( git.commit id : bbcf4b76) => 9.026 seconds (completed
successfully).
On Drill 1.14.0 ( git.commit.id.abbrev=da24113 ) query 66 failed after running
for 27 minutes, due to OutOfMemoryException
Stack trace from sqlline console, no stack trace was written to drillbit.log
{noformat}
Error: RESOURCE ERROR: One or more nodes ran out of memory while executing the
query.
Too little memory available
Fragment 2:0
[Error Id: 5636a939-a318-4b59-b3e8-9eb93f6b82f3 on qa102-45.qa.lab:31010]
(org.apache.drill.exec.exception.OutOfMemoryException) Too little memory
available
org.apache.drill.exec.test.generated.HashAggregatorGen7120.delayedSetup():409
org.apache.drill.exec.test.generated.HashAggregatorGen7120.doWork():579
org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():176
org.apache.drill.exec.record.AbstractRecordBatch.next():164
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():134
org.apache.drill.exec.record.AbstractRecordBatch.next():164
org.apache.drill.exec.physical.impl.BaseRootExec.next():105
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():93
org.apache.drill.exec.physical.impl.BaseRootExec.next():95
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():292
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():279
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():422
org.apache.hadoop.security.UserGroupInformation.doAs():1595
org.apache.drill.exec.work.fragment.FragmentExecutor.run():279
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748 (state=,code=0)
java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory
while executing the query.
Too little memory available
Fragment 2:0
[Error Id: 5636a939-a318-4b59-b3e8-9eb93f6b82f3 on qa102-45.qa.lab:31010]
(org.apache.drill.exec.exception.OutOfMemoryException) Too little memory
available
org.apache.drill.exec.test.generated.HashAggregatorGen7120.delayedSetup():409
org.apache.drill.exec.test.generated.HashAggregatorGen7120.doWork():579
org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():176
org.apache.drill.exec.record.AbstractRecordBatch.next():164
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():134
org.apache.drill.exec.record.AbstractRecordBatch.next():164
org.apache.drill.exec.physical.impl.BaseRootExec.next():105
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():93
org.apache.drill.exec.physical.impl.BaseRootExec.next():95
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():292
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():279
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():422
org.apache.hadoop.security.UserGroupInformation.doAs():1595
org.apache.drill.exec.work.fragment.FragmentExecutor.run():279
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
...
Caused by: org.apache.drill.common.exceptions.UserRemoteException: RESOURCE
ERROR: One or more nodes ran out of memory while executing the query.
Too little memory available
Fragment 2:0
[Error Id: 5636a939-a318-4b59-b3e8-9eb93f6b82f3 on qa102-45.qa.lab:31010]
(org.apache.drill.exec.exception.OutOfMemoryException) Too little memory
available
org.apache.drill.exec.test.generated.HashAggregatorGen7120.delayedSetup():409
org.apache.drill.exec.test.generated.HashAggregatorGen7120.doWork():579
[jira] [Created] (DRILL-6308) Exception handling - OOM
Khurram Faraaz created DRILL-6308:
-
Summary: Exception handling - OOM
Key: DRILL-6308
URL: https://issues.apache.org/jira/browse/DRILL-6308
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Assignee: Sorabh Hamirwasia
For RpcHandlers in Drill, when there is an OutOfMemory condition ,then all the
handlers call OutOfMemoryHandler with the caught exception. Every channel,
except for DataServer the default instance of MemoryHandler is used which
throws UnsupportedOperationException. This is misleading since actual exception
is for OutOfMemory not UnsupportedOperationException.
Here is an example where it is not handled the right way.
{noformat}
Caused by: java.lang.UnsupportedOperationException: null
at
org.apache.drill.exec.rpc.OutOfMemoryHandler$1.handle(OutOfMemoryHandler.java:25)
~[drill-rpc-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.rpc.SaslEncryptionHandler.encode(SaslEncryptionHandler.java:170)
~[drill-rpc-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.rpc.SaslEncryptionHandler.encode(SaslEncryptionHandler.java:44)
~[drill-rpc-1.13.0-mapr.jar:1.13.0-mapr]
at
io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88)
[netty-codec-4.0.48.Final.jar:4.0.48.Final]
... 25 common frames omitted{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6302) NPE in Drillbit.java in close method
Khurram Faraaz created DRILL-6302:
-
Summary: NPE in Drillbit.java in close method
Key: DRILL-6302
URL: https://issues.apache.org/jira/browse/DRILL-6302
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.13.0
Environment: git.commit.id=bb07ebbb9ba8742f44689f8bd8efb5853c5edea0
Reporter: Khurram Faraaz
Assignee: Sorabh Hamirwasia
registrationHandle in the close method of Drillbit.java (line 228) is null,
which causes an NPE and as a result the server does not close.
[https://github.com/mapr/private-drill/blob/drill-1.12.0-mapr/exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java#L228]
registrationHandle = coord.update(registrationHandle, State.QUIESCENT);
Stack trace from drillbit.log
{noformat}
/opt/mapr/drill/drill-1.12.0/logs/drillbit.out
{noformat}
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was
removed in 8.0
Exception in thread "main" java.lang.NullPointerException
at
org.apache.drill.exec.coord.zk.ZKClusterCoordinator.update(ZKClusterCoordinator.java:223)
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:228)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:401)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:372)
at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:368)
{noformat}{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6285) Failure while attempting to fail rpc response.
Khurram Faraaz created DRILL-6285:
-
Summary: Failure while attempting to fail rpc response.
Key: DRILL-6285
URL: https://issues.apache.org/jira/browse/DRILL-6285
Project: Apache Drill
Issue Type: Bug
Components: Execution - RPC
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
When a query is marked as Canceled on Web UI, there is this Exception in
drillbit.log
Drill 1.13.0 commit cac2882d5a9e22fbc251e4caf622fe30242ad557
{noformat}
2018-03-21 15:35:43,198 [UserServer-1] INFO o.a.d.e.w.fragment.FragmentExecutor
- 254d2796-7744-cc2a-d77e-5ec030b04211:0:0: State change requested RUNNING -->
FAILED
2018-03-21 15:35:43,198 [254d2796-7744-cc2a-d77e-5ec030b04211:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 254d2796-7744-cc2a-d77e-5ec030b04211:0:0:
State change requested FAILED --> FAILED
2018-03-21 15:35:43,199 [254d2796-7744-cc2a-d77e-5ec030b04211:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 254d2796-7744-cc2a-d77e-5ec030b04211:0:0:
State change requested FAILED --> FAILED
2018-03-21 15:35:43,202 [254d2796-7744-cc2a-d77e-5ec030b04211:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 254d2796-7744-cc2a-d77e-5ec030b04211:0:0:
State change requested FAILED --> FAILED
2018-03-21 15:35:43,202 [UserServer-1] WARN
o.apache.drill.exec.rpc.RequestIdMap - Failure while attempting to fail rpc
response.
java.lang.IllegalArgumentException: Self-suppression not permitted
at java.lang.Throwable.addSuppressed(Throwable.java:1043) ~[na:1.8.0_161]
at
org.apache.drill.common.DeferredException.addException(DeferredException.java:88)
~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:97)
~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:412)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.access$700(FragmentExecutor.java:55)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$ExecutorStateImpl.fail(FragmentExecutor.java:426)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.ops.FragmentContextImpl.fail(FragmentContextImpl.java:233)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.ops.FragmentContextImpl$1.accept(FragmentContextImpl.java:100)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.ops.FragmentContextImpl$1.accept(FragmentContextImpl.java:97)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at org.apache.drill.exec.ops.StatusHandler.failed(StatusHandler.java:42)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.RequestIdMap$RpcListener.setException(RequestIdMap.java:139)
~[drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:76)
[drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:66)
[drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at com.carrotsearch.hppc.IntObjectHashMap.forEach(IntObjectHashMap.java:692)
[hppc-0.7.1.jar:na]
at org.apache.drill.exec.rpc.RequestIdMap.channelClosed(RequestIdMap.java:62)
[drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.AbstractRemoteConnection.channelClosed(AbstractRemoteConnection.java:192)
[drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.AbstractServerConnection.channelClosed(AbstractServerConnection.java:165)
[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:167)
[drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:144)
[drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at
io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at
[jira] [Created] (DRILL-6254) IllegalArgumentException: the requested size must be non-negative
Khurram Faraaz created DRILL-6254:
-
Summary: IllegalArgumentException: the requested size must be
non-negative
Key: DRILL-6254
URL: https://issues.apache.org/jira/browse/DRILL-6254
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.13.0
Reporter: Khurram Faraaz
Assignee: Padma Penumarthy
Attachments: genAllTypesJSN.py
Flatten query fails due to IllegalArgumentException: the requested size must be
non-negative.
Script to generate JSON data file is attached here.
{noformat}
0: jdbc:drill:schema=dfs.tmp> CREATE TABLE tbl_all_types_jsn_to_parquet AS
. . . . . . . . . . . . . . > SELECT
. . . . . . . . . . . . . . > CAST( col_int AS INT) col_int,
. . . . . . . . . . . . . . > CAST( col_bigint AS BIGINT) col_bigint,
. . . . . . . . . . . . . . > CAST( col_char AS CHAR(10)) col_char,
. . . . . . . . . . . . . . > CAST( col_fxdln_str AS VARCHAR(256))
col_fxdln_str,
. . . . . . . . . . . . . . > CAST( col_varln_str AS VARCHAR(256))
col_varln_str,
. . . . . . . . . . . . . . > CAST( col_float AS FLOAT) col_float,
. . . . . . . . . . . . . . > CAST( col_double AS DOUBLE PRECISION) col_double,
. . . . . . . . . . . . . . > CAST( col_date AS DATE) col_date,
. . . . . . . . . . . . . . > CAST( col_time AS TIME) col_time,
. . . . . . . . . . . . . . > CAST( col_tmstmp AS TIMESTAMP) col_tmstmp,
. . . . . . . . . . . . . . > CAST( col_boolean AS BOOLEAN) col_boolean,
. . . . . . . . . . . . . . > col_binary,
. . . . . . . . . . . . . . > array_of_ints from `all_supported_types.json`;
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 9 |
+---++
1 row selected (0.29 seconds)
{noformat}
Reset all options and set slice_target=1
alter system reset all;
alter system set `planner.slice_target`=1;
output_batch_size was set to its default value
drill.exec.memory.operator.output_batch_size = 16777216
{noformat}
select *, flatten(array_of_ints) from tbl_all_types_jsn_to_parquet;
Error: SYSTEM ERROR: IllegalArgumentException: the requested size must be
non-negative
Fragment 0:0
[Error Id: 480bae96-ae89-45a7-b937-011c0f87c14d on qa102-45.qa.lab:31010]
(state=,code=0)
0: jdbc:drill:schema=dfs.tmp>
{noformat}
Stack trace from drillbit.log
{noformat}
2018-03-15 12:19:43,916 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query id
255538af-bcd5-98ee-32e0-68d98fc4a6fa: select *, flatten(array_of_ints) from
tbl_all_types_jsn_to_parquet
2018-03-15 12:19:43,952 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2018-03-15 12:19:43,953 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2018-03-15 12:19:43,966 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
o.a.d.exec.store.parquet.Metadata - Took 0 ms to get file statuses
2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of 1
using 1 threads. Time: 2ms total, 2.927366ms avg, 2ms max.
2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of 1
using 1 threads. Earliest start: 2.829000 μs, Latest start: 2.829000 μs,
Average start: 2.829000 μs .
2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
o.a.d.exec.store.parquet.Metadata - Took 3 ms to read file metadata
2018-03-15 12:19:44,000 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0:
State change requested AWAITING_ALLOCATION --> RUNNING
2018-03-15 12:19:44,000 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0:
State to report: RUNNING
2018-03-15 12:19:44,905 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0:
State change requested RUNNING --> FAILED
2018-03-15 12:19:44,927 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0:
State change requested FAILED --> FINISHED
2018-03-15 12:19:44,928 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalArgumentException:
the requested size must be non-negative
Fragment 0:0
[Error Id: 480bae96-ae89-45a7-b937-011c0f87c14d on qa102-45.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
[jira] [Created] (DRILL-6247) Minor fragments remain in CANCELLATION_REQUESTED state after query failure
Khurram Faraaz created DRILL-6247:
-
Summary: Minor fragments remain in CANCELLATION_REQUESTED state
after query failure
Key: DRILL-6247
URL: https://issues.apache.org/jira/browse/DRILL-6247
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.13.0
Reporter: Khurram Faraaz
Attachments: cancellation_requested_march_14.png,
drillbit_snippet.log, jstack_cancellation_requested.txt
Once a query fails, in this case due to an OOM in RPC we see many minor
fragments are reported to be in CANCELLATION_REQUESTED state on Web UI after
query has failed. The problem is reproducible. drillbit.log for this failure
and jstack output are attached here.
To reproduce the problem on a 4 node cluster.
alter system reset all;
alter system set `planner.slice_target`=1;
Failing query => SELECT * , FLATTEN(arr) FROM many_json_files;
Drill 1.13.0-SNAPSHOT, commit id: 766315ea17377199897d685ab801edd38394fe01
Stack trace from output of jstack, fragment 0:0 is reported to be in
CANCELLATION_REQUESTED state on Drill Web UI
jstack -l 13488 > jstack_DRILL_6235.txt
{noformat}
"25569e98-10f9-2fe2-9dec-0a42f3ad45fa:frag:0:0" #87 daemon prio=10 os_prio=0
tid=0x7f9d01374360 nid=0x2ff5 waiting on condition [0x7f9cd5536000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0007a388b300> (a
org.apache.drill.exec.rpc.ResettableBarrier$InternalSynchronizer)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at org.apache.drill.exec.rpc.ResettableBarrier.await(ResettableBarrier.java:70)
at
org.apache.drill.exec.rpc.AbstractRemoteConnection$WriteManager.waitForWritable(AbstractRemoteConnection.java:114)
at
org.apache.drill.exec.rpc.AbstractRemoteConnection.blockOnNotWritable(AbstractRemoteConnection.java:76)
at org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:108)
at
org.apache.drill.exec.rpc.user.UserServer$BitToUserConnection.sendData(UserServer.java:275)
at
org.apache.drill.exec.ops.AccountingUserConnection.sendData(AccountingUserConnection.java:42)
at
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:120)
at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95)
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:233)
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:226)
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
Stack trace from drillbit.log
{noformat}
2018-03-14 10:52:44,545 [25569e98-10f9-2fe2-9dec-0a42f3ad45fa:frag:1:49] INFO
o.a.d.e.w.fragment.FragmentExecutor - User Error Occurred: One or more nodes
ran out of memory while executing the query. (Failure allocating buffer.)
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more
nodes ran out of memory while executing the query.
Failure allocating buffer.
[Error Id: b83884df-af31-411a-9b28-554c294a7357 ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:243)
[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Failure
allocating buffer.
at
io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:67)
~[drill-memory-base-1.13.0-SNAPSHOT.jar:4.0.48.Final]
at
[jira] [Created] (DRILL-6235) Flatten query leads to out of memory in RPC layer.
Khurram Faraaz created DRILL-6235:
-
Summary: Flatten query leads to out of memory in RPC layer.
Key: DRILL-6235
URL: https://issues.apache.org/jira/browse/DRILL-6235
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Khurram Faraaz
Assignee: Padma Penumarthy
Attachments: 25593391-512d-23ab-7c84-3651006931e2.sys.drill
Flatten query leads to out of memory in RPC layer. Query profile is attached
here.
Total number of JSON files = 4095
Each JSON file has nine rows
And each row in the JSON has an array with 1024 integer values, and there are
other string values outside of the array.
Two major fragments and eighty eight minor fragments were created
On a 4 node CentOS cluster
number of CPU cores
[root@qa102-45 ~]# grep -c ^processor /proc/cpuinfo
32
Details of memory
{noformat}
0: jdbc:drill:schema=dfs.tmp> select * from sys.memory;
+--++---+-+-+-+-+
| hostname | user_port | heap_current | heap_max | direct_current |
jvm_direct_current | direct_max |
+--++---+-+-+-+-+
| qa102-45.qa.lab | 31010 | 1130364912 | 4294967296 | 0 | 170528 | 8589934592 |
| qa102-47.qa.lab | 31010 | 171823104 | 4294967296 | 0 | 21912 | 8589934592 |
| qa102-48.qa.lab | 31010 | 201326576 | 4294967296 | 0 | 21912 | 8589934592 |
| qa102-46.qa.lab | 31010 | 214780896 | 4294967296 | 0 | 21912 | 8589934592 |
+--++---+-+-+-+-+
4 rows selected (0.166 seconds)
{noformat}
Reset all options and set slice_target=1
alter system reset all;
alter system set `planner.slice_target`=1;
{noformat}
SELECT * , FLATTEN(arr) FROM many_json_files
...
Error: RESOURCE ERROR: One or more nodes ran out of memory while executing the
query.
Failure allocating buffer.
Fragment 1:38
[Error Id: cf4fd273-d8a2-45e8-8d72-15c738e53b0f on qa102-45.qa.lab:31010]
(state=,code=0)
{noformat}
Stack trace from drillbit.log fir above failing query.
{noformat}
2018-03-12 11:52:33,849 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 25593391-512d-23ab-7c84-3651006931e2:0:0:
State change requested AWAITING_ALLOCATION --> RUNNING
2018-03-12 11:52:33,849 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 25593391-512d-23ab-7c84-3651006931e2:0:0:
State to report: RUNNING
2018-03-12 11:52:33,854 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO
o.a.d.e.c.ClassCompilerSelector - Java compiler policy: DEFAULT, Debug option:
true
2018-03-12 11:52:35,929 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder
- Failure allocating buffer on incoming stream due to memory limits. Current
Allocation: 92340224.
2018-03-12 11:52:35,929 [BitServer-3] WARN o.a.d.exec.rpc.ProtobufLengthDecoder
- Failure allocating buffer on incoming stream due to memory limits. Current
Allocation: 92340224.
2018-03-12 11:52:35,930 [BitServer-3] ERROR o.a.drill.exec.rpc.data.DataServer
- Out of memory in RPC layer.
2018-03-12 11:52:35,930 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer
- Out of memory in RPC layer.
2018-03-12 11:52:35,930 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder
- Failure allocating buffer on incoming stream due to memory limits. Current
Allocation: 83886080.
2018-03-12 11:52:35,930 [BitServer-3] WARN o.a.d.exec.rpc.ProtobufLengthDecoder
- Failure allocating buffer on incoming stream due to memory limits. Current
Allocation: 83886080.
2018-03-12 11:52:35,930 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer
- Out of memory in RPC layer.
2018-03-12 11:52:35,930 [BitServer-3] ERROR o.a.drill.exec.rpc.data.DataServer
- Out of memory in RPC layer.
2018-03-12 11:52:35,931 [BitServer-3] WARN o.a.d.exec.rpc.ProtobufLengthDecoder
- Failure allocating buffer on incoming stream due to memory limits. Current
Allocation: 83886080.
2018-03-12 11:52:35,931 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder
- Failure allocating buffer on incoming stream due to memory limits. Current
Allocation: 83886080.
2018-03-12 11:52:35,931 [BitServer-3] ERROR o.a.drill.exec.rpc.data.DataServer
- Out of memory in RPC layer.
2018-03-12 11:52:35,931 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer
- Out of memory in RPC layer.
...
...
2018-03-12 11:52:35,939 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder
- Failure allocating buffer on incoming stream due to memory limits. Current
Allocation: 67174400.
2018-03-12 11:52:35,939 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer
- Out of memory in RPC layer.
2018-03-12 11:52:35,939 [BitServer-2] WARN
[jira] [Created] (DRILL-6229) memory leak when output_batch_size is set to 4096
Khurram Faraaz created DRILL-6229:
-
Summary: memory leak when output_batch_size is set to 4096
Key: DRILL-6229
URL: https://issues.apache.org/jira/browse/DRILL-6229
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Khurram Faraaz
When drill.exec.memory.operator.output_batch_size` = 4096 (4KB), we
continuously see in each test run, the memory leak when tests are run from test
framework.
Drill 1.13.0 git.commit.id=766315ea17377199897d685ab801edd38394fe01
alter system set `drill.exec.memory.operator.output_batch_size` = 4096;
To run flatten tests from drill test framework
bin/run_tests -s Functional/flatten_operators -g functional -d -t 500 -n 40
The below two tests time out and we see the memory leak
Timeouts:
/root/drillTEST_FRAMEWORK/drill-test-framework/framework/resources/Functional/flatten_operators/10rows/filter24.q
Timeouts:
/root/drillTEST_FRAMEWORK/drill-test-framework/framework/resources/Functional/flatten_operators/10rows/filter19.q
oadd.org.apache.drill.common.exceptions.UserException: CONNECTION ERROR:
Connection ... (user client) closed unexpectedly. Drillbit down?
Stack trace printed in drill test framework log
{noformat}
[Error Id: 54a16569-6555-4e92-aa5f-f76a8c11b31f ]
at
oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
at
oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$ChannelClosedHandler$1.operationComplete(QueryResultHandler.java:376)
at
oadd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at
oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
at
oadd.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
at
oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
at
oadd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
at
oadd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
at
oadd.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1003)
at
oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:632)
at
oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:610)
at
oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:553)
at
oadd.io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1236)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:602)
at
oadd.io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:602)
at
oadd.io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:35)
at
oadd.io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:607)
at
oadd.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at
oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at java.lang.Thread.run(Thread.java:748)
Channel is closed, discarding remaining 908 byte(s) in buffer.
Exception while closing client.
java.lang.RuntimeException: Exception while closing
at
oadd.org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46)
at oadd.org.apache.drill.exec.client.DrillClient.close(DrillClient.java:459)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.closeOrWarn(DrillConnectionImpl.java:810)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.cleanup(DrillConnectionImpl.java:823)
at
org.apache.drill.jdbc.impl.DrillHandler.onConnectionClose(DrillHandler.java:36)
at
oadd.org.apache.calcite.avatica.AvaticaConnection.close(AvaticaConnection.java:205)
at org.apache.drill.test.framework.ConnectionPool.close(ConnectionPool.java:102)
at org.apache.drill.test.framework.TestDriver.runTests(TestDriver.java:562)
at org.apache.drill.test.framework.TestDriver.main(TestDriver.java:100)
Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory
leaked: (4198400)
Allocator(ROOT) 0/4096/6713344/28631367680 (res/actual/peak/limit)
at
[jira] [Created] (DRILL-6215) Use prepared statement instead of Statement in JdbcRecordReader class
Khurram Faraaz created DRILL-6215:
-
Summary: Use prepared statement instead of Statement in
JdbcRecordReader class
Key: DRILL-6215
URL: https://issues.apache.org/jira/browse/DRILL-6215
Project: Apache Drill
Issue Type: Bug
Components: Storage - JDBC
Affects Versions: 1.12.0
Reporter: Khurram Faraaz
Use prepared statement instead of Statement in JdbcRecordReader class, which is
more efficient and less vulnerable to SQL injection attacks.
Apache Drill 1.13.0-SNAPSHOT, commit : 9073aed67d89e8b2188870d6c812706085c9c41b
Findbugs reports the below bug and suggests that we use prepared statement
instead of Statement.
{noformat}
In class org.apache.drill.exec.store.jdbc.JdbcRecordReader
In method
org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(OperatorContext,
OutputMutator)
At JdbcRecordReader.java:[line 170]
org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(OperatorContext,
OutputMutator) passes a nonconstant String to an execute method on an SQL
statement
The method invokes the execute method on an SQL statement with a String that
seems to be dynamically generated.
Consider using a prepared statement instead.
It is more efficient and less vulnerable to SQL injection attacks.
{noformat}
LOC -
https://github.com/apache/drill/blob/a9ea4ec1c5645ddab4b7aef9ac060ff5f109b696/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcRecordReader.java#L170
{noformat}
To run with findbugs:
mvn clean install -Pfindbugs -DskipTests
Findbugs will wirite the output to finbugsXml.html in the target directory of
each module.
For example the java-exec module report is located at:
./exec/java-exec/target/findbugs/findbugsXml.html
Use
find . -name "findbugsXml.html"
to locate the files.
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6214) DrillRuntimeException: Unexpected byte 0xfe at position 0 encountered while decoding UTF8 string.
Khurram Faraaz created DRILL-6214:
-
Summary: DrillRuntimeException: Unexpected byte 0xfe at position 0
encountered while decoding UTF8 string.
Key: DRILL-6214
URL: https://issues.apache.org/jira/browse/DRILL-6214
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Affects Versions: 1.12.0
Reporter: Khurram Faraaz
binary_string function returns DrillRuntimeException.
Apache Drill 1.13.0-SNAPSHOT, commit id:
9073aed67d89e8b2188870d6c812706085c9c41b
{noformat}
0: jdbc:drill:schema=dfs.tmp> select
binary_string(convert_to('abcdefghi','UTF16')) binStr from (values(1));
Error: SYSTEM ERROR: DrillRuntimeException: Unexpected byte 0xfe at position 0
encountered while decoding UTF8 string.
Fragment 1:0
[Error Id: 6233b0fe-19f2-40ff-9c9e-c4e317ff19d0 on qa102-48.qa.lab:31010]
(state=,code=0)
Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Unexpected
byte 0xfe at position 0 encountered while decoding UTF8 string.
at
org.apache.drill.exec.expr.fn.impl.StringFunctionUtil.utf8CharLen(StringFunctionUtil.java:88)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.expr.fn.impl.StringFunctionUtil.getUTF8CharLength(StringFunctionUtil.java:32)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.test.generated.ProjectorGen346.doSetup(ProjectorTemplate.java:116)
~[na:na]
at
org.apache.drill.exec.test.generated.ProjectorGen346.setup(ProjectorTemplate.java:101)
~[na:na]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchemaFromInput(ProjectRecordBatch.java:490)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:498)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:82)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:233)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_131]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_131]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
~[hadoop-common-2.7.0-mapr-1707.jar:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:226)
[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
... 4 common frames omitted
{noformat}
{noformat}
0: jdbc:drill:schema=dfs.tmp> select
binary_string(convert_to('123456','UTF16')) binStr from (values(1));
Error: SYSTEM ERROR: DrillRuntimeException: Unexpected byte 0xfe at position 0
encountered while decoding UTF8 string.
Fragment 1:0
[Error Id: 8a1bb0f0-7a9a-46da-b5e9-46e184690e2b on qa102-46.qa.lab:31010]
(state=,code=0)
Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Unexpected
byte 0xfe at position 0 encountered while decoding UTF8 string.
at
org.apache.drill.exec.expr.fn.impl.StringFunctionUtil.utf8CharLen(StringFunctionUtil.java:88)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.expr.fn.impl.StringFunctionUtil.getUTF8CharLength(StringFunctionUtil.java:32)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.test.generated.ProjectorGen272.doSetup(ProjectorTemplate.java:116)
~[na:na]
at
org.apache.drill.exec.test.generated.ProjectorGen272.setup(ProjectorTemplate.java:101)
~[na:na]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchemaFromInput(ProjectRecordBatch.java:490)
~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:498)
[jira] [Created] (DRILL-6181) CTAS should support writing nested structures (nested lists) to parquet.
Khurram Faraaz created DRILL-6181:
-
Summary: CTAS should support writing nested structures (nested
lists) to parquet.
Key: DRILL-6181
URL: https://issues.apache.org/jira/browse/DRILL-6181
Project: Apache Drill
Issue Type: Bug
Components: Storage - Parquet
Affects Versions: 1.12.0
Reporter: Khurram Faraaz
Both Parquet and Hive support writing nested structures into parquet
https://issues.apache.org/jira/browse/HIVE-8909
https://issues.apache.org/jira/browse/PARQUET-113
A CTAS from Drill fails when there is a nested list of lists, in one of the
columns in the project.
JSON data used in the test, note that "arr" is a nested list of lists
{noformat}
[root@qa102-45 ~]# cat jsonToParquet_02.json
{"id":"123","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"3","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"13","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"12","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"2","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"1","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"230","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"1230","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"1123","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"2123","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
{"id":"1523","arr":[[1,2,3,4],[5,6,7,8,9,10],[11,12,13,14,15]]}
[root@qa102-45 ~]#
{noformat}
CTAS fails with UnsupportedOperationException on Drill 1.12.0-mapr commit id
bb07ebbb9ba8742f44689f8bd8efb5853c5edea0
{noformat}
0: jdbc:drill:schema=dfs.tmp> CREATE TABLE tbl_prq_from_json_02 as select id,
arr from `jsonToParquet_02.json`;
Error: SYSTEM ERROR: UnsupportedOperationException: Unsupported type LIST
Fragment 0:0
[Error Id: 7e5b3c2d-9cf1-4e87-96c8-e7e7e8055ddf on qa102-45.qa.lab:31010]
(state=,code=0)
{noformat}
Stack trace from drillbit.log
{noformat}
2018-02-22 09:56:54,368 [2570fb99-62da-a516-2c1f-0381e21723ae:frag:0:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
UnsupportedOperationException: Unsupported type LIST
Fragment 0:0
[Error Id: 7e5b3c2d-9cf1-4e87-96c8-e7e7e8055ddf on qa102-45.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
UnsupportedOperationException: Unsupported type LIST
Fragment 0:0
[Error Id: 7e5b3c2d-9cf1-4e87-96c8-e7e7e8055ddf on qa102-45.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
~[drill-common-1.12.0-mapr.jar:1.12.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301)
[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.12.0-mapr.jar:1.12.0-mapr]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.lang.UnsupportedOperationException: Unsupported type LIST
at
org.apache.drill.exec.store.parquet.ParquetRecordWriter.getType(ParquetRecordWriter.java:253)
~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
at
org.apache.drill.exec.store.parquet.ParquetRecordWriter.newSchema(ParquetRecordWriter.java:205)
~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
at
org.apache.drill.exec.store.parquet.ParquetRecordWriter.updateSchema(ParquetRecordWriter.java:190)
~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
at
org.apache.drill.exec.physical.impl.WriterRecordBatch.setupNewSchema(WriterRecordBatch.java:157)
~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
at
org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:103)
~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134)
[jira] [Created] (DRILL-6155) support for pagination of results on Drill's web UI
Khurram Faraaz created DRILL-6155: - Summary: support for pagination of results on Drill's web UI Key: DRILL-6155 URL: https://issues.apache.org/jira/browse/DRILL-6155 Project: Apache Drill Issue Type: Bug Components: Client - Java Affects Versions: 1.12.0 Reporter: Khurram Faraaz We need support for pagination of results on Drill's web UI. Today if a query returns say a million results, those results are not paginated on Drill's web UI. Such large number of results should be paginated as required on the web UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6146) UNION with empty input on any one side returns incorrect results
Khurram Faraaz created DRILL-6146:
-
Summary: UNION with empty input on any one side returns incorrect
results
Key: DRILL-6146
URL: https://issues.apache.org/jira/browse/DRILL-6146
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 1.12.0
Reporter: Khurram Faraaz
When any one side of the UNION has an empty file as input, Drill returns
incorrect results.
table t3 does not have any data inserted into its rows. Postgress returns 1 as
the result for both the queries, whereas Drill does not.
{noformat}
postgres=# create table t3(id int, name varchar(25));
CREATE TABLE
postgres=# select * from (values(1)) t union select id from t3;
1
postgres=# select id from t3 union select * from (values(1)) t;
1
{noformat}
Results from Drill 1.12.0-mapr, note we return result 1 as result after the
union.
We have a directory named empty_JSON_f , and it has a single empty JSON file
(that JSON file has no content in it, it is empty).
{noformat}
0: jdbc:drill:schema=dfs.tmp> select * from (values(1)) UNION select id from
empty_JSON_f;
+-+
| EXPR$0 |
+-+
| 1 |
+-+
1 row selected (2.272 seconds){noformat}
However, in this query we return null and loose the value 1 from the right hand
side, after the union, this doesn't seem correct
{noformat}
0: jdbc:drill:schema=dfs.tmp> select id from empty_JSON_f UNION select * from
(values(1));
+---+
| id |
+---+
| null |
+---+
1 row selected (0.33 seconds){noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6141) JOIN query that uses USING clause returns incorrect results
Khurram Faraaz created DRILL-6141:
-
Summary: JOIN query that uses USING clause returns incorrect
results
Key: DRILL-6141
URL: https://issues.apache.org/jira/browse/DRILL-6141
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 1.12.0
Reporter: Khurram Faraaz
Join query that uses USING clause returns incorrect results.
Postgres 9.2.23 returns only one occurrence of the "id" column
{noformat}
postgres=# create table t1(id int, name varchar(30));
CREATE TABLE
postgres=# create table t2(id int, name varchar(30));
CREATE TABLE
postgres=# select * from t1;
id | name
+---
10 | John
13 | Kevin
15 | Susan
(3 rows)
postgres=# select * from t2;
id | name
+---
19 | Kyle
13 | Kevin
1 | Bob
17 | Kumar
(4 rows)
postgres=# select * from t1 JOIN t2 USING(id);
id | name | name
+---+---
13 | Kevin | Kevin
(1 row)
{noformat}
results from Drill 1.12.0-mapr commit : 2de42491be795721bcb4059bd46e27fc33272309
{noformat}
0: jdbc:drill:schema=dfs.tmp> create table t1 as select cast(columns[0] as int)
c1, cast(columns[1] as varchar(30)) c2 from `t1.csv`;
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 3 |
+---++
1 row selected (0.213 seconds)
0: jdbc:drill:schema=dfs.tmp> create table t2 as select cast(columns[0] as int)
c1, cast(columns[1] as varchar(30)) c2 from `t2.csv`;
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 4 |
+---++
1 row selected (0.168 seconds)
0: jdbc:drill:schema=dfs.tmp> select * from t1;
+-++
| c1 | c2 |
+-++
| 10 | John |
| 13 | Kevin |
| 15 | Susan |
+-++
3 rows selected (0.15 seconds)
0: jdbc:drill:schema=dfs.tmp> select * from t2;
+-++
| c1 | c2 |
+-++
| 19 | Kyle |
| 13 | Kevin |
| 1 | Bob |
| 17 | Kumar |
+-++
4 rows selected (0.171 seconds)
## Note that Drill returns an extra column, unlike Postgres, for the same query
over same data
0: jdbc:drill:schema=dfs.tmp> select * from t1 JOIN t2 USING(c1);
+-++--++
| c1 | c2 | c10 | c20 |
+-++--++
| 13 | Kevin | 13 | Kevin |
+-++--++
1 row selected (0.256 seconds)
## explain plan for above query
0: jdbc:drill:schema=dfs.tmp> explain plan for select * from t1 JOIN t2
USING(c1);
+--+--+
| text | json |
+--+--+
| 00-00 Screen
00-01 ProjectAllowDup(*=[$0], *0=[$1])
00-02 Project(T49¦¦*=[$0], T48¦¦*=[$2])
00-03 Project(T49¦¦*=[$2], c10=[$3], T48¦¦*=[$0], c1=[$1])
00-04 HashJoin(condition=[=($3, $1)], joinType=[inner])
00-06 Project(T48¦¦*=[$0], c1=[$1])
00-08 Scan(table=[[dfs, tmp, t2]], groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=maprfs:///tmp/t2]],
selectionRoot=maprfs:/tmp/t2, numFiles=1, numRowGroups=1,
usedMetadataFile=false, columns=[`*`]]])
00-05 Project(T49¦¦*=[$0], c10=[$1])
00-07 Project(T49¦¦*=[$0], c1=[$1])
00-09 Scan(table=[[dfs, tmp, t1]], groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=maprfs:///tmp/t1]],
selectionRoot=maprfs:/tmp/t1, numFiles=1, numRowGroups=1,
usedMetadataFile=false, columns=[`*`]]])
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-5965) Array index access returns an empty array
Khurram Faraaz created DRILL-5965:
-
Summary: Array index access returns an empty array
Key: DRILL-5965
URL: https://issues.apache.org/jira/browse/DRILL-5965
Project: Apache Drill
Issue Type: Bug
Components: Storage - JSON
Affects Versions: 1.12.0
Reporter: Khurram Faraaz
Accessing an array with [] from JSON data, returns an empty
string, whereas it should return the actual data from the array at the index
and not an empty array.
Drill 1.11.0-mapr commit: 065d72ba48c7af6b389b763753ecb6bf7d229ce8
{noformat}
0: jdbc:drill:schema=dfs.tmp> select t.structured_rep[0] from
`cornell_nlvr_train.json` t limit 1;
+-+
| EXPR$0 |
+-+
| [] |
+-+
1 row selected (0.249 seconds)
{noformat}
Where as accessing the elements of the array returns correct results.
{noformat}
0: jdbc:drill:schema=dfs.tmp> select t.structured_rep[0][0] from
`cornell_nlvr_train.json` t limit 1;
+---+
|EXPR$0 |
+---+
| {"y_loc":21,"size":20,"type":"triangle","x_loc":27,"color":"Yellow"} |
+---+
1 row selected (0.325 seconds)
0: jdbc:drill:schema=dfs.tmp> select t.structured_rep[0][1] from
`cornell_nlvr_train.json` t limit 1;
+-+
| EXPR$0|
+-+
| {"y_loc":60,"size":10,"type":"circle","x_loc":59,"color":"Yellow"} |
+-+
1 row selected (0.247 seconds)
{noformat}
Data used in the test
{noformat}
{
"sentence": "There is a circle closely touching a corner of a box.",
"label": "true",
"identifier": "1304-0",
"directory": "74",
"evals": {
"r0": "true"
},
"structured_rep": [
[{
"y_loc": 21,
"size": 20,
"type": "triangle",
"x_loc": 27,
"color": "Yellow"
}, {
"y_loc": 60,
"size": 10,
"type": "circle",
"x_loc": 59,
"color": "Yellow"
}],
[{
"y_loc": 81,
"size": 10,
"type": "triangle",
"x_loc": 48,
"color": "Yellow"
}, {
"y_loc": 64,
"size": 20,
"type": "circle",
"x_loc": 77,
"color": "#0099ff"
}],
[{
"y_loc": 2,
"size": 20,
"type": "triangle",
"x_loc": 62,
"color": "Yellow"
}, {
"y_loc": 70,
"size": 30,
"type": "circle",
"x_loc": 70,
"color": "Black"
}, {
"y_loc": 51,
"size": 20,
"type": "circle",
"x_loc": 30,
"color": "#0099ff"
}, {
"y_loc": 42,
"size": 20,
"type": "circle",
"x_loc": 67,
"color": "Yellow"
}, {
"y_loc": 73,
"size": 20,
"type": "circle",
"x_loc": 37,
"color": "Black"
}, {
"y_loc": 14,
"size": 30,
"type": "triangle",
"x_loc": 7,
"color": "Yellow"
}, {
"y_loc": 27,
"size": 10,
"type": "circle",
"x_loc": 48,
"color": "Black"
}]
]
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.
Khurram Faraaz created DRILL-5963:
-
Summary: Canceling a query hung in planning state, leaves the
query in ENQUEUED state for ever.
Key: DRILL-5963
URL: https://issues.apache.org/jira/browse/DRILL-5963
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.12.0
Environment: Drill 1.12.0-SNAPSHOT, commit:
4a718a0bd728ae02b502ac93620d132f0f6e1b6c
Reporter: Khurram Faraaz
Priority: Critical
Canceling the below query that is hung in planning state, leaves the query in
ENQUEUED state for ever.
Here is the query that is hung in planning state
{noformat}
0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 ||
',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS
CSV_DATA from (values(1));
+--+
| |
+--+
+--+
No rows selected (304.291 seconds)
{noformat}
Explain plan for that query also just hangs.
{noformat}
explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 ||
',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA
from (values(1));
...
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (DRILL-5947) COUNT() on empty JSON results in schema change error
Khurram Faraaz created DRILL-5947:
-
Summary: COUNT() on empty JSON results in schema
change error
Key: DRILL-5947
URL: https://issues.apache.org/jira/browse/DRILL-5947
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Khurram Faraaz
Count over a field, from a non-empty and another empty JSON file results in
schema change error, "Streaming aggregate does not support schema changes"
{noformat}
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was
removed in 8.0
apache drill 1.11.0-mapr
"drill baby drill"
0: jdbc:drill:schema=dfs.tmp> select count(id) from `diff_types`;
Error: UNSUPPORTED_OPERATION ERROR: Streaming aggregate does not support schema
changes
Prior schema :
BatchSchema [fields=[id(BIGINT:OPTIONAL)], selectionVector=NONE]
New schema :
BatchSchema [fields=[id(INT:OPTIONAL)], selectionVector=NONE]
Fragment 0:0
[Error Id: c1c77a0e-c7cf-494e-b8d2-ef2cc7e816e0 on centos-01.qa.lab:31010]
(state=,code=0)
0: jdbc:drill:schema=dfs.tmp> select id from `dif`Display all 872
possibilities? (y or n)
0: jdbc:drill:schema=dfs.tmp> select id from `diff_types/m1.json`;
+---+
| id |
+---+
| 1234 |
+---+
1 row selected (0.157 seconds)
0: jdbc:drill:schema=dfs.tmp> select id from `diff_types/m2.json`;
+---+
| id |
+---+
| null |
+---+
1 row selected (0.169 seconds)
0: jdbc:drill:schema=dfs.tmp> select * from `diff_types`;
+---+
| id |
+---+
| 1234 |
| null |
+---+
2 rows selected (0.209 seconds)
0: jdbc:drill:schema=dfs.tmp> select count(*) from `diff_types`;
+-+
| EXPR$0 |
+-+
| 2 |
+-+
1 row selected (0.208 seconds)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (DRILL-5945) Streaming aggregate does not support schema changes
Khurram Faraaz created DRILL-5945:
-
Summary: Streaming aggregate does not support schema changes
Key: DRILL-5945
URL: https://issues.apache.org/jira/browse/DRILL-5945
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Khurram Faraaz
There are two JSON files, one of the arrays "arr" in f2.json is empty, whereas
array "arr" in f1.json is not empty, a count on arr results in, "Streaming
aggregate does not support schema changes"
{noformat}
0: jdbc:drill:schema=dfs.tmp> select count(arr) from `f*.json`;
Error: UNSUPPORTED_OPERATION ERROR: Streaming aggregate does not support schema
changes
Prior schema :
BatchSchema [fields=[arr(VARCHAR:REPEATED) [$data$(VARCHAR:REQUIRED)]],
selectionVector=NONE]
New schema :
BatchSchema [fields=[arr(INT:REPEATED) [$data$(INT:REQUIRED)]],
selectionVector=NONE]
Fragment 0:0
[Error Id: 9081b02f-0de8-486a-b58b-f28ec3b84d98 on centos-01.qa.lab:31010]
(state=,code=0)
{noformat}
Stack trace from drillbit.log
{noformat}
2017-11-08 13:36:48,159 [25fc870f-e4d6-7107-c6cc-6e68ba92c4ba:frag:0:0] INFO
o.a.d.e.p.i.a.StreamingAggBatch - User Error Occurred: Streaming aggregate does
not support schema changes
Prior schema :
BatchSchema [fields=[arr(VARCHAR:REPEATED) [$data$(VARCHAR:REQUIRED)]],
selectionVector=NONE]
New schema :
BatchSchema [fields=[arr(INT:REPEATED) [$data$(INT:REQUIRED)]],
selectionVector=NONE]
org.apache.drill.common.exceptions.UserException: UNSUPPORTED_OPERATION ERROR:
Streaming aggregate does not support schema changes
Prior schema :
BatchSchema [fields=[arr(VARCHAR:REPEATED) [$data$(VARCHAR:REQUIRED)]],
selectionVector=NONE]
New schema :
BatchSchema [fields=[arr(INT:REPEATED) [$data$(INT:REQUIRED)]],
selectionVector=NONE]
[Error Id: 9081b02f-0de8-486a-b58b-f28ec3b84d98 ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
~[drill-common-1.11.0-mapr.jar:1.11.0-mapr]
at
org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext(StreamingAggBatch.java:207)
[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134)
[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105)
[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95)
[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:234)
[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:227)
[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at java.security.AccessController.doPrivileged(Native Method)
[na:1.8.0_144]
at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_144]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
[hadoop-common-2.7.0-mapr-1607.jar:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:227)
[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.11.0-mapr.jar:1.11.0-mapr]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_144]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
{noformat}
Data used in the query
{noformat}
[root@centos-01 json_data]# cat f1.json
{
"id":1,
"arr":["str1","str2","str3","str4"]
}
[root@centos-01 json_data]# cat f2.json
{
"id":2,
"arr":[]
}
>From sqlline
OpenJDK 64-Bit Server VM warning: ignoring option
[jira] [Created] (DRILL-5909) need new JMX metrics for (FAILED and CANCELED) queries
Khurram Faraaz created DRILL-5909:
-
Summary: need new JMX metrics for (FAILED and CANCELED) queries
Key: DRILL-5909
URL: https://issues.apache.org/jira/browse/DRILL-5909
Project: Apache Drill
Issue Type: Bug
Components: Execution - Monitoring
Affects Versions: 1.11.0, 1.12.0
Reporter: Khurram Faraaz
we have these JMX metrics today
{noformat}
drill.queries.running
drill.queries.completed
{noformat}
we need these new JMX metrics
{noformat}
drill.queries.failed
drill.queries.canceled
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (DRILL-5806) DrillRuntimeException: Interrupted but context.shouldContinue() is true
Khurram Faraaz created DRILL-5806:
-
Summary: DrillRuntimeException: Interrupted but
context.shouldContinue() is true
Key: DRILL-5806
URL: https://issues.apache.org/jira/browse/DRILL-5806
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.12.0
Environment: Drill 1.12.0 commit :
aaff1b35b7339fb4e6ab480dd517994ff9f0a5c5
Reporter: Khurram Faraaz
On a three node cluster
1. run concurrent queries (TPC-DS query 11) from a Java program.
2. stop the drillbit (foreman drillbit) this way,
/opt/mapr/drill/drill-1.12.0/bin/drillbit.sh stop
3. InterruptedException: null, is written to the drillbit.log
Stack trace from drillbit.log
{noformat}
2017-09-19 21:49:20,867 [263e6f48-0ace-0c0d-4f90-55ae2f0d778b:frag:5:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: InterruptedException
Fragment 5:0
[Error Id: 63ce8c18-040a-47f9-9643-e826de9a1a27 on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
InterruptedException
Fragment 5:0
[Error Id: 63ce8c18-040a-47f9-9643-e826de9a1a27 on centos-01.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:298)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_91]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: org.apache.drill.common.exceptions.DrillRuntimeException:
Interrupted but context.shouldContinue() is true
at
org.apache.drill.exec.work.batch.BaseRawBatchBuffer.getNext(BaseRawBatchBuffer.java:178)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.unorderedreceiver.UnorderedReceiverBatch.getNextBatch(UnorderedReceiverBatch.java:141)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.unorderedreceiver.UnorderedReceiverBatch.next(UnorderedReceiverBatch.java:164)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:141)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.test.generated.HashAggregatorGen498.doWork(HashAggTemplate.java:581)
~[na:na]
at
org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext(HashAggBatch.java:168)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[jira] [Resolved] (DRILL-3241) Query with window function runs out of direct memory and does not report back to client that it did
[
https://issues.apache.org/jira/browse/DRILL-3241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz resolved DRILL-3241.
---
Resolution: Fixed
> Query with window function runs out of direct memory and does not report back
> to client that it did
> ---
>
> Key: DRILL-3241
> URL: https://issues.apache.org/jira/browse/DRILL-3241
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Deneche A. Hakim
>Priority: Critical
> Fix For: 1.12.0
>
>
> Even though query run out of memory and was cancelled on the server, client
> (sqlline) was never notified of the event and it appears to the user that
> query is hung.
> Configuration:
> Single drillbit configured with:
> DRILL_MAX_DIRECT_MEMORY="2G"
> DRILL_HEAP="1G"
> TPCDS100 parquet files
> Query:
> {code}
> select
> sum(ss_quantity) over(partition by ss_store_sk order by ss_sold_date_sk)
> from store_sales;
> {code}
> drillbit.log
> {code}
> 2015-06-01 21:42:29,514 [BitServer-5] ERROR
> o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.
> Connection: /10.10.88.133:31012 <--> /10.10.88.133:38887 (data server).
> Closing connection.
> io.netty.handler.codec.DecoderException: java.lang.OutOfMemoryError: Direct
> buffer memory
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:233)
> ~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
> [netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:618)
> [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:329)
> [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:250)
> [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> [netty-common-4.0.27.Final.jar:4.0.27.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_71]
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
> ~[na:1.7.0_71]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> ~[na:1.7.0_71]
> at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:437)
> ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
> at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:179)
> ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
> at io.netty.buffer.PoolArena.allocate(PoolArena.java:168)
> ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
> at io.netty.buffer.PoolArena.reallocate(PoolArena.java:280)
> ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
> at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:110)
> ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:251)
> ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:849)
> ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:841)
> ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:831)
> ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
> at
[jira] [Created] (DRILL-5793) NPE on close
Khurram Faraaz created DRILL-5793:
-
Summary: NPE on close
Key: DRILL-5793
URL: https://issues.apache.org/jira/browse/DRILL-5793
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.12.0
Environment: Drill 1.12.0 commit :
aaff1b35b7339fb4e6ab480dd517994ff9f0a5c5
Reporter: Khurram Faraaz
The code looks wrong:
{noformat}
@Override
public void close() throws Exception {
options.close();
}
If the shutdown occurs to early, options is not yet assigned and an NPE results.
{noformat}
{noformat}
2017-09-14 20:16:39,551 [main] DEBUG o.apache.drill.exec.server.Drillbit -
Shutdown begun.
2017-09-14 20:16:41,560 [pool-5-thread-1] INFO
o.a.drill.exec.rpc.user.UserServer - closed eventLoopGroup
io.netty.channel.nio.NioEventLoopGroup@71a84ff4 in 1006 ms
2017-09-14 20:16:41,560 [pool-5-thread-2] INFO
o.a.drill.exec.rpc.data.DataServer - closed eventLoopGroup
io.netty.channel.nio.NioEventLoopGroup@f711283 in 1005 ms
2017-09-14 20:16:41,561 [pool-5-thread-1] INFO
o.a.drill.exec.service.ServiceEngine - closed userServer in 1007 ms
2017-09-14 20:16:41,562 [pool-5-thread-2] DEBUG
o.a.drill.exec.memory.BaseAllocator - closed allocator[rpc:bit-data].
2017-09-14 20:16:41,562 [pool-5-thread-2] INFO
o.a.drill.exec.service.ServiceEngine - closed dataPool in 1008 ms
2017-09-14 20:16:41,563 [main] DEBUG o.a.drill.exec.memory.BaseAllocator -
closed allocator[rpc:user].
2017-09-14 20:16:41,563 [main] DEBUG o.a.drill.exec.memory.BaseAllocator -
closed allocator[rpc:bit-control].
2017-09-14 20:16:41,593 [main] DEBUG o.a.drill.exec.memory.BaseAllocator -
closed allocator[ROOT].
2017-09-14 20:16:41,593 [main] WARN o.apache.drill.exec.server.Drillbit -
Failure on close()
java.lang.NullPointerException: null
at
org.apache.drill.exec.server.options.SystemOptionManager.close(SystemOptionManager.java:369)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at
org.apache.drill.exec.server.DrillbitContext.close(DrillbitContext.java:241)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at org.apache.drill.exec.work.WorkManager.close(WorkManager.java:154)
~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:173)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:314)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:290)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:286)
[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (DRILL-4296) Query hangs in CANCELLATION_REQUESTED when cancelled after it starts returning results
[
https://issues.apache.org/jira/browse/DRILL-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz resolved DRILL-4296.
---
Resolution: Fixed
> Query hangs in CANCELLATION_REQUESTED when cancelled after it starts
> returning results
> --
>
> Key: DRILL-4296
> URL: https://issues.apache.org/jira/browse/DRILL-4296
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.5.0
> Environment: commit.id=c9dbfbd
> 2 nodes with 32 cores and 32GB of max direct memory for drill
>Reporter: Deneche A. Hakim
> Attachments: 295eefc3-8d15-d63b-a721-3fde365b639c.sys.drill,
> data.tar.gz, node1_jstack.txt, node2_jstack.txt
>
>
> After running the following queries (it's the same reproduction from
> DRILL-2274):
> {noformat}
> set planner.memory.max_query_memory_per_node=8589934592;
> select sub1.uid from `all2274.json` sub1 inner join `all2274.json` sub2 on
> sub1.uid = sub2.uid order by sub1.uid;
> {noformat}
> After the query starts returning results, I cancelled the query from sqlline.
> This caused the query to hang in a CANCELLATION_REQUESTED state.
> Looking at jstack (attached) the root fragment is blocked waiting for Ack
> from the client.
> The the foreman node (which also runs Zookeeper) runs out of disk space once
> the query finishes spilling, which seems to contribute to the issue. Once I
> changed the spill directory to nfs I no longer so the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (DRILL-5591) non-ASCII characters in text file result in MalformedInputException
Khurram Faraaz created DRILL-5591:
-
Summary: non-ASCII characters in text file result in
MalformedInputException
Key: DRILL-5591
URL: https://issues.apache.org/jira/browse/DRILL-5591
Project: Apache Drill
Issue Type: Bug
Components: Storage - Text & CSV
Affects Versions: 1.11.0
Reporter: Khurram Faraaz
I am on Drill 1.11.0 commit id: 874bf62
To repro the issue:
wget http://cfdisat.blob.core.windows.net/lco/l_RFC_2017_05_11_2.txt.gz
gunzip l_RFC_2017_05_11_2.txt.gz
hadoop fs -put l_RFC_2017_05_11_2.txt /tmp
There are some non-ASCII characters at the beginning and end of the file used
in the test.
{noformat}
[root@centos-01 drill_5590]# head l_RFC_2017_05_11_2.txt
����0���1��
��� ��RFC|SNCF|SUBCONTRATACION
CUBB910321AC1|0|0
CUBB9104187K9|0|0
CUBB910709KD0|0|0
CUBB910817CE8|0|0
CUBB9111286YA|0|0
CUBB920408J69|0|0
{noformat}
Failing query
{noformat}
0: jdbc:drill:schema=dfs.tmp> select count(1) from `l_RFC_2017_05_11_2.txt` t
where columns[0] like 'CUBA7706%';
Error: SYSTEM ERROR: MalformedInputException: Input length = 1
Fragment 0:0
[Error Id: cdfa704c-0bc8-4791-95ae-d05b4c63beab on centos-01.qa.lab:31010]
(state=,code=0)
{noformat}
Stack trace from drillbit.log
{noformat}
Caused by: java.lang.RuntimeException:
java.nio.charset.MalformedInputException: Input length = 1
at
org.apache.drill.exec.expr.fn.impl.CharSequenceWrapper.decodeUT8(CharSequenceWrapper.java:185)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.expr.fn.impl.CharSequenceWrapper.setBuffer(CharSequenceWrapper.java:119)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.test.generated.FiltererGen15.doEval(FilterTemplate2.java:50)
~[na:na]
at
org.apache.drill.exec.test.generated.FiltererGen15.filterBatchNoSV(FilterTemplate2.java:100)
~[na:na]
at
org.apache.drill.exec.test.generated.FiltererGen15.filterBatch(FilterTemplate2.java:73)
~[na:na]
at
org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.doWork(FilterRecordBatch.java:81)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:93)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:133)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
[jira] [Created] (DRILL-5581) Query with CASE statement returns wrong results
Khurram Faraaz created DRILL-5581:
-
Summary: Query with CASE statement returns wrong results
Key: DRILL-5581
URL: https://issues.apache.org/jira/browse/DRILL-5581
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Khurram Faraaz
A query that uses case statement, returns wrong results.
{noformat}
Apache Drill 1.11.0-SNAPSHOT, commit id: 874bf629
[test@centos-101 ~]# cat order_sample.csv
202634342,2101,20160301
apache drill 1.11.0-SNAPSHOT
"this isn't your grandfather's sql"
0: jdbc:drill:schema=dfs.tmp> ALTER SESSION SET `store.format`='csv';
+---++
| ok |summary |
+---++
| true | store.format updated. |
+---++
1 row selected (0.245 seconds)
0: jdbc:drill:schema=dfs.tmp> CREATE VIEW `vw_order_sample_csv` as
. . . . . . . . . . . . . . > SELECT
. . . . . . . . . . . . . . > `columns`[0] AS `ND`,
. . . . . . . . . . . . . . > CAST(`columns`[1] AS BIGINT) AS `col1`,
. . . . . . . . . . . . . . > CAST(`columns`[2] AS BIGINT) AS `col2`
. . . . . . . . . . . . . . > FROM `order_sample.csv`;
+---+--+
| ok | summary|
+---+--+
| true | View 'vw_order_sample_csv' created successfully in 'dfs.tmp' schema |
+---+--+
1 row selected (0.253 seconds)
0: jdbc:drill:schema=dfs.tmp> select
. . . . . . . . . . . . . . > case
. . . . . . . . . . . . . . > when col1 > col2 then col1
. . . . . . . . . . . . . . > else col2
. . . . . . . . . . . . . . > end as temp_col,
. . . . . . . . . . . . . . > case
. . . . . . . . . . . . . . > when col1 = 2101 and (20170302 - col2) >
1 then 'D'
. . . . . . . . . . . . . . > when col2 = 2101 then 'P'
. . . . . . . . . . . . . . > when col1 - col2 > 1 then '0'
. . . . . . . . . . . . . . > else 'A'
. . . . . . . . . . . . . . > end as status
. . . . . . . . . . . . . . > from `vw_order_sample_csv`;
+---+-+
| temp_col | status |
+---+-+
| 20160301 | A |
+---+-+
1 row selected (0.318 seconds)
0: jdbc:drill:schema=dfs.tmp> explain plan for
. . . . . . . . . . . . . . > select
. . . . . . . . . . . . . . > case
. . . . . . . . . . . . . . > when col1 > col2 then col1
. . . . . . . . . . . . . . > else col2
. . . . . . . . . . . . . . > end as temp_col,
. . . . . . . . . . . . . . > case
. . . . . . . . . . . . . . > when col1 = 2101 and (20170302 - col2) >
1 then 'D'
. . . . . . . . . . . . . . > when col2 = 2101 then 'P'
. . . . . . . . . . . . . . > when col1 - col2 > 1 then '0'
. . . . . . . . . . . . . . > else 'A'
. . . . . . . . . . . . . . > end as status
. . . . . . . . . . . . . . > from `vw_order_sample_csv`;
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 Project(temp_col=[CASE(>(CAST(ITEM($0, 1)):BIGINT, CAST(ITEM($0,
2)):BIGINT), CAST(ITEM($0, 1)):BIGINT, CAST(ITEM($0, 2)):BIGINT)],
status=[CASE(AND(=(CAST(ITEM($0, 1)):BIGINT, 2101), >(-(20170302,
CAST(ITEM($0, 2)):BIGINT), 1)), 'D', =(CAST(ITEM($0, 2)):BIGINT, 2101),
'P', >(-(CAST(ITEM($0, 1)):BIGINT, CAST(ITEM($0, 2)):BIGINT), 1), '0',
'A')])
00-02Scan(groupscan=[EasyGroupScan
[selectionRoot=maprfs:/tmp/order_sample.csv, numFiles=1, columns=[`columns`[1],
`columns`[2]], files=[maprfs:///tmp/order_sample.csv]]])
// Details of Java compiler from sys.options
0: jdbc:drill:schema=dfs.tmp> select name, status from sys.options where name
like '%java_compiler%';
++--+
| name | status |
++--+
| exec.java.compiler.exp_in_method_size | DEFAULT |
| exec.java_compiler | DEFAULT |
| exec.java_compiler_debug | DEFAULT |
| exec.java_compiler_janino_maxsize | DEFAULT |
++--+
4 rows selected (0.21 seconds)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (DRILL-5576) OutOfMemoryException when some CPU cores are taken offline while concurrent queries are under execution
Khurram Faraaz created DRILL-5576:
-
Summary: OutOfMemoryException when some CPU cores are taken
offline while concurrent queries are under execution
Key: DRILL-5576
URL: https://issues.apache.org/jira/browse/DRILL-5576
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.11.0
Environment: 3 nodes CentOS cluster
Reporter: Khurram Faraaz
When we reduce the number of available CPU cores while concurrent queries are
under execution we see an OOM.
Drill 1.11.0 commit ID: d11aba2
three node CentOS 6.8 cluster
On each of the nodes Drill's direct memory was set to
export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-"16G"}
There are 24 cores on the node where foreman Drillbit is under execution.
{noformat}
[root@centos-01 logs]# lscpu
Architecture: x86_64
CPU op-mode(s):32-bit, 64-bit
Byte Order:Little Endian
CPU(s):24
On-line CPU(s) list: 0,2,4,5,8,9,12,14,15,18,20,22
Off-line CPU(s) list: 1,3,6,7,10,11,13,16,17,19,21,23
Thread(s) per core:1
Core(s) per socket:4
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family:6
Model: 44
Model name:Intel(R) Xeon(R) CPU E5645 @ 2.40GHz
Stepping: 2
CPU MHz: 1600.000
BogoMIPS: 4799.86
Virtualization:VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 12288K
NUMA node0 CPU(s): 0,2,4,5,12,14,15
NUMA node1 CPU(s): 8,9,18,20,22
{noformat}
Java code snippet that creates threads and executes TPC-DS query 11 concurrently
{noformat}
ExecutorService executor = Executors.newFixedThreadPool(48);
try {
for (int i = 1; i <= 48; i++) {
executor.submit(new ConcurrentQuery(conn));
}
} catch (Exception e) {
System.out.println(e.getMessage());
e.printStackTrace();
}
{noformat}
While the TPC-DS Query 11 is under execution using above program, we take half
of the available CPU cores offline
{noformat}
[root@centos-01 ~]# sh turnCPUCoresOffline.sh
OFFLINE cores are :
1,3,6-7,10-11,13,16-17,19,21,23
ONLINE cores are :
0,2,4-5,8-9,12,14-15,18,20,22
{noformat}
The result is we see an OutOfMemoryException, drillbit.log files are attached.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (DRILL-5573) Query profiles missing from Web UI and physical *.sys.drill file missing for query that was in CANCELLATION_REQUESTED state
Khurram Faraaz created DRILL-5573:
-
Summary: Query profiles missing from Web UI and physical
*.sys.drill file missing for query that was in CANCELLATION_REQUESTED state
Key: DRILL-5573
URL: https://issues.apache.org/jira/browse/DRILL-5573
Project: Apache Drill
Issue Type: Bug
Components: Web Server
Affects Versions: 1.11.0
Environment: 3 node CentOS cluster
Reporter: Khurram Faraaz
A query that was in CANCELLATION_REQUESTED state after the Java concurrent
program was terminated by issuing Ctrl-C from the command line.
Next, stop and start Drillbits (Drill 1.11.0 commit id: d11aba2) on all nodes,
and we see the below error message on Drill's Web UI, and we do not find the
physical file .sys.drill under drill-1.11.0/logs/profiles folder
clush -a /opt/mapr/drill/drill-1.11.0/bin/drillbit.sh stop
clush -a /opt/mapr/drill/drill-1.11.0/bin/drillbit.sh start
http://:8047/profiles/26c9073d-a8ff-17b0-5a86-34042bdfb494
{
"errorMessage" : "VALIDATION ERROR: No profile with given query id
'26c9073d-a8ff-17b0-5a86-34042bdfb494' exists. Please verify the query
id.\n\n\n[Error Id: c8f1ed5c-d93f-42b7-840d-54a1ed8da132 ]"
}
Stack trace from drillbit.log from foreman node, for the above error id.
{noformat}
2017-06-07 10:18:37,612 [main] INFO o.a.drill.exec.server.rest.WebServer -
Setting up HTTP connector for web server
2017-06-07 10:18:38,601 [main] INFO o.apache.drill.exec.server.Drillbit -
Startup completed (4149 ms).
2017-06-07 10:20:03,718 [qtp1575043618-89] INFO
o.a.d.e.s.r.profile.ProfileResources - User Error Occurred: No profile with
given query id '26c9073d-a8ff-17b0-5a86-34042bdfb494' exists. Please verify the
query id.
org.apache.drill.common.exceptions.UserException: VALIDATION ERROR: No profile
with given query id '26c9073d-a8ff-17b0-5a86-34042bdfb494' exists. Please
verify the query id.
[Error Id: c8f1ed5c-d93f-42b7-840d-54a1ed8da132 ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.server.rest.profile.ProfileResources.getQueryProfile(ProfileResources.java:288)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.server.rest.profile.ProfileResources.getProfile(ProfileResources.java:310)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[na:1.8.0_91]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[na:1.8.0_91]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269)
[jersey-server-2.8.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
[jersey-common-2.8.jar:na]
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
[jersey-common-2.8.jar:na]
at
[jira] [Created] (DRILL-5569) NullPointerException
Khurram Faraaz created DRILL-5569:
-
Summary: NullPointerException
Key: DRILL-5569
URL: https://issues.apache.org/jira/browse/DRILL-5569
Project: Apache Drill
Issue Type: Bug
Components: Storage - Parquet
Affects Versions: 1.11.0
Reporter: Khurram Faraaz
The below Exception was seen when TPC-DS Query 4 was executed against Drill
1.11.0
Drill 1.11.0 git commit ID: d11aba2
[root@centos-01 mapr]# cat MapRBuildVersion
5.2.1.42646.GA
Stack trace from drillbit.log
{noformat}
2017-06-06 07:46:43,160 [Drillbit-ShutdownHook#0] WARN
o.apache.drill.exec.work.WorkManager - Closing WorkManager but there are 80
running fragments.
2017-06-06 07:46:43,207 [Drillbit-ShutdownHook#0] INFO
o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 959, cache miss
count: 12, hit rate: 99%
2017-06-06 07:46:43,504 [scan-3] ERROR o.a.d.e.u.f.BufferedDirectBufInputStream
- Error reading from stream 1_1_0.parquet. Error was : Error reading out of an
FSDataInputStream using the Hadoop 2 ByteBuffer based read method.
2017-06-06 07:46:43,510 [scan-8] ERROR o.a.d.e.u.f.BufferedDirectBufInputStream
- Error reading from stream 1_1_0.parquet. Error was : Error reading out of an
FSDataInputStream using the Hadoop 2 ByteBuffer based read method.
2017-06-06 07:46:43,514 [scan-8] INFO o.a.d.e.s.p.c.AsyncPageReader - User
Error Occurred: Exception occurred while reading from disk.
(java.io.IOException: Error reading out of an FSDataInputStream using the
Hadoop 2 ByteBuffer based read method.)
org.apache.drill.common.exceptions.UserException: DATA_READ ERROR: Exception
occurred while reading from disk.
File: /drill/testdata/tpcds_sf1/parquet/store_sales/1_1_0.parquet
Column: ss_ext_list_price
Row Group Start: 75660513
[Error Id: 3a758095-fcc4-4364-a50b-33a027c1beb6 ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.handleAndThrowException(AsyncPageReader.java:199)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.access$600(AsyncPageReader.java:81)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$AsyncPageReaderTask.call(AsyncPageReader.java:483)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$AsyncPageReaderTask.call(AsyncPageReader.java:392)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_65]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_65]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: java.io.IOException: java.io.IOException: Error reading out of an
FSDataInputStream using the Hadoop 2 ByteBuffer based read method.
at
org.apache.drill.exec.util.filereader.BufferedDirectBufInputStream.getNextBlock(BufferedDirectBufInputStream.java:185)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.util.filereader.BufferedDirectBufInputStream.readInternal(BufferedDirectBufInputStream.java:212)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.util.filereader.BufferedDirectBufInputStream.read(BufferedDirectBufInputStream.java:277)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.util.filereader.DirectBufInputStream.getNext(DirectBufInputStream.java:111)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader$AsyncPageReaderTask.call(AsyncPageReader.java:437)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
... 5 common frames omitted
Caused by: java.io.IOException: Error reading out of an FSDataInputStream using
the Hadoop 2 ByteBuffer based read method.
at
org.apache.parquet.hadoop.util.CompatibilityUtil.getBuf(CompatibilityUtil.java:99)
~[parquet-hadoop-1.8.1-drill-r0.jar:1.8.1-drill-r0]
at
org.apache.drill.exec.util.filereader.BufferedDirectBufInputStream.getNextBlock(BufferedDirectBufInputStream.java:182)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
... 9 common frames omitted
Caused by: java.lang.NullPointerException: null
at
com.mapr.fs.MapRFsInStream.readIntoDirectByteBuffer(MapRFsInStream.java:219)
~[maprfs-5.2.1-mapr.jar:5.2.1-mapr]
at com.mapr.fs.MapRFsInStream.read(MapRFsInStream.java:333)
[jira] [Created] (DRILL-5570) InterruptedException: null
Khurram Faraaz created DRILL-5570:
-
Summary: InterruptedException: null
Key: DRILL-5570
URL: https://issues.apache.org/jira/browse/DRILL-5570
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.11.0
Environment: 3 node CentOS cluster
Reporter: Khurram Faraaz
When TPC-DS query11 was executed concurrently and one of the non-foreman
Drillbits was stopped (./bin/drillbit.sh stop)
we see the below system error InterruptedException in the drillbit.log of the
non-foreman node
Drill 1.11.0 git commit ID: d11aba2
[root@centos-01 mapr]# cat MapRBuildVersion
5.2.1.42646.GA
{noformat}
2017-06-06 07:46:44,288 [26c9a242-dfa1-35be-b5f1-ff6b4fa66086:frag:11:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: InterruptedException
Fragment 11:0
[Error Id: 40723399-8983-4777-a2bb-dc9d55ae338e on centos-02.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
InterruptedException
Fragment 11:0
[Error Id: 40723399-8983-4777-a2bb-dc9d55ae338e on centos-02.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:295)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_65]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: org.apache.drill.common.exceptions.DrillRuntimeException:
Interrupted but context.shouldContinue() is true
at
org.apache.drill.exec.work.batch.BaseRawBatchBuffer.getNext(BaseRawBatchBuffer.java:178)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.unorderedreceiver.UnorderedReceiverBatch.getNextBatch(UnorderedReceiverBatch.java:141)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.unorderedreceiver.UnorderedReceiverBatch.next(UnorderedReceiverBatch.java:159)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext(PartitionSenderRootExec.java:144)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:234)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:227)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method)
~[na:1.8.0_65]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_65]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
~[hadoop-common-2.7.0-mapr-1607.jar:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:227)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
... 4 common frames omitted
Caused by: java.lang.InterruptedException: null
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
~[na:1.8.0_65]
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
~[na:1.8.0_65]
at
java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
~[na:1.8.0_65]
at
java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
~[na:1.8.0_65]
at
[jira] [Created] (DRILL-5566) AssertionError: Internal error: invariant violated: call to wrong operator
Khurram Faraaz created DRILL-5566:
-
Summary: AssertionError: Internal error: invariant violated: call
to wrong operator
Key: DRILL-5566
URL: https://issues.apache.org/jira/browse/DRILL-5566
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Khurram Faraaz
CHARACTER_LENGTH is a non-reserved keyword as per the SQL specification. It is
a monadic function that accepts exactly one operand or parameter.
{noformat}
::=
|
|
|
|
...
...
::=
|
::=
{ CHAR_LENGTH | CHARACTER_LENGTH }
[ USING ]
...
...
::=
CHARACTERS
| OCTETS
{noformat}
Drill reports an assertion error in drillbit.log when character_length function
is used in a SQL query.
{noformat}
0: jdbc:drill:schema=dfs.tmp> select character_length(cast('hello' as
varchar(10))) col1 from (values(1));
Error: SYSTEM ERROR: AssertionError: Internal error: invariant violated: call
to wrong operator
[Error Id: 49198839-5a1b-4786-9257-59739b27d2a8 on centos-01.qa.lab:31010]
(org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
during fragment initialization: Internal error: invariant violated: call to
wrong operator
org.apache.drill.exec.work.foreman.Foreman.run():297
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745
Caused By (java.lang.AssertionError) Internal error: invariant violated: call
to wrong operator
org.apache.calcite.util.Util.newInternal():777
org.apache.calcite.util.Util.permAssert():885
org.apache.calcite.sql2rel.ReflectiveConvertletTable$3.convertCall():219
org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall():59
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit():4148
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit():3581
org.apache.calcite.sql.SqlCall.accept():130
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression():4040
org.apache.calcite.sql2rel.StandardConvertletTable$8.convertCall():185
org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall():59
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit():4148
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit():3581
org.apache.calcite.sql.SqlCall.accept():130
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression():4040
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectList():3411
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl():612
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect():568
org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive():2773
org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery():522
org.apache.drill.exec.planner.sql.SqlConverter.toRel():269
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRel():623
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert():195
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan():164
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():131
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():79
org.apache.drill.exec.work.foreman.Foreman.runSQL():1050
org.apache.drill.exec.work.foreman.Foreman.run():280
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745 (state=,code=0)
{noformat}
Calcite supports character_length function
{noformat}
[root@centos-0170 csv]# ./sqlline
sqlline version 1.1.9
sqlline> !connect jdbc:calcite:model=target/test-classes/model.json admin admin
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
0: jdbc:calcite:model=target/test-classes/mod> select
character_length(cast('hello' as varchar(10))) col1 from (values(1));
++
|COL1|
++
| 5 |
++
1 row selected (1.379 seconds)
{noformat}
Postgres 9.3 also supports character_length function
{noformat}
postgres=# select character_length(cast('hello' as varchar(10))) col1 from
(values(1)) foo;
col1
--
5
(1 row)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (DRILL-5564) IllegalStateException: allocator[op:21:1:5:HashJoinPOP]: buffer space (16674816) + prealloc space (0) + child space (0) != allocated (16740352)
Khurram Faraaz created DRILL-5564:
-
Summary: IllegalStateException: allocator[op:21:1:5:HashJoinPOP]:
buffer space (16674816) + prealloc space (0) + child space (0) != allocated
(16740352)
Key: DRILL-5564
URL: https://issues.apache.org/jira/browse/DRILL-5564
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.11.0
Environment: 3 node CentOS cluster
Reporter: Khurram Faraaz
Run a concurrent Java program that executes TPCDS query11
while the above concurrent java program is under execution
stop foreman Drillbit (from another shell, using below command)
./bin/drillbit.sh stop
and you will see the IllegalStateException: allocator[op:21:1:5:HashJoinPOP]:
and another assertion error, in the drillbit.log
AssertionError: Failure while stopping processing for operator id 10. Currently
have states of processing:false, setup:false, waiting:true.
Drill 1.11.0 git commit ID: d11aba2 (with assertions enabled)
details from drillbit.log from the foreman Drillbit node.
{noformat}
2017-06-05 18:38:33,838 [26ca5afa-7f6d-991b-1fdf-6196faddc229:frag:23:1] INFO
o.a.d.e.w.fragment.FragmentExecutor -
26ca5afa-7f6d-991b-1fdf-6196faddc229:23:1: State change requested RUNNING -->
FAILED
2017-06-05 18:38:33,849 [26ca5afa-7f6d-991b-1fdf-6196faddc229:frag:23:1] INFO
o.a.d.e.w.fragment.FragmentExecutor -
26ca5afa-7f6d-991b-1fdf-6196faddc229:23:1: State change requested FAILED -->
FINISHED
2017-06-05 18:38:33,852 [26ca5afa-7f6d-991b-1fdf-6196faddc229:frag:23:1] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: AssertionError: Failure
while stopping processing for operator id 10. Currently have states of
processing:false, setup:false, waiting:true.
Fragment 23:1
[Error Id: a116b326-43ed-4569-a20e-a10ba03d215e on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: AssertionError:
Failure while stopping processing for operator id 10. Currently have states of
processing:false, setup:false, waiting:true.
Fragment 23:1
[Error Id: a116b326-43ed-4569-a20e-a10ba03d215e on centos-01.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:295)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_91]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: java.lang.RuntimeException: java.lang.AssertionError: Failure while
stopping processing for operator id 10. Currently have states of
processing:false, setup:false, waiting:true.
at
org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:101)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:409)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:250)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
... 4 common frames omitted
Caused by: java.lang.AssertionError: Failure while stopping processing for
operator id 10. Currently have states of processing:false, setup:false,
waiting:true.
at
org.apache.drill.exec.ops.OperatorStats.stopProcessing(OperatorStats.java:167)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:255)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
[jira] [Created] (DRILL-5563) Stop non foreman Drillbit results in IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.
Khurram Faraaz created DRILL-5563:
-
Summary: Stop non foreman Drillbit results in
IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.
Key: DRILL-5563
URL: https://issues.apache.org/jira/browse/DRILL-5563
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.11.0
Environment: 3 node CentOS cluster
Reporter: Khurram Faraaz
Stopping the non-foreman Drillbit normally (as shown below) results in
IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.
/opt/mapr/drill/drill-1.11.0/bin/drillbit.sh stop
Drill 1.11.0 commit ID: d11aba2
Details from drillbit.log
{noformat}
Mon Jun 5 09:29:09 UTC 2017 Terminating drillbit pid 28182
2017-06-05 09:29:09,651 [Drillbit-ShutdownHook#0] INFO
o.apache.drill.exec.server.Drillbit - Received shutdown request.
2017-06-05 09:29:11,691 [pool-6-thread-1] INFO
o.a.drill.exec.rpc.user.UserServer - closed eventLoopGroup
io.netty.channel.nio.NioEventLoopGroup@55511dc2 in 1004 ms
2017-06-05 09:29:11,691 [pool-6-thread-2] INFO
o.a.drill.exec.rpc.data.DataServer - closed eventLoopGroup
io.netty.channel.nio.NioEventLoopGroup@4078d750 in 1004 ms
2017-06-05 09:29:11,692 [pool-6-thread-1] INFO
o.a.drill.exec.service.ServiceEngine - closed userServer in 1005 ms
2017-06-05 09:29:11,692 [pool-6-thread-2] INFO
o.a.drill.exec.service.ServiceEngine - closed dataPool in 1005 ms
2017-06-05 09:29:11,701 [Drillbit-ShutdownHook#0] INFO
o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 21, cache miss
count: 7, hit rate: 67%
2017-06-05 09:29:11,709 [Drillbit-ShutdownHook#0] ERROR
o.a.d.exec.server.BootStrapContext - Error while closing
java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding child
allocators.
Allocator(ROOT) 0/800/201359872/17179869184 (res/actual/peak/limit)
child allocators: 4
Allocator(frag:3:2) 200/0/0/200 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(frag:4:2) 200/0/0/200 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(frag:1:2) 200/0/0/200 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(frag:2:2) 200/0/0/200 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
ledgers: 0
reservations: 0
at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:492)
~[drill-memory-base-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:247)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:159)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:253)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
2017-06-05 09:29:11,709 [Drillbit-ShutdownHook#0] INFO
o.apache.drill.exec.server.Drillbit - Shutdown completed (2057 ms).
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (DRILL-5561) IllegalArgumentException: Self-suppression not permitted, seen when foreman Drillbit is killed
Khurram Faraaz created DRILL-5561:
-
Summary: IllegalArgumentException: Self-suppression not permitted,
seen when foreman Drillbit is killed
Key: DRILL-5561
URL: https://issues.apache.org/jira/browse/DRILL-5561
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.11.0
Environment: 3 node Centos cluster
Reporter: Khurram Faraaz
Priority: Minor
Drill 1.11.0 running on three nodes
kill foreman Drillbit, [root@centos-01 ~]# kill -9 15852; while a long running
SQL query (TPCDS Query 11) is under execution
drillbit.log from centos-01.qa.lab (foreman Drillbit) has no Exceptions or
Errors logged in it.
snippet of drillbit.log from centos-02.qa.lab ( non foreman Drillbit )
{noformat}
2017-05-30 08:21:32,834 [BitClient-1] INFO o.a.d.e.w.fragment.FragmentExecutor
- 26d2d481-d949-1de8-9497-c1492f086501:12:0: State change requested RUNNING -->
FAILED
2017-05-30 08:21:32,834 [BitServer-9] WARN o.a.d.exec.rpc.RpcExceptionHandler
- Exception occurred with closed channel. Connection: /10.10.100.202:31012
<--> /10.10.100.201:50773 (data server)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_65]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
~[na:1.8.0_65]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
~[na:1.8.0_65]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_65]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
~[na:1.8.0_65]
at
io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311)
~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
at io.netty.buffer.WrappedByteBuf.setBytes(WrappedByteBuf.java:407)
~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.buffer.UnsafeDirectLittleEndian.setBytes(UnsafeDirectLittleEndian.java:30)
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at io.netty.buffer.DrillBuf.setBytes(DrillBuf.java:770)
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at
io.netty.buffer.MutableWrappedByteBuf.setBytes(MutableWrappedByteBuf.java:280)
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at
io.netty.buffer.ExpandableByteBuf.setBytes(ExpandableByteBuf.java:26)
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
2017-05-30 08:21:32,848 [BitClient-1] INFO o.a.d.e.w.fragment.FragmentExecutor
- 26d2d481-d949-1de8-9497-c1492f086501:16:0: State change requested RUNNING -->
FAILED
2017-05-30 08:21:32,850 [BitClient-1] INFO o.a.d.e.w.fragment.FragmentExecutor
- 26d2d481-d949-1de8-9497-c1492f086501:14:0: State change requested RUNNING -->
FAILED
2017-05-30 08:21:32,851 [BitClient-1] WARN
o.apache.drill.exec.rpc.RequestIdMap - Failure while attempting to fail rpc
response.
java.lang.IllegalArgumentException: Self-suppression not permitted
at java.lang.Throwable.addSuppressed(Throwable.java:1043) ~[na:1.8.0_65]
at
org.apache.drill.common.DeferredException.addException(DeferredException.java:88)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:97)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:409)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.access$700(FragmentExecutor.java:55)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
[jira] [Created] (DRILL-5543) Inner join query results in IndexOutOfBoundsException
Khurram Faraaz created DRILL-5543:
-
Summary: Inner join query results in IndexOutOfBoundsException
Key: DRILL-5543
URL: https://issues.apache.org/jira/browse/DRILL-5543
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Khurram Faraaz
Inner join query results in IndexOutOfBoundsException
Steps to reproduce the IOB.
Copy data from drill test framework../framework/resources/Datasources/joins
directory to /drill/testdata/joins directory
Run the below SQL.
Apache Drill 1.11.0 commit ID: d11aba2e
MapR version : 5.2.1.42646.GA
[test@centos-ab xyzb]# uname -r
2.6.32-642.el6.x86_64
Three node CentOS cluster
{noformat}
0: jdbc:drill:schema=dfs.tmp> use dfs.joins;
+---++
| ok |summary |
+---++
| true | Default schema changed to [dfs.joins] |
+---++
1 row selected (0.279 seconds)
0: jdbc:drill:schema=dfs.tmp> select
. . . . . . . . . . . . . . > count(*)
. . . . . . . . . . . . . . > from
. . . . . . . . . . . . . . > j1 INNER JOIN j4 ON
. . . . . . . . . . . . . . > (
. . . . . . . . . . . . . . > j1.c_date = j4.c_date
. . . . . . . . . . . . . . > and j1.c_time = j4.c_time
. . . . . . . . . . . . . . > and j1.c_timestamp = j4.c_timestamp
. . . . . . . . . . . . . . > and j1.c_boolean = j4.c_boolean
. . . . . . . . . . . . . . > and j1.c_varchar = j4.c_varchar
. . . . . . . . . . . . . . > and j1.c_float = j4.c_float
. . . . . . . . . . . . . . > and j1.c_double = j4.c_double
. . . . . . . . . . . . . . > and j1.d9 = j4.d9
. . . . . . . . . . . . . . > and j1.d18 = j4.d18
. . . . . . . . . . . . . . >
. . . . . . . . . . . . . . > )
. . . . . . . . . . . . . . > where j1.c_date IN (select c_date from j2);
Error: SYSTEM ERROR: IndexOutOfBoundsException: index: 9, length: 1 (expected:
range(0, 0))
Fragment 0:0
[Error Id: f8323164-581e-4377-87e0-f44e0bc1d037 on centos-01.qa.lab:31010]
(state=,code=0)
{noformat}
Stack trace from drillbit.log
{noformat}
2017-05-26 08:47:05,151 [26d8147b-394f-3fea-f1e1-ece75d9b3f41:frag:0:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IndexOutOfBoundsException:
index: 9, length: 1 (expected: range(0, 0))
Fragment 0:0
[Error Id: f8323164-581e-4377-87e0-f44e0bc1d037 on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IndexOutOfBoundsException: index: 9, length: 1 (expected: range(0, 0))
Fragment 0:0
[Error Id: f8323164-581e-4377-87e0-f44e0bc1d037 on centos-01.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:295)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_91]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: java.lang.IndexOutOfBoundsException: index: 9, length: 1 (expected:
range(0, 0))
at io.netty.buffer.DrillBuf.checkIndexD(DrillBuf.java:123)
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at io.netty.buffer.DrillBuf.chk(DrillBuf.java:147)
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at io.netty.buffer.DrillBuf.getByte(DrillBuf.java:775)
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at
org.apache.drill.exec.vector.UInt1Vector$Accessor.get(UInt1Vector.java:354)
~[vector-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.vector.NullableDateVector$Accessor.isSet(NullableDateVector.java:394)
~[vector-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.test.generated.StreamingAggregatorGen383.outputRecordKeysPrev(StreamingAggTemplate.java:186)
~[na:na]
at
org.apache.drill.exec.test.generated.StreamingAggregatorGen383.outputToBatchPrev(StreamingAggTemplate.java:322)
~[na:na]
at
[jira] [Created] (DRILL-5499) TPC-DS Query 4 concurrent execution RESOURCE ERROR: One or more nodes ran out of memory while executing the query.
Khurram Faraaz created DRILL-5499:
-
Summary: TPC-DS Query 4 concurrent execution RESOURCE ERROR: One
or more nodes ran out of memory while executing the query.
Key: DRILL-5499
URL: https://issues.apache.org/jira/browse/DRILL-5499
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.11.0
Environment: single node CentOS
Reporter: Khurram Faraaz
Running TPC-DS query 4 sequentially does not result in error/OOM. TPC-DS query
4 takes 7 GB memory on an average, on Drill 1.11.0.
However, there is an OOM and RESOURCE ERROR when we run TPC-DS query 4
concurrently using 8 threads on a single node that has 24 processing units
available.
{noformat}
2017-05-09 13:42:30,547 [BitServer-5] ERROR o.a.d.exec.rpc.RpcExceptionHandler
- Exception in RPC communication. Connection: /10.10.100.201:31012 <-->
/10.10.100.201:59746 (data server). Closing connection.
io.netty.handler.codec.DecoderException:
org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate buffer
of size 25165824 due to memory limit. Current allocation: 21037056
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:233)
~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to
allocate buffer of size 25165824 due to memory limit. Current allocation:
21037056
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:220)
~[drill-memory-base-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:195)
~[drill-memory-base-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
io.netty.buffer.ExpandableByteBuf.capacity(ExpandableByteBuf.java:43)
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at
io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:251)
~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:849)
~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:841)
~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:831)
~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:92)
~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:227)
~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
... 13 common frames omitted
...
2017-05-09 13:42:30,547 [26ee38c2-75c8-05cd-0cc0-9263fefba955:frag:34:0] INFO
o.a.d.e.w.fragment.FragmentExecutor -
26ee38c2-75c8-05cd-0cc0-9263fefba955:34:0: State change requested
AWAITING_ALLOCATION --> RUNNING
2017-05-09 13:42:30,547 [26ee38c2-75c8-05cd-0cc0-9263fefba955:frag:34:0] INFO
[jira] [Created] (DRILL-5458) REPEATED_COUNT over varchar data results in SchemaChangeException
Khurram Faraaz created DRILL-5458:
-
Summary: REPEATED_COUNT over varchar data results in
SchemaChangeException
Key: DRILL-5458
URL: https://issues.apache.org/jira/browse/DRILL-5458
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.11.0
Reporter: Khurram Faraaz
REPEATED_COUNT over varchar type data results in SchemaChangeException
Apache Drill 1.11.0 commit id: 3e8b01d5
{noformat}
// CTAS over JSON file.
0: jdbc:drill:schema=dfs.tmp> create table tblwarr01 as select
convert_to(t.arr,'JSON') arr, id from `rptd_count.json` t;
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 10 |
+---++
1 row selected (0.221 seconds)
0: jdbc:drill:schema=dfs.tmp> select * from tblwarr01;
+--+-+
| arr | id |
+--+-+
| [B@67cf1438 | 1 |
| [B@4c389dc6 | 2 |
| [B@18fe5942 | 3 |
| [B@629608df | 4 |
| [B@68209b09 | 5 |
| [B@34a2a147 | 6 |
| [B@210a5750 | 7 |
| [B@2dea5622 | 8 |
| [B@73bce9ba | 9 |
| [B@7794edb2 | 10 |
+--+-+
10 rows selected (0.209 seconds)
{noformat}
{noformat}
0: jdbc:drill:schema=dfs.tmp> select convert_from(t.arr,'UTF8') from tblwarr01
t;
+---+
|EXPR$0 |
+---+
| [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] |
| [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] |
| [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] |
| [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] |
| [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] |
| [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] |
| [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] |
| [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] |
| [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] |
| [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] |
+---+
10 rows selected (0.195 seconds)
// Performing a REPEATED_COUNT on the array results in SchemaChangeException
0: jdbc:drill:schema=dfs.tmp> select REPEATED_COUNT(convert_from(t.arr,'UTF8'))
from tblwarr01 t;
Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to materialize
incoming schema. Errors:
Error in expression at index -1. Error: Missing function implementation:
[repeated_count(VARCHAR-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--..
Fragment 0:0
[Error Id: 1a891918-79f6-49d1-a594-f9ec5f28a99a on centos-01.qa.lab:31010]
(state=,code=0)
{noformat}
Data used in CTAS.
{noformat}
[root@centos-01 ~]# cat rptd_count.json
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":1}
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":2}
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":3}
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":4}
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":5}
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":6}
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":7}
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":8}
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":9}
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":10}
[root@centos-01 ~]#
{noformat}
Stack trace from drillbit.log
{noformat}
2017-05-02 05:17:03,651 [26f7e9b0-5f11-e764-6fa8-92c27ca2c6cf:frag:0:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: SchemaChangeException:
Failure while trying to materialize incoming schema. Errors:
Error in expression at index -1. Error: Missing function implementation:
[repeated_count(VARCHAR-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--..
Fragment 0:0
[Error Id: 1a891918-79f6-49d1-a594-f9ec5f28a99a on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
SchemaChangeException: Failure while trying to materialize incoming schema.
Errors:
Error in expression at index -1. Error: Missing function implementation:
[repeated_count(VARCHAR-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--..
Fragment 0:0
[Error Id: 1a891918-79f6-49d1-a594-f9ec5f28a99a on centos-01.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:295)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_91]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_91]
[jira] [Created] (DRILL-5456) StringIndexOutOfBoundsException when converting a JSON array to UTF-8
Khurram Faraaz created DRILL-5456:
-
Summary: StringIndexOutOfBoundsException when converting a JSON
array to UTF-8
Key: DRILL-5456
URL: https://issues.apache.org/jira/browse/DRILL-5456
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Khurram Faraaz
Convert a JSON array to UTF-8 using CONVERT_TO function results in
StringIndexOutOfBoundsException
Apache Drill 1.11.0 commit ID: 3e8b01d
Data used in test
{noformat}
[root@centos-01 ~]# cat rptd_count.json
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":1}
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":2}
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":3}
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":4}
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":5}
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":6}
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":7}
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":8}
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":9}
{"arr":[0,1,2,3,4,5,6,7,8,9,10],"id":10}
[root@centos-01 ~]#
{noformat}
{noformat}
0: jdbc:drill:schema=dfs.tmp> select convert_to(t.arr,'UTF-8') c, id from
`rptd_count.json` t;
Error: SYSTEM ERROR: StringIndexOutOfBoundsException: String index out of
range: -3
[Error Id: 056a13c0-6c9f-403e-877e-040e907d6581 on centos-01.qa.lab:31010]
(state=,code=0)
0: jdbc:drill:schema=dfs.tmp>
{noformat}
Data from the JSON file
{noformat}
0: jdbc:drill:schema=dfs.tmp> select * from `rptd_count.json`;
+---+-+
|arr| id |
+---+-+
| [0,1,2,3,4,5,6,7,8,9,10] | 1 |
| [0,1,2,3,4,5,6,7,8,9,10] | 2 |
| [0,1,2,3,4,5,6,7,8,9,10] | 3 |
| [0,1,2,3,4,5,6,7,8,9,10] | 4 |
| [0,1,2,3,4,5,6,7,8,9,10] | 5 |
| [0,1,2,3,4,5,6,7,8,9,10] | 6 |
| [0,1,2,3,4,5,6,7,8,9,10] | 7 |
| [0,1,2,3,4,5,6,7,8,9,10] | 8 |
| [0,1,2,3,4,5,6,7,8,9,10] | 9 |
| [0,1,2,3,4,5,6,7,8,9,10] | 10 |
+---+-+
10 rows selected (0.224 seconds)
{noformat}
Stack trace from drillbit.log
{noformat}
2017-05-01 19:32:34,209 [26f872ad-62a3-d7a7-aec1-9c7d937a2416:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query id
26f872ad-62a3-d7a7-aec1-9c7d937a2416: select convert_to(t.arr,'UTF-8') c, id
from `rptd_count.json` t
...
2017-05-01 19:32:34,328 [26f872ad-62a3-d7a7-aec1-9c7d937a2416:foreman] ERROR
o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR:
StringIndexOutOfBoundsException: String index out of range: -3
[Error Id: 056a13c0-6c9f-403e-877e-040e907d6581 on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
StringIndexOutOfBoundsException: String index out of range: -3
[Error Id: 056a13c0-6c9f-403e-877e-040e907d6581 on centos-01.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:847)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:977)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:297)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_91]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected
exception during fragment initialization: String index out of range: -3
... 4 common frames omitted
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of
range: -3
at java.lang.String.substring(String.java:1931) ~[na:1.8.0_91]
at
org.apache.drill.exec.planner.logical.PreProcessLogicalRel.getConvertFunctionException(PreProcessLogicalRel.java:244)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.planner.logical.PreProcessLogicalRel.visit(PreProcessLogicalRel.java:148)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.calcite.rel.logical.LogicalProject.accept(LogicalProject.java:129)
~[calcite-core-1.4.0-drill-r21.jar:1.4.0-drill-r21]
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.preprocessNode(DefaultSqlHandler.java:641)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert(DefaultSqlHandler.java:196)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:164)
[jira] [Created] (DRILL-5434) IllegalStateException: Memory was leaked by query.
Khurram Faraaz created DRILL-5434:
-
Summary: IllegalStateException: Memory was leaked by query.
Key: DRILL-5434
URL: https://issues.apache.org/jira/browse/DRILL-5434
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Khurram Faraaz
Issue a long running COUNT query.
While the query is being executed, stop the foreman drillbit, ./drillbit.sh stop
A memory leak is reported in the drillbit.log
Apache Drill 1.11.0
git.commit.id.abbrev=06e1522
Stack trace from drillbit.log
{noformat}
2017-04-13 06:14:36,828 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query id
2710e8b2-d4dc-1bee-016e-b69fd4966916: SELECT COUNT(*) FROM `twoKeyJsn.json`
2017-04-13 06:14:36,929 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-04-13 06:14:36,929 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-04-13 06:14:36,929 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-04-13 06:14:36,929 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-04-13 06:14:36,930 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-04-13 06:14:36,930 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-04-13 06:14:36,930 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-04-13 06:14:36,932 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-04-13 06:14:36,934 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using
1 threads. Time: 2ms total, 2.102992ms avg, 2ms max.
2017-04-13 06:14:36,934 [2710e8b2-d4dc-1bee-016e-b69fd4966916:foreman] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using
1 threads. Earliest start: 0.555000 μs, Latest start: 0.555000 μs, Average
start: 0.555000 μs .
2017-04-13 06:14:36,949 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 2710e8b2-d4dc-1bee-016e-b69fd4966916:0:0:
State change requested AWAITING_ALLOCATION --> RUNNING
2017-04-13 06:14:36,949 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 2710e8b2-d4dc-1bee-016e-b69fd4966916:0:0:
State to report: RUNNING
Thu Apr 13 06:14:40 UTC 2017 Terminating drillbit pid 5107
2017-04-13 06:14:40,756 [Drillbit-ShutdownHook#0] INFO
o.apache.drill.exec.server.Drillbit - Received shutdown request.
2017-04-13 06:14:47,819 [pool-169-thread-2] INFO
o.a.drill.exec.rpc.data.DataServer - closed eventLoopGroup
io.netty.channel.nio.NioEventLoopGroup@4f3ee67c in 1024 ms
2017-04-13 06:14:47,819 [pool-169-thread-2] INFO
o.a.drill.exec.service.ServiceEngine - closed dataPool in 1024 ms
2017-04-13 06:14:49,806 [Drillbit-ShutdownHook#0] WARN
o.apache.drill.exec.work.WorkManager - Closing WorkManager but there are 1
running fragments.
2017-04-13 06:14:49,807 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 2710e8b2-d4dc-1bee-016e-b69fd4966916:0:0:
State change requested RUNNING --> FAILED
2017-04-13 06:14:49,807 [Drillbit-ShutdownHook#0] INFO
o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 6964, cache miss
count: 335, hit rate: 95%
2017-04-13 06:14:49,807 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 2710e8b2-d4dc-1bee-016e-b69fd4966916:0:0:
State change requested FAILED --> FINISHED
2017-04-13 06:14:49,809 [2710e8b2-d4dc-1bee-016e-b69fd4966916:frag:0:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException
Fragment 0:0
[Error Id: 94817261-98a9-4153-8b3a-2d9c95d80cc1 on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
NullPointerException
Fragment 0:0
[Error Id: 94817261-98a9-4153-8b3a-2d9c95d80cc1 on centos-01.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[jira] [Created] (DRILL-5430) select on fastparquet created parquet file results in IOB Exception
Khurram Faraaz created DRILL-5430:
-
Summary: select on fastparquet created parquet file results in IOB
Exception
Key: DRILL-5430
URL: https://issues.apache.org/jira/browse/DRILL-5430
Project: Apache Drill
Issue Type: Bug
Components: Storage - Parquet
Affects Versions: 1.11.0
Reporter: Khurram Faraaz
Select on a parquet file created using python's fastparquet results in IOB
Exception.
Parquet file was created using fastparquet (python implementation of parquet
format)
Parquet schema information for parquet file used in test.
{noformat}
[root@centos-12q parquet-tools]# ./parquet-schema frmPandas_9.parquet
message schema {
required int32 c_int8_signed (INT_8);
required int32 c_uint8 (UINT_8);
}
{noformat}
Apache Drill 1.11.0
git.commit.id.abbrev=06e1522
Select on the parquet results in DATA_READ error, stack trace shows an
IndexOutOfBoundsException
{noformat}
0: jdbc:drill:schema=dfs.tmp> select * from `frmPandas_9.parquet`;
Error: DATA_READ ERROR: Error reading from Parquet file
File: /tmp/frmPandas_9.parquet
Column: c_int8_signed
Row Group Start: 4
Fragment 0:0
[Error Id: 32b4d0ec-4198-4b38-bdd7-a27c881072e6 on centos-01.qa.lab:31010]
(state=,code=0)
{noformat}
Stack trace from drillbit.log
{noformat}
2017-04-12 04:23:55,214 [27125424-657c-7f8f-c27b-4d1516e4bb97:frag:0:0] INFO
o.a.d.e.s.p.c.ColumnReader - User Error Occurred: Error reading from Parquet
file (srcIndex: 0)
org.apache.drill.common.exceptions.UserException: DATA_READ ERROR: Error
reading from Parquet file
File: /tmp/frmPandas_9.parquet
Column: c_int8_signed
Row Group Start: 4
[Error Id: 32b4d0ec-4198-4b38-bdd7-a27c881072e6 ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.readValues(ColumnReader.java:151)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.processPageData(ColumnReader.java:199)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.determineSize(ColumnReader.java:179)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.processPages(ColumnReader.java:129)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFieldsSerial(ParquetRecordReader.java:512)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.readAllFixedFields(ParquetRecordReader.java:505)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:590)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:179)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:232)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method)
[na:1.8.0_91]
at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_91]
at
[jira] [Created] (DRILL-5410) Unicode characters in JSON data result in DATA_READ error.
Khurram Faraaz created DRILL-5410:
-
Summary: Unicode characters in JSON data result in DATA_READ error.
Key: DRILL-5410
URL: https://issues.apache.org/jira/browse/DRILL-5410
Project: Apache Drill
Issue Type: Bug
Components: Storage - JSON
Affects Versions: 1.11.0
Reporter: Khurram Faraaz
Unicode characters in JSON data result in DATA_READ error.
Apache Drill 1.11.0 commit ID : adbf363
{noformat}
Data used in test.
[root@centos-01 ~]# cat test.json
"arr":[{u'\ue4ea\ue4ba': 5.167460680922489e+18}]
apache drill 1.11.0-SNAPSHOT
"a drill in the hand is better than two in the bush"
0: jdbc:drill:schema=dfs.tmp> alter session set `store.json.all_text_mode`=true;
+---++
| ok | summary |
+---++
| true | store.json.all_text_mode updated. |
+---++
1 row selected (0.156 seconds)
0: jdbc:drill:schema=dfs.tmp> select * from `test.json`;
Error: DATA_READ ERROR: Failure while parsing JSON. Found token of
[VALUE_STRING]. Drill currently only supports parsing json strings that
contain either lists or maps. The root object cannot be a scalar.
Line 1
Column 3
Field
Fragment 0:0
[Error Id: 535e08fb-89b0-443b-86dc-032e40bbb4e1 on centos-01.qa.lab:31010]
(state=,code=0)
{noformat}
{noformat}
2017-04-03 09:42:45,845 [271de6e9-e9f3-51bc-7091-1a2df15f12a7:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query id
271de6e9-e9f3-51bc-7091-1a2df15f12a7: select * from `test.json`
...
2017-04-03 09:42:45,935 [271de6e9-e9f3-51bc-7091-1a2df15f12a7:frag:0:0] INFO
o.a.d.e.vector.complex.fn.JsonReader - User Error Occurred: Failure while
parsing JSON. Found token of [VALUE_STRING]. Drill currently only supports
parsing json strings that contain either lists or maps. The root object cannot
be a scalar.
org.apache.drill.common.exceptions.UserException: DATA_READ ERROR: Failure
while parsing JSON. Found token of [VALUE_STRING]. Drill currently only
supports parsing json strings that contain either lists or maps. The root
object cannot be a scalar.
Line 1
Column 3
Field
[Error Id: 535e08fb-89b0-443b-86dc-032e40bbb4e1 ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.fn.JsonReader.writeToVector(JsonReader.java:305)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.fn.JsonReader.write(JsonReader.java:217)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.store.easy.json.JSONRecordReader.next(JSONRecordReader.java:206)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:179)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:232)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method)
[na:1.8.0_91]
at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_91]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
[hadoop-common-2.7.0-mapr-1607.jar:na]
at
[jira] [Created] (DRILL-5401) wrong results - IS NULL / IS NOT NULL over a list in JSON
Khurram Faraaz created DRILL-5401:
-
Summary: wrong results - IS NULL / IS NOT NULL over a list in JSON
Key: DRILL-5401
URL: https://issues.apache.org/jira/browse/DRILL-5401
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Khurram Faraaz
Checking if a list is null or if it is not null, results in incorrect results.
Drill 1.11.0 commit id: adbf363d
Data used in test
{noformat}
[root@centos-01 ~]# cat empty_array.json
{ "a": { "b": { "c": [ { "d": { "e": "f" } } ] } } }
{ "a": { "b": { "c": [] } } }
{noformat}
{noformat}
0: jdbc:drill:schema=dfs.tmp> alter session set `store.json.all_text_mode`=true;
+---++
| ok | summary |
+---++
| true | store.json.all_text_mode updated. |
+---++
1 row selected (0.189 seconds)
0: jdbc:drill:schema=dfs.tmp> select * from `empty_array.json`;
++
| a|
++
| {"b":{"c":[{"d":{"e":"f"}}]}} |
| {"b":{"c":[]}} |
++
2 rows selected (0.138 seconds)
/* wrong results */
0: jdbc:drill:schema=dfs.tmp> select * from `empty_array.json` t where t.b.c IS
NULL;
++
| a|
++
| {"b":{"c":[{"d":{"e":"f"}}]}} |
| {"b":{"c":[]}} |
++
2 rows selected (0.152 seconds)
/* wrong results */
0: jdbc:drill:schema=dfs.tmp> select * from `empty_array.json` t where t.b.c IS
NOT NULL;
++
| a |
++
++
No rows selected (0.154 seconds)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (DRILL-5392) implement castINT(BIT-REQUIRED) and castINT(BIT-OPTIONAL)
Khurram Faraaz created DRILL-5392:
-
Summary: implement castINT(BIT-REQUIRED) and castINT(BIT-OPTIONAL)
Key: DRILL-5392
URL: https://issues.apache.org/jira/browse/DRILL-5392
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Khurram Faraaz
We need to implement castINT(BIT-REQUIRED) and castINT(BIT-OPTIONAL)
Below SQL fails on Drill 1.11.0
{noformat}
0: jdbc:drill:schema=dfs.tmp> SELECT res, CASE res WHEN true THEN
CAST('1990-10-10 22:40:50' AS TIMESTAMP) ELSE null END
. . . . . . . . . . . . . . > FROM
. . . . . . . . . . . . . . > (
. . . . . . . . . . . . . . > SELECT
. . . . . . . . . . . . . . > (CASE WHEN (true) THEN 1 ELSE null END) res
. . . . . . . . . . . . . . > FROM (values(1)) foo
. . . . . . . . . . . . . . > ) foobar;
Error: PLAN ERROR: Failure while materializing expression in constant
expression evaluator [CASE(=(CASE(true, 1, null), true), CAST('1990-10-10
22:40:50'):TIMESTAMP(0) NOT NULL, null)]. Errors:
Error in expression at index -1. Error: Missing function implementation:
[castINT(BIT-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.
[Error Id: 1c0c850f-99b7-4d64-a0e3-d6ee70b11259 on centos-01.qa.lab:31010]
(state=,code=0)
{noformat}
Stack trace from drillbit.log
{noformat}
2017-03-28 09:50:54,500 [2725ce01-529d-3479-7467-ed4aa20d3ca5:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query id
2725ce01-529d-3479-7467-ed4aa20d3ca5: SELECT res, CASE res WHEN true THEN
CAST('1990-10-10 22:40:50' AS TIMESTAMP) ELSE null END
FROM
(
SELECT
(CASE WHEN (true) THEN 1 ELSE null END) res
FROM (values(1)) foo
) foobar
2017-03-28 09:50:54,585 [2725ce01-529d-3479-7467-ed4aa20d3ca5:foreman] INFO
o.a.d.e.p.logical.DrillConstExecutor - User Error Occurred: Failure while
materializing expression in constant expression evaluator [CASE(=(CASE(true, 1,
null), true), CAST('1990-10-10 22:40:50'):TIMESTAMP(0) NOT NULL, null)].
Errors:
Error in expression at index -1. Error: Missing function implementation:
[castINT(BIT-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.
org.apache.drill.common.exceptions.UserException: PLAN ERROR: Failure while
materializing expression in constant expression evaluator [CASE(=(CASE(true, 1,
null), true), CAST('1990-10-10 22:40:50'):TIMESTAMP(0) NOT NULL, null)].
Errors:
Error in expression at index -1. Error: Missing function implementation:
[castINT(BIT-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.
[Error Id: 1c0c850f-99b7-4d64-a0e3-d6ee70b11259 ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.planner.logical.DrillConstExecutor.reduce(DrillConstExecutor.java:131)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.calcite.rel.rules.ReduceExpressionsRule.reduceExpressions(ReduceExpressionsRule.java:499)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at
org.apache.calcite.rel.rules.ReduceExpressionsRule$1.onMatch(ReduceExpressionsRule.java:241)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at
org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303)
[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:404)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:343)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:240)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:290)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:168)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:131)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:79)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1050)
[jira] [Created] (DRILL-5382) Error: Missing function implementation: [isnotnull(MAP-REQUIRED)]
Khurram Faraaz created DRILL-5382:
-
Summary: Error: Missing function implementation:
[isnotnull(MAP-REQUIRED)]
Key: DRILL-5382
URL: https://issues.apache.org/jira/browse/DRILL-5382
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Khurram Faraaz
Projecting a map from JSON data and filter non null values results in
SchemaChangeException, Error: Missing function implementation:
[isnotnull(MAP-REQUIRED)].
Data used in test is available here - wget
http://data.githubarchive.org/2015-01-01-15.json.gz
Drill 1.11.0 commit id: adbf363
{noformat}
0: jdbc:drill:schema=dfs.tmp> select org from `2015-01-01-15.json` where org is
not null;
Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to materialize
incoming schema. Errors:
Error in expression at index -1. Error: Missing function implementation:
[isnotnull(MAP-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--..
Fragment 0:0
[Error Id: 3a776c68-6476-4bd8-a9e6-928bfc2ef5bd on centos-01.qa.lab:31010]
(state=,code=0)
0: jdbc:drill:schema=dfs.tmp>
{noformat}
Select without IS NOT NULL filter, returns correct results.
{noformat}
apache drill 1.11.0-SNAPSHOT
"just drill it"
0: jdbc:drill:schema=dfs.tmp> select org from `2015-01-01-15.json`;
+-+
| org |
+-+
| {} |
| {} |
| {} |
|
{"id":9285252,"login":"visionmedia","gravatar_id":"","url":"https://api.github.com/orgs/visionmedia","avatar_url":"https://avatars.githubusercontent.com/u/9285252?"}
|
| {} |
| {} |
| {} |
...
...
| {} |
| {} |
| {} |
|
{"id":9216151,"login":"osp","gravatar_id":"","url":"https://api.github.com/orgs/osp","avatar_url":"https://avatars.githubusercontent.com/u/9216151?"}
|
| {} |
|
{"id":296074,"login":"zendframework","gravatar_id":"","url":"https://api.github.com/orgs/zendframework","avatar_url":"https://avatars.githubusercontent.com/u/296074?"}
|
| {} |
| {} |
| {} |
| {} |
|
{"id":9216151,"login":"osp","gravatar_id":"","url":"https://api.github.com/orgs/osp","avatar_url":"https://avatars.githubusercontent.com/u/9216151?"}
|
| {} |
|
{"id":5822862,"login":"SuaveIO","gravatar_id":"","url":"https://api.github.com/orgs/SuaveIO","avatar_url":"https://avatars.githubusercontent.com/u/5822862?"}
|
|
{"id":5822862,"login":"SuaveIO","gravatar_id":"","url":"https://api.github.com/orgs/SuaveIO","avatar_url":"https://avatars.githubusercontent.com/u/5822862?"}
|
| {} |
|
{"id":5822862,"login":"SuaveIO","gravatar_id":"","url":"https://api.github.com/orgs/SuaveIO","avatar_url":"https://avatars.githubusercontent.com/u/5822862?"}
|
|
{"id":2918581,"login":"twbs","gravatar_id":"","url":"https://api.github.com/orgs/twbs","avatar_url":"https://avatars.githubusercontent.com/u/2918581?"}
|
|
{"id":1104713,"login":"s9y","gravatar_id":"","url":"https://api.github.com/orgs/s9y","avatar_url":"https://avatars.githubusercontent.com/u/1104713?"}
|
| {} |
| {} |
| {} |
| {} |
+-+
11,351 rows selected (0.865 seconds)
{noformat}
Stack trace from drillbit.log
{noformat}
Caused by: org.apache.drill.exec.exception.SchemaChangeException: Failure while
trying to materialize incoming schema. Errors:
Error in expression at index -1. Error: Missing function implementation:
[isnotnull(MAP-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--..
at
org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.generateSV2Filterer(FilterRecordBatch.java:186)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.setupNewSchema(FilterRecordBatch.java:111)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:78)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:93)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[jira] [Created] (DRILL-5367) Join query returns wrong results
Khurram Faraaz created DRILL-5367:
-
Summary: Join query returns wrong results
Key: DRILL-5367
URL: https://issues.apache.org/jira/browse/DRILL-5367
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.10.0
Environment: 3 node cluster
Reporter: Khurram Faraaz
Join query returns wrong results
Drill 1.10.0 does not return any results.
{noformat}
0: jdbc:drill:schema=dfs.tmp> SELECT * FROM using_f1 JOIN (SELECT * FROM
using_f2) foo USING(col_prime);
+-+++-+-+---+--+-+-+--+--++
| col_dt | col_state | col_prime | col_varstr | col_id | col_name |
col_dt0 | col_state0 | col_prime0 | col_varstr0 | col_id0 | col_name0 |
+-+++-+-+---+--+-+-+--+--++
+-+++-+-+---+--+-+-+--+--++
No rows selected (0.314 seconds)
{noformat}
{noformat}
Explain plan for above failing query
0: jdbc:drill:schema=dfs.tmp> explain plan for SELECT * FROM using_f1 JOIN
(SELECT * FROM using_f2) foo USING(col_prime);
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 ProjectAllowDup(*=[$0], *0=[$1])
00-02Project(T49¦¦*=[$0], T48¦¦*=[$2])
00-03 Project(T49¦¦*=[$1], col_prime=[$2], T48¦¦*=[$0])
00-04HashJoin(condition=[=($2, $0)], joinType=[inner])
00-06 Project(T48¦¦*=[$0])
00-08Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=maprfs:///tmp/using_f2]],
selectionRoot=maprfs:/tmp/using_f2, numFiles=1, usedMetadataFile=false,
columns=[`*`]]])
00-05 Project(T49¦¦*=[$0], col_prime=[$1])
00-07Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=maprfs:///tmp/using_f1]],
selectionRoot=maprfs:/tmp/using_f1, numFiles=1, usedMetadataFile=false,
columns=[`*`]]])
{noformat}
Whereas Postgres 9.3 returns expected results for the same data.
{noformat}
postgres=# SELECT * FROM using_f1 JOIN (SELECT * FROM using_f2) foo
USING(col_prime);
col_prime | col_dt | col_state | col_varstr
| col_id | col_name | col_dt | col_state |
col_varstr| col_id | col_name
---++---+---
++---++---+-
++--
103 | 2014-12-24 | TX|
LUW2QzWGdJfnxHrqm3vwyndzRBFwH8l5xVDaM3hTiZAanp
j | 19462 | Julie Lennox | 1990-01-11 | WV|
KKzEOgle6E5h
NANduNAAIp9DQnGLGxO | 54217 | Derek Wilson
103 | 1985-07-18 | CA|
aYQ2uLpPxebGGRvcX0fahrAOO4yhkDRvMPES6PuYsIfwkU
Mrcq6NSdt0j | 48987 | Lillian Lupo | 1990-01-11 | WV|
KKzEOgle6E5h
NANduNAAIp9DQnGLGxO | 54217 | Derek Wilson
103 | 1988-02-27 | SC|
OcVKheHMyeKLgcvamrJHUxKyCGGJGci3Y9ht2LI9T5Ek1n
wckB| 52840 | Martha Rose | 1990-01-11 | WV|
KKzEOgle6E5h
NANduNAAIp9DQnGLGxO | 54217 | Derek Wilson
211 | 1989-12-06 | SD| HHlmvV4
| 1131 | Kenneth Hayes | 1989-05-31 | MT|
yhHfCGaCqnAr
XUCD4jRoZQ4fj6IQIKZHUGLlIsSr1L7voCE3lEmj3DOSFqJ0Kq | 49191 | Joan Stein
43 | 2006-01-24 | NV|
EJAN2JjRqoQWgp7rHLT1yPMBR50g1Kil3klu1vPritFKB2
5EjmL1tLXleagAP | 30179 | William Strassel | 2006-03-02 | MI|
W9G0nWo8QNtH
r9YxOscigPbtXEtNPZ | 44849 | Catherine Turner
193 | 1990-01-14 | NV| 9nd3po1bnyasqINVA
| 47775 | James Walters
...
1990-01-14 | NV| 9nd3po1bnyasqINVA
| 47775 | James Walters | 1980-04-22 | ID|
jR8jr1lqDprU
FPhAX4xZnulndYNd3 | 5876 | Rosie Johnson
5 | 2004-01-27 | KS| 0A8Gwqm66k6wQ1KzcUdSQKZU3AZtPImxb8
| 57787 | Dean Salazar | 1997-09-13 | SC|
uq35Sqf1GfPt
IV1mE2CzwxKaX | 17041 | Dorothy Paulsen
5 | 1999-07-12 | UT|
hQk9DBx1egLNIpi9btvN7GPewgvPROWaNArsxAbRILW3dN
fwi526 | 38130 | Beverly Flores| 1997-09-13 | SC|
uq35Sqf1GfPt
IV1mE2CzwxKaX | 17041 | Dorothy Paulsen
(239 rows)
{noformat}
--
This message was sent by Atlassian JIRA
[jira] [Created] (DRILL-5357) Partition pruning information not available in query plan for COUNT aggregate query
Khurram Faraaz created DRILL-5357:
-
Summary: Partition pruning information not available in query plan
for COUNT aggregate query
Key: DRILL-5357
URL: https://issues.apache.org/jira/browse/DRILL-5357
Project: Apache Drill
Issue Type: Bug
Components: Query Planning & Optimization
Affects Versions: 1.10.0
Environment: 3 node CentOS cluster
Reporter: Khurram Faraaz
Priority: Critical
We are not seeing partition pruning information in the query plan for the
below, COUNT(*) and COUNT() query ?
Drill 1.10.0-SNAPSHOT
git commit id: b657d44f
parquet table has 6 columns
total number of rows = 1638640
{noformat}
0: jdbc:drill:schema=dfs.tmp> CREATE TABLE tbl_prtn_prune_01 PARTITION BY
(col_state)
AS
SELECT CAST(columns[0] AS DATE) col_date,
CAST(columns[1] AS CHAR(3)) col_state,
CAST(columns[2] AS INTEGER) col_prime,
CAST(columns[3] AS VARCHAR(256)) col_varstr,
CAST(columns[4] AS INTEGER) col_id,
CAST(columns[5] AS VARCHAR(50)) col_name
from `partition_prune_data.csv`;
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 1638640|
+---++
1 row selected (17.675 seconds)
0: jdbc:drill:schema=dfs.tmp> select COUNT(*) from tbl_prtn_prune_01 where
col_state = 'CA';
+-+
| EXPR$0 |
+-+
| 35653 |
+-+
1 row selected (0.471 seconds)
0: jdbc:drill:schema=dfs.tmp> explain plan for select COUNT(*) from
tbl_prtn_prune_01 where col_state = 'CA';
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 Project(EXPR$0=[$0])
00-02Project(EXPR$0=[$0])
00-03
Scan(groupscan=[org.apache.drill.exec.store.pojo.PojoRecordReader@1d4bb67d[columns
= null, isStarQuery = false, isSkipQuery = false]])
{noformat}
And then I did a REFRESH TABLE METADATA on the parquet table
{noformat}
0: jdbc:drill:schema=dfs.tmp> refresh table metadata tbl_prtn_prune_01;
+---+-+
| ok | summary |
+---+-+
| true | Successfully updated metadata for table tbl_prtn_prune_01. |
+---+-+
1 row selected (0.321 seconds)
0: jdbc:drill:schema=dfs.tmp> explain plan for select COUNT(col_state) from
tbl_prtn_prune_01 where col_state = 'CA';
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 Project(EXPR$0=[$0])
00-02Project(EXPR$0=[$0])
00-03
Scan(groupscan=[org.apache.drill.exec.store.pojo.PojoRecordReader@2e0f4be9[columns
= null, isStarQuery = false, isSkipQuery = false]])
0: jdbc:drill:schema=dfs.tmp> explain plan for select COUNT(*) from
tbl_prtn_prune_01 where col_state = 'CA';
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 Project(EXPR$0=[$0])
00-02Project(EXPR$0=[$0])
00-03
Scan(groupscan=[org.apache.drill.exec.store.pojo.PojoRecordReader@3fc1f8e7[columns
= null, isStarQuery = false, isSkipQuery = false]])
0: jdbc:drill:schema=dfs.tmp> explain plan for select COUNT(col_date) from
tbl_prtn_prune_01 where col_state = 'CA';
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 Project(EXPR$0=[$0])
00-02Project(EXPR$0=[$0])
00-03
Scan(groupscan=[org.apache.drill.exec.store.pojo.PojoRecordReader@7afc851e[columns
= null, isStarQuery = false, isSkipQuery = false]])
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (DRILL-5347) Join query returns incorrect results, returns an extra column.
Khurram Faraaz created DRILL-5347:
-
Summary: Join query returns incorrect results, returns an extra
column.
Key: DRILL-5347
URL: https://issues.apache.org/jira/browse/DRILL-5347
Project: Apache Drill
Issue Type: Bug
Components: Query Planning & Optimization
Affects Versions: 1.10.0
Reporter: Khurram Faraaz
Priority: Critical
Join query returns incorrect results, returns an extra column.
Note that the columns in both the parquet files are in different order.
Postgres returns correct results.
{noformat}
postgres=# create table using_l1(c1 int, c2 varchar(25), c3 date);
CREATE TABLE
postgres=# create table using_r1(c2 varchar(25), c3 date, c1 int);
CREATE TABLE
postgres=# copy using_l1 from '/root/using_clause/using_l1.txt' DELIMITER ','
CSV;
COPY 17
postgres=# copy using_r1 from '/root/using_clause/using_r1.txt' DELIMITER ','
CSV;
COPY 18
postgres=# select * from using_l1 JOIN using_r1 USING(c1);
c1 |c2 | c3 | c2 | c3
---+---+++
1 | test | 1982-09-11 | teststring | 2009-10-10
1 | test | 1982-09-11 | yes| 2001-09-09
2 | string| 1989-04-12 | never | 1989-11-11
3 | foo | 2000-10-12 | none | 1967-03-05
3 | foo | 2000-10-12 | here | 2011-09-09
4 | foobar| 2015-01-07 | champ | 2008-06-07
5 | bar | 2004-05-08 | bar| 2010-02-04
17 | barfoobar | 1992-12-06 | hello | 2014-07-02
19 | hello | 1980-10-10 | hi | 2000-01-01
0 | now | 2007-07-02 | foobar | 2011-01-01
10 | here | 2009-11-11 | done | 1947-08-14
9 | there | 1993-09-03 | there | 1993-09-03
22 | hi| 2000-01-04 | there | 2016-03-22
13 | no| 2016-10-01 | foo| 1982-10-11
13 | no| 2016-10-01 | no | 1992-12-06
(15 rows)
{noformat}
Results from Drill 1.10.0 (3dfb4972), for the same input data we see an extra
column c10 in the results of join query.
{noformat}
## using_l1 => (c1 INTEGER, c2 VARCHAR(25), c3 DATE)
0: jdbc:drill:schema=dfs.tmp> CREATE TABLE using_l1 AS SELECT CAST(columns[0]
AS INTEGER) c1, CAST(columns[1] AS VARCHAR(25)) c2, CAST(columns[2] AS DATE) c3
FROM `using_l1.txt`;
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 17 |
+---++
1 row selected (0.258 seconds)
## using_r1 => (c2 VARCHAR, c3 DATE, c1 INTEGER)
0: jdbc:drill:schema=dfs.tmp> CREATE TABLE using_r1 AS SELECT CAST(columns[0]
AS VARCHAR(25)) c2, CAST(columns[1] AS DATE) c3, CAST(columns[2] AS INTEGER) c1
FROM `using_r1.txt`;
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 18 |
+---++
1 row selected (0.249 seconds)
0: jdbc:drill:schema=dfs.tmp>
## Last column c10 is un-expected in the results of the JOIN
0: jdbc:drill:schema=dfs.tmp> select * from using_l1 JOIN using_r1 USING(c1);
+++-+-+-++
| c1 | c2 | c3 | c20 | c30 | c10 |
+++-+-+-++
| 1 | test | 1982-09-11 | yes | 2001-09-09 | 1 |
| 1 | test | 1982-09-11 | teststring | 2009-10-10 | 1 |
| 2 | string | 1989-04-12 | never | 1989-11-11 | 2 |
| 3 | foo| 2000-10-12 | here| 2011-09-09 | 3 |
| 3 | foo| 2000-10-12 | none| 1967-03-05 | 3 |
| 4 | foobar | 2015-01-07 | champ | 2008-06-07 | 4 |
| 5 | bar| 2004-05-08 | bar | 2010-02-04 | 5 |
| 17 | barfoobar | 1992-12-06 | hello | 2014-07-02 | 17 |
| 19 | hello | 1980-10-10 | hi | 2000-01-01 | 19 |
| 0 | now| 2007-07-02 | foobar | 2011-01-01 | 0 |
| 10 | here | 2009-11-11 | done| 1947-08-14 | 10 |
| 9 | there | 1993-09-03 | there | 1993-09-03 | 9 |
| 22 | hi | 2000-01-04 | there | 2016-03-22 | 22 |
| 13 | no | 2016-10-01 | no | 1992-12-06 | 13 |
| 13 | no | 2016-10-01 | foo | 1982-10-11 | 13 |
+++-+-+-++
15 rows selected (0.358 seconds)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (DRILL-5345) USING() should return error, when column in not present in the right table
Khurram Faraaz created DRILL-5345:
-
Summary: USING() should return error, when column in
not present in the right table
Key: DRILL-5345
URL: https://issues.apache.org/jira/browse/DRILL-5345
Project: Apache Drill
Issue Type: Bug
Components: Query Planning & Optimization
Affects Versions: 1.10.0
Reporter: Khurram Faraaz
When the column used in, USING(column) is not present in the right table of the
JOIN, we show report an error to user instead of returning zero rows.
{noformat}
::=
USING
::=
::=
[ { }... ]
::=
{noformat}
Postgres 9.3 reports an error for such a scenario, because column c1 is not
present in the other table in the join.
{noformat}
postgres=# select * from tbl_l JOIN tbl_r USING (c1);
ERROR: column "c1" specified in USING clause does not exist in right table
{noformat}
Drill 1.10.0 returns zero rows in one case and RuntimeException in the other
case. In both cases Drill should return a meaning error message, like the one
reported by Postgres.
{noformat}
0: jdbc:drill:schema=dfs.tmp> create table tbl_l as select cast(columns[0] as
integer) c1, cast(columns[1] as varchar(25)) c2 from `tbl_l.txt`;
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 10 |
+---++
1 row selected (0.197 seconds)
0: jdbc:drill:schema=dfs.tmp> create table tbl_r as select cast(columns[0] as
integer) c3, cast(columns[1] as varchar(25)) c4 from `tbl_r.txt`;
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 10 |
+---++
1 row selected (0.221 seconds)
## this is not right, we should report error for below query.
0: jdbc:drill:schema=dfs.tmp> select * from tbl_l JOIN tbl_r USING (c1);
+-+-+-+-+
| c1 | c2 | c3 | c4 |
+-+-+-+-+
+-+-+-+-+
No rows selected (0.222 seconds)
## we should return a meaningful error message
0: jdbc:drill:schema=dfs.tmp> select * from tbl_l JOIN tbl_r USING (c2);
Error: SYSTEM ERROR: DrillRuntimeException: Join only supports implicit casts
between 1. Numeric data
2. Varchar, Varbinary data 3. Date, Timestamp data Left type: INT, Right type:
VARCHAR. Add explicit casts to avoid this error
Fragment 0:0
[Error Id: 56107e6e-54b5-4905-9aa4-b0af2aea4dd9 on centos-01.qa.lab:31010]
(state=,code=0)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (DRILL-5336) Columns returned by select over CTAS created parquet are not in correct order.
Khurram Faraaz created DRILL-5336:
-
Summary: Columns returned by select over CTAS created parquet are
not in correct order.
Key: DRILL-5336
URL: https://issues.apache.org/jira/browse/DRILL-5336
Project: Apache Drill
Issue Type: Bug
Components: Storage - Parquet
Affects Versions: 1.10.0
Reporter: Khurram Faraaz
The ordering of the columns in the result of the SELECT over CTAS created
parquet file is not right.
Column col_int should be present before col_chr, however col_chr appears before
col_int, in the result of select.
Note that there is a UNION ALL in the CTAS's SELECT statement.
And each of the select statements in the UNION ALL has an ORDER BY
The problem seems to be with the use of LIMIT clause in the SELECT on
dfs.tmp.temp_tbl_unall.
Here is the parquet schema for the CTAS created parquet file. Note that col_int
appears before col_chr in the parquet schema too.
{noformat}
[root@centos-01 parquet-tools]# hadoop fs -get
/tmp/temp_tbl_unall/0_0_0.parquet .
[root@centos-01 parquet-tools]# ./parquet-schema 0_0_0.parquet
message root {
optional int32 col_int;
optional binary col_chr (UTF8);
optional binary col_vrchr1 (UTF8);
optional binary col_vrchr2 (UTF8);
}
{noformat}
Drill 1.10.0 git commit id : 3dfb4972
{noformat}
0: jdbc:drill:schema=dfs.tmp> CREATE TABLE dfs.tmp.temp_tbl_unall as ( SELECT
col_int, col_chr, col_vrchr1, col_vrchr2 FROM typeall_l order by col_int )
UNION ALL ( SELECT col_int, col_chr, col_vrchr1, col_vrchr2 FROM typeall_r
order by col_int );
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 1107 |
+---++
1 row selected (0.381 seconds)
0: jdbc:drill:schema=dfs.tmp> SELECT * FROM dfs.tmp.temp_tbl_unall ORDER BY
col_int LIMIT 100;
+-+-+++
| col_chr | col_int | col_vrchr1 | col_vrchr2 |
+-+-+++
| MI | 0 | Felecia Gourd | NLBQMg9 |
| DE | 1 | Alvina Jenkins | f9MqJlnNettlCVGcShifgMgnzL5FrZmHysoMBe6kDtA |
| HI | 1 | Fredrick Vanderburg |
eN3CNLW8FE5voAksuJCSYnMdJrVown7my6DiAlI8KhrG69kQoAxKFJmOHPVca1FjGyHWd5Ag53vvODvKB8YwqXcbDihjR0DDbed1cgs7L1tndiPRvU1OreN5ByB8pF0QisgwSBWRKRvS8RVOzA3CyxOpjyxVujRLLlctww0jWwn09m3iINTi6Delw
|
| CA | 19 | John Doe | test string |
| CA | 19 | John Doe | test string |
...
| LA | 6854 | William Burk |
5krBT7wj8BkoiRUWV9HjkyIT1DRpPj6bNixK15g4gs9IEsKc5myCyzMKQk5k1 |
++--+---+-+
| col_chr | col_int | col_vrchr1 | col_vrchr2 |
++--+---+-+
| IN | 6870 | Caroline Bell |
M2811poVmVJLuxqsHz0jzRSGrAJDXfl3UuE0Iz8ldqvRURURvq2dO4Q1358eiureI20NCGBl9lBpoKPc78TWS0gsWhIt280E8JZPQpj7lOJXnHUmvydDiBPgAzNoGn7SSP6xYlnMyBhvWRxB5NF3I9vszosjmpW1Yx7et56QvwLfWBb3unJPnrxVYXX5tAfeyednJ4A90aOE2dhMXy1wLwewMJ91SWBEUM8TU3aGikQ5Ax6dDhDBQLaP
|
++--+---+-+
100 rows selected (0.173 seconds)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (DRILL-5278) CTTAS store.format=csv, returns one extra record
[
https://issues.apache.org/jira/browse/DRILL-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz resolved DRILL-5278.
---
Resolution: Not A Bug
> CTTAS store.format=csv, returns one extra record
>
>
> Key: DRILL-5278
> URL: https://issues.apache.org/jira/browse/DRILL-5278
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Text & CSV
>Affects Versions: 1.10.0
>Reporter: Khurram Faraaz
>
> When store.format = csv, we see incorrect results returned, an extra record
> is returned, as compared to when store.format = parquet. The difference is
> seen when doing a count over a temporary table created in Drill.
> Drill 1.10.0 git commit ID : 300e9349
> Steps to reproduce the problem.
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> reset all;
> +---+---+
> | ok |summary|
> +---+---+
> | true | ALL updated. |
> +---+---+
> 1 row selected (0.279 seconds)
> 0: jdbc:drill:schema=dfs.tmp> ALTER SESSION SET `store.format`='csv';
> +---++
> | ok |summary |
> +---++
> | true | store.format updated. |
> +---++
> 1 row selected (0.21 seconds)
> 0: jdbc:drill:schema=dfs.tmp> CREATE TEMPORARY TABLE dfs.tmp.temp_tbl
> . . . . . . . . . . . . . . > AS
> . . . . . . . . . . . . . . > SELECT * FROM typeall_l;
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 105|
> +---++
> 1 row selected (0.233 seconds)
> 0: jdbc:drill:schema=dfs.tmp> SELECT COUNT(*) FROM dfs.tmp.temp_tbl;
> +-+
> | EXPR$0 |
> +-+
> | 106 |
> +-+
> 1 row selected (0.189 seconds)
> 0: jdbc:drill:schema=dfs.tmp> drop table dfs.tmp.temp_tbl;
> +---+-+
> | ok | summary |
> +---+-+
> | true | Temporary table [temp_tbl] dropped |
> +---+-+
> 1 row selected (0.186 seconds)
> 0: jdbc:drill:schema=dfs.tmp> ALTER SESSION SET `store.format`='parquet';
> +---++
> | ok |summary |
> +---++
> | true | store.format updated. |
> +---++
> 1 row selected (0.196 seconds)
> 0: jdbc:drill:schema=dfs.tmp> CREATE TEMPORARY TABLE dfs.tmp.temp_tbl
> . . . . . . . . . . . . . . > AS
> . . . . . . . . . . . . . . > SELECT * FROM typeall_l;
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 105|
> +---++
> 1 row selected (0.263 seconds)
> 0: jdbc:drill:schema=dfs.tmp> SELECT COUNT(*) FROM dfs.tmp.temp_tbl;
> +-+
> | EXPR$0 |
> +-+
> | 105 |
> +-+
> 1 row selected (0.169 seconds)
> 0: jdbc:drill:schema=dfs.tmp> drop table dfs.tmp.temp_tbl;
> +---+-+
> | ok | summary |
> +---+-+
> | true | Temporary table [temp_tbl] dropped |
> +---+-+
> 1 row selected (0.165 seconds)
> 0: jdbc:drill:schema=dfs.tmp>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (DRILL-5298) CTAS with 0 records from a SELECT query should create the table with metadata
[ https://issues.apache.org/jira/browse/DRILL-5298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Khurram Faraaz resolved DRILL-5298. --- Resolution: Duplicate > CTAS with 0 records from a SELECT query should create the table with metadata > - > > Key: DRILL-5298 > URL: https://issues.apache.org/jira/browse/DRILL-5298 > Project: Apache Drill > Issue Type: Bug > Components: Metadata, Query Planning & Optimization >Affects Versions: 1.9.0 > Environment: MapR 5.2 >Reporter: Senthilkumar > Fix For: 1.9.0 > > > Hello team, > I create a table in Drill using CTAS as > CREATE TABLE CTAS_TEST AS SELECT * FROM `hive.default`.`test` WHERE 1 = 0 > It runs successfully. > But the table is not getting created as there are 0 records getting returned > from the SELECT query. > CTAS should still go ahead and create the table with the column metadata. > When BI tools fire up multi-pass queries, with CTAS in the first query, the > subsequent queries fail because of a missing table. > In databases like SQL Server, Postgres, CTAS will create the table, even if > the SELECT doesnt return any rows. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-5288) JSON data not read as string when store.json.all_text_mode=true
Khurram Faraaz created DRILL-5288:
-
Summary: JSON data not read as string when
store.json.all_text_mode=true
Key: DRILL-5288
URL: https://issues.apache.org/jira/browse/DRILL-5288
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Reporter: Khurram Faraaz
Setting all text mode to true (at system/session level) does not help in this
case. Data from the JSON input file is not returned as string.
Drill 1.10.0 git commit id: 300e9349
Data used in test
{noformat}
[root@centos-01 ~]# cat f2.json
{"key":"string", "key":123, "key":[1,2,3], "key":true, "key":false, "key":null,
"key":{"key2":"b"}, "key":"2011-08-21"}
[root@centos-01 ~]#
{noformat}
{noformat}
0: jdbc:drill:schema=dfs.tmp> alter session set `store.json.all_text_mode`=true;
+---++
| ok | summary |
+---++
| true | store.json.all_text_mode updated. |
+---++
1 row selected (0.176 seconds)
0: jdbc:drill:schema=dfs.tmp> select key from `f2.json`;
Error: SYSTEM ERROR: IllegalStateException: You tried to start when you are
using a ValueWriter of type NullableVarCharWriterImpl.
Fragment 0:0
[Error Id: 3cb5b806-53b6-49bf-ab5e-59c666259463 on centos-01.qa.lab:31010]
(state=,code=0)
{noformat}
stack trace from drillbit.log
{noformat}
[Error Id: 3cb5b806-53b6-49bf-ab5e-59c666259463 on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: You tried to start when you are using a ValueWriter of
type NullableVarCharWriterImpl.
Fragment 0:0
[Error Id: 3cb5b806-53b6-49bf-ab5e-59c666259463 on centos-01.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_91]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: java.lang.IllegalStateException: You tried to start when you are
using a ValueWriter of type NullableVarCharWriterImpl.
at
org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.startList(AbstractFieldWriter.java:108)
~[vector-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.impl.NullableVarCharWriterImpl.startList(NullableVarCharWriterImpl.java:88)
~[vector-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.fn.JsonReader.writeDataAllText(JsonReader.java:621)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.fn.JsonReader.writeDataAllText(JsonReader.java:466)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.fn.JsonReader.writeDataSwitch(JsonReader.java:319)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.fn.JsonReader.writeToVector(JsonReader.java:262)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.fn.JsonReader.write(JsonReader.java:217)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.store.easy.json.JSONRecordReader.next(JSONRecordReader.java:206)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:179)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
[jira] [Created] (DRILL-5278) CTTAS store.format=csv, returns one extra record
Khurram Faraaz created DRILL-5278:
-
Summary: CTTAS store.format=csv, returns one extra record
Key: DRILL-5278
URL: https://issues.apache.org/jira/browse/DRILL-5278
Project: Apache Drill
Issue Type: Bug
Components: Storage - Text & CSV
Affects Versions: 1.10.0
Reporter: Khurram Faraaz
When store.format = csv, we see incorrect results returned, an extra record is
returned, as compared to when store.format = parquet. The difference is seen
when doing a count over a temporary table created in Drill.
Drill 1.10.0 git commit ID : 300e9349
Steps to reproduce the problem.
{noformat}
0: jdbc:drill:schema=dfs.tmp> reset all;
+---+---+
| ok |summary|
+---+---+
| true | ALL updated. |
+---+---+
1 row selected (0.279 seconds)
0: jdbc:drill:schema=dfs.tmp> ALTER SESSION SET `store.format`='csv';
+---++
| ok |summary |
+---++
| true | store.format updated. |
+---++
1 row selected (0.21 seconds)
0: jdbc:drill:schema=dfs.tmp> CREATE TEMPORARY TABLE dfs.tmp.temp_tbl
. . . . . . . . . . . . . . > AS
. . . . . . . . . . . . . . > SELECT * FROM typeall_l;
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 105|
+---++
1 row selected (0.233 seconds)
0: jdbc:drill:schema=dfs.tmp> SELECT COUNT(*) FROM dfs.tmp.temp_tbl;
+-+
| EXPR$0 |
+-+
| 106 |
+-+
1 row selected (0.189 seconds)
0: jdbc:drill:schema=dfs.tmp> drop table dfs.tmp.temp_tbl;
+---+-+
| ok | summary |
+---+-+
| true | Temporary table [temp_tbl] dropped |
+---+-+
1 row selected (0.186 seconds)
0: jdbc:drill:schema=dfs.tmp> ALTER SESSION SET `store.format`='parquet';
+---++
| ok |summary |
+---++
| true | store.format updated. |
+---++
1 row selected (0.196 seconds)
0: jdbc:drill:schema=dfs.tmp> CREATE TEMPORARY TABLE dfs.tmp.temp_tbl
. . . . . . . . . . . . . . > AS
. . . . . . . . . . . . . . > SELECT * FROM typeall_l;
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 105|
+---++
1 row selected (0.263 seconds)
0: jdbc:drill:schema=dfs.tmp> SELECT COUNT(*) FROM dfs.tmp.temp_tbl;
+-+
| EXPR$0 |
+-+
| 105 |
+-+
1 row selected (0.169 seconds)
0: jdbc:drill:schema=dfs.tmp> drop table dfs.tmp.temp_tbl;
+---+-+
| ok | summary |
+---+-+
| true | Temporary table [temp_tbl] dropped |
+---+-+
1 row selected (0.165 seconds)
0: jdbc:drill:schema=dfs.tmp>
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (DRILL-5191) OutOfMemoryException - TPCDS query4
Khurram Faraaz created DRILL-5191:
-
Summary: OutOfMemoryException - TPCDS query4
Key: DRILL-5191
URL: https://issues.apache.org/jira/browse/DRILL-5191
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.10.0
Environment: 4 node cluster CentOS
Reporter: Khurram Faraaz
Priority: Critical
TPC-DS Query4 against SF100 on Drill 1.10.0 (ee399317), on a 4 node CentOS
cluster
Query4 => https://raw.githubusercontent.com/Agirish/tpcds/master/query4.sql
total number of fragments : 1,125
Stack trace from drillbit.log
{noformat}
2017-01-11 11:17:57,007 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO
o.a.d.e.w.fragment.FragmentExecutor -
2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: State change requested
AWAITING_ALLOCATION --> RUNNING
2017-01-11 11:17:57,008 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO
o.a.d.e.w.f.FragmentStatusReporter - 2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5:
State to report: RUNNING
2017-01-11 11:17:57,009 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:33:5] INFO
o.a.d.e.w.fragment.FragmentExecutor -
2789eba3-60f0-0b2f-eba8-82331735d5c4:33:5: State change requested RUNNING -->
FAILED
2017-01-11 11:17:57,009 [BitServer-6] ERROR o.a.d.exec.rpc.RpcExceptionHandler
- Exception in RPC communication. Connection: /10.10.100.202:31012 <-->
/10.10.100.201:44712 (data server). Closing connection.
org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate buffer
of size 16384 due to memory limit. Current allocation: 16777216
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216)
~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:191)
~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.memory.DrillByteBufAllocator.buffer(DrillByteBufAllocator.java:49)
~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.memory.DrillByteBufAllocator.ioBuffer(DrillByteBufAllocator.java:64)
~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
2017-01-11 11:17:57,009 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:24:5] INFO
o.a.d.e.w.fragment.FragmentExecutor -
2789eba3-60f0-0b2f-eba8-82331735d5c4:24:5: State change requested
AWAITING_ALLOCATION --> FAILED
2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:63:5] INFO
o.a.d.e.w.fragment.FragmentExecutor -
2789eba3-60f0-0b2f-eba8-82331735d5c4:63:5: State change requested
AWAITING_ALLOCATION --> RUNNING
2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:63:5] INFO
o.a.d.e.w.f.FragmentStatusReporter - 2789eba3-60f0-0b2f-eba8-82331735d5c4:63:5:
State to report: RUNNING
2017-01-11 11:17:57,010 [2789eba3-60f0-0b2f-eba8-82331735d5c4:frag:24:5] INFO
o.a.d.e.w.fragment.FragmentExecutor -
2789eba3-60f0-0b2f-eba8-82331735d5c4:24:5: State change requested FAILED -->
FINISHED
2017-01-11 11:17:57,010 [BitServer-6] INFO
o.a.d.exec.rpc.ProtobufLengthDecoder - Channel is closed, discarding remaining
3240924 byte(s) in buffer.
2017-01-11 11:17:57,011 [BitServer-10] ERROR o.a.d.exec.rpc.RpcExceptionHandler
- Exception in RPC communication. Connection: /10.10.100.202:31012 <-->
/10.10.100.202:52127 (data server). Closing connection.
org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate buffer
of size 4096 due to memory limit. Current allocation: 16777216
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216)
~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:191)
~[drill-memory-base-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
[jira] [Created] (DRILL-5188) TPC-DS query16 fails - IllegalArgumentException: Target must be less than target count
Khurram Faraaz created DRILL-5188:
-
Summary: TPC-DS query16 fails - IllegalArgumentException: Target
must be less than target count
Key: DRILL-5188
URL: https://issues.apache.org/jira/browse/DRILL-5188
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.10.0
Environment: 4 node cluster
Reporter: Khurram Faraaz
TPC-DS query 16 fails with below Exception. Query 32, Query 92, query 95 also
fail due to same Exception, on Drill 1.10.0 (commit ID: bbcf4b76) against SF1
data.
Details from drillbit.log and stack trace.
{noformat}
2017-01-10 11:42:54,769 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query id
278b3740-fa7b-0f27-5c42-92f2f63f7f3d: SELECT
Count(DISTINCT cs_order_number) AS `order count` ,
Sum(cs_ext_ship_cost) AS `total shipping cost` ,
Sum(cs_net_profit) AS `total net profit`
FROM catalog_sales cs1 ,
date_dim ,
customer_address ,
call_center
WHEREd_date BETWEEN '2002-3-01' AND (
Cast('2002-3-01' AS DATE) + INTERVAL '60' day)
AND cs1.cs_ship_date_sk = d_date_sk
AND cs1.cs_ship_addr_sk = ca_address_sk
AND ca_state = 'IA'
AND cs1.cs_call_center_sk = cc_call_center_sk
AND cc_county IN ('Williamson County',
'Williamson County',
'Williamson County',
'Williamson County',
'Williamson County' )
AND EXISTS
(
SELECT *
FROM catalog_sales cs2
WHERE cs1.cs_order_number = cs2.cs_order_number
ANDcs1.cs_warehouse_sk <> cs2.cs_warehouse_sk)
AND NOT EXISTS
(
SELECT *
FROM catalog_returns cr1
WHERE cs1.cs_order_number = cr1.cr_order_number)
ORDER BY count(DISTINCT cs_order_number)
LIMIT 100
2017-01-10 11:42:54,887 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-01-10 11:42:54,888 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-01-10 11:42:54,895 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-01-10 11:42:54,895 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-01-10 11:42:54,902 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-01-10 11:42:54,903 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-01-10 11:42:54,908 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-01-10 11:42:54,909 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-01-10 11:42:54,919 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-01-10 11:42:54,920 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-01-10 11:42:54,946 [278b3740-fa7b-0f27-5c42-92f2f63f7f3d:foreman] ERROR
o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: IllegalArgumentException:
Target must be less than target count, 3
[Error Id: 61befda9-3479-48a4-a6ff-181703aea1b7 on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalArgumentException: Target must be less than target count, 3
[Error Id: 61befda9-3479-48a4-a6ff-181703aea1b7 on centos-01.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:824)
[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:944)
[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:281)
[drill-java-exec-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_91]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected
exception during fragment initialization: Target must be less than target
count, 3
[jira] [Created] (DRILL-5187) TPC-DS query 72 takes 32 minutes to complete
Khurram Faraaz created DRILL-5187: - Summary: TPC-DS query 72 takes 32 minutes to complete Key: DRILL-5187 URL: https://issues.apache.org/jira/browse/DRILL-5187 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 1.10.0 Environment: 4 node cluster Reporter: Khurram Faraaz Priority: Critical TPC-DS query takes close to 32 minutes to complete execution on a 4 node CentOS cluster, on Drill 1.10.0 against SF1 data. Attached here are, - drillbit.log (has only query 72 information) - JSON profile for query 72 - screen shot of fragment information from profiles tab on UI HASH_JOIN operator takes 21minutes (fragment : 04-xx-53). HASH_JOIN operator seems to be taking longer, it can be verified by looking at the profile information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-5174) when different Drillbit versions are up on a cluster - we need to report an error
Khurram Faraaz created DRILL-5174:
-
Summary: when different Drillbit versions are up on a cluster - we
need to report an error
Key: DRILL-5174
URL: https://issues.apache.org/jira/browse/DRILL-5174
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.10.0
Reporter: Khurram Faraaz
Drill 1.10.0 git commit : bbcf4b76
Note that one of the nodes in the 4 node cluster had a Drillbit from 1.9.0 and
the other three Drillbits are of 1.10.0 version. We report an error on the Web
UI (see DRILL-4604) for such a scenario, we need a similar message when
querying sys.drillbits table from sqlline.
{noformat}
0: jdbc:drill:schema=dfs.tmp> select * from sys.drillbits;
+---++---++--+--+
| hostname | user_port | control_port | data_port | current |
version |
+---++---++--+--+
| centos-03.qa.lab | 31010 | 31011 | 31012 | false|
1.10.0-SNAPSHOT |
| centos-02.qa.lab | 31010 | 31011 | 31012 | false|
|
| centos-04.qa.lab | 31010 | 31011 | 31012 | false|
1.10.0-SNAPSHOT |
| centos-01.qa.lab | 31010 | 31011 | 31012 | true |
1.10.0-SNAPSHOT |
+---++---++--+--+
4 rows selected (0.285 seconds)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-5165) wrong results - LIMIT ALL and OFFSET clause in same query
Khurram Faraaz created DRILL-5165:
-
Summary: wrong results - LIMIT ALL and OFFSET clause in same query
Key: DRILL-5165
URL: https://issues.apache.org/jira/browse/DRILL-5165
Project: Apache Drill
Issue Type: Bug
Components: Query Planning & Optimization
Affects Versions: 1.10.0
Reporter: Khurram Faraaz
Priority: Critical
This issue was reported by a user on Drill's user list.
Drill 1.10.0 commit ID : bbcf4b76
I tried a similar query on apache Drill 1.10.0 and Drill returns wrong results
when compared to Postgres, for a query that uses LIMIT ALL and OFFSET clause in
the same query. We need to file a JIRA to track this issue.
{noformat}
0: jdbc:drill:schema=dfs.tmp> select col_int from typeall_l order by 1 limit
all offset 10;
+--+
| col_int |
+--+
+--+
No rows selected (0.211 seconds)
0: jdbc:drill:schema=dfs.tmp> select col_int from typeall_l order by col_int
limit all offset 10;
+--+
| col_int |
+--+
+--+
No rows selected (0.24 seconds)
{noformat}
Query => select col_int from typeall_l limit all offset 10;
Drill 1.10.0 returns 85 rows
whereas for same query,
postgres=# select col_int from typeall_l limit all offset 10;
Postgres 9.3 returns 95 rows, which is the correct expected result.
Query plan for above query that returns wrong results
{noformat}
0: jdbc:drill:schema=dfs.tmp> explain plan for select col_int from typeall_l
limit all offset 10;
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 Project(col_int=[$0])
00-02SelectionVectorRemover
00-03 Limit(offset=[10])
00-04Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=maprfs:///tmp/typeall_l]], selectionRoot=maprfs:/tmp/typeall_l,
numFiles=1, usedMetadataFile=false, columns=[`col_int`]]])
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-5164) equi-join query results in CompileException
Khurram Faraaz created DRILL-5164:
-
Summary: equi-join query results in CompileException
Key: DRILL-5164
URL: https://issues.apache.org/jira/browse/DRILL-5164
Project: Apache Drill
Issue Type: Bug
Components: Execution - Codegen
Affects Versions: 1.9.0
Reporter: Khurram Faraaz
Priority: Critical
Drill 1.9.0
git commit ID : 4c1b420b
4 node CentOS cluster
JSON file has 4095 keys (columns)
{noformat}
0: jdbc:drill:schema=dfs.tmp> select * from `manyColsInJson.json` t1,
`manyColsInJson.json` t2 where t1.key2000 = t2.key2000;
Error: SYSTEM ERROR: CompileException: File
'org.apache.drill.exec.compile.DrillJavaFileObject[HashJoinProbeGen294.java]',
Line 16397, Column 17: HashJoinProbeGen294.java:16397: error: code too large
public void doSetup(FragmentContext context, VectorContainer buildBatch,
RecordBatch probeBatch, RecordBatch outgoing)
^ (compiler.err.limit.code)
Fragment 0:0
[Error Id: 7d0efa7e-e183-4c40-939a-4908699f94bf on centos-01.qa.lab:31010]
(state=,code=0)
{noformat}
Stack trace from drillbit.log
{noformat}
2016-12-26 09:52:11,321 [279f17fd-c8f0-5d18-1124-76099f0a5cc8:frag:0:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: CompileException: File
'org.apache.drill.exec.compile.DrillJavaFileObject[HashJoinProbeGen294.java]',
Line 16397, Column 17: HashJoinProbeGen294.java:16397: error: code too large
public void doSetup(FragmentContext context, VectorContainer buildBatch,
RecordBatch probeBatch, RecordBatch outgoing)
^ (compiler.err.limit.code)
Fragment 0:0
[Error Id: 7d0efa7e-e183-4c40-939a-4908699f94bf on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
CompileException: File
'org.apache.drill.exec.compile.DrillJavaFileObject[HashJoinProbeGen294.java]',
Line 16397, Column 17: HashJoinProbeGen294.java:16397: error: code too large
public void doSetup(FragmentContext context, VectorContainer buildBatch,
RecordBatch probeBatch, RecordBatch outgoing)
^ (compiler.err.limit.code)
Fragment 0:0
[Error Id: 7d0efa7e-e183-4c40-939a-4908699f94bf on centos-01.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
~[drill-common-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.9.0.jar:1.9.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_91]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: org.apache.drill.common.exceptions.DrillRuntimeException:
org.apache.drill.exec.exception.SchemaChangeException:
org.apache.drill.exec.exception.ClassTransformationException:
java.util.concurrent.ExecutionException:
org.apache.drill.exec.exception.ClassTransformationException: Failure
generating transformation classes for value:
package org.apache.drill.exec.test.generated;
...
public class HashJoinProbeGen294 {
NullableVarCharVector[] vv0;
NullableVarCharVector vv3;
NullableVarCharVector[] vv6;
...
vv49137 .copyFromSafe((probeIndex), (outIndex), vv49134);
vv49143 .copyFromSafe((probeIndex), (outIndex), vv49140);
vv49149 .copyFromSafe((probeIndex), (outIndex), vv49146);
}
}
public void __DRILL_INIT__()
throws SchemaChangeException
{
}
}
at
org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:302)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.compile.CodeCompiler$Loader.load(CodeCompiler.java:78)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.compile.CodeCompiler$Loader.load(CodeCompiler.java:74)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
~[guava-18.0.jar:na]
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
~[guava-18.0.jar:na]
... 34 common frames omitted
Caused by: org.codehaus.commons.compiler.CompileException: File
'org.apache.drill.exec.compile.DrillJavaFileObject[HashJoinProbeGen294.java]',
Line 16397, Column 17: HashJoinProbeGen294.java:16397: error: code too large
public void
[jira] [Created] (DRILL-5141) Difference in output date format on sqlline vs Drill's Web UI
Khurram Faraaz created DRILL-5141:
-
Summary: Difference in output date format on sqlline vs Drill's
Web UI
Key: DRILL-5141
URL: https://issues.apache.org/jira/browse/DRILL-5141
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.9.0
Reporter: Khurram Faraaz
Priority: Minor
Difference in output date format on sqlline vs Drill's Web UI
Drill 1.9.0 git commit ID : 4c1b420b
>From sqlline prompt
{noformat}
0: jdbc:drill:schema=dfs.tmp> SELECT cast('2016/09/22' as date) FROM
(VALUES(1));
+-+
| EXPR$0|
+-+
| 2016-09-22 |
+-+
1 row selected (0.474 seconds)
{noformat}
The same query returns a date in a different output format on the web UI
{noformat}
2016-09-22T00:00:00.000Z
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-5130) UNION ALL difference in results
Khurram Faraaz created DRILL-5130:
-
Summary: UNION ALL difference in results
Key: DRILL-5130
URL: https://issues.apache.org/jira/browse/DRILL-5130
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.9.0
Reporter: Khurram Faraaz
Drill 1.9.0 git commit ID: 51246693
{noformat}
0: jdbc:drill:schema=dfs.tmp> values(1,2,3,4,5,6) union all
values(7,8,9,10,11,12);
+-+-+-+-+-+-+
| EXPR$0 | EXPR$1 | EXPR$2 | EXPR$3 | EXPR$4 | EXPR$5 |
+-+-+-+-+-+-+
| 7 | 8 | 9 | 10 | 11 | 12 |
| 7 | 8 | 9 | 10 | 11 | 12 |
+-+-+-+-+-+-+
2 rows selected (0.209 seconds)
{noformat}
Postgres 9.3
{noformat}
postgres=# values(1,2,3,4,5,6) union all values(7,8,9,10,11,12);
column1 | column2 | column3 | column4 | column5 | column6
-+-+-+-+-+-
1 | 2 | 3 | 4 | 5 | 6
7 | 8 | 9 | 10 | 11 | 12
(2 rows)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-5099) support OFFSET values greater than one, for LEAD & LAG window functions
Khurram Faraaz created DRILL-5099:
-
Summary: support OFFSET values greater than one, for LEAD & LAG
window functions
Key: DRILL-5099
URL: https://issues.apache.org/jira/browse/DRILL-5099
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.9.0
Reporter: Khurram Faraaz
Priority: Minor
Provide support for OFFSET values greater than one, for LEAD & LAG window
functions.
Adding [~adeneche] comments from the dev list here
{noformat}
Things that need to be done to make Lag (or Lead) support offsets other
than 1:
- WindowFunction.Lead should extract the offset value from its FunctionCall
argument, you can look at WindowFunctionNtile.numTilesFromExpression() for
and example on how to do that.
- make sure calls to copyNext() and copyPrev() in NoFrameSupportTemplate
use the offset and not the hard coded value (you already figured that out)
- finally make sure you update UnsupportedOperatorsVisitor to no longer
throw an exception when we pass an offset value other than 1 to Lead or
Lag. Just search for DRILL-3596 in that class and you will find the if
block that need to be removed
I think this should be enough to get it to work in the general case.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-5077) Memory Leak -
Khurram Faraaz created DRILL-5077:
-
Summary: Memory Leak -
Key: DRILL-5077
URL: https://issues.apache.org/jira/browse/DRILL-5077
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.9.0
Environment: 4 node cluster CentOS
Reporter: Khurram Faraaz
Priority: Blocker
terminating foreman drillbit while a query is running results in memory leak
Drill 1.9.0 git commit id: 4312d65b
Stack trace from drillbit.log
{noformat}
2016-11-28 06:12:45,338 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query id
27c43522-79ca-989f-3659-d7ccbc77e2e7: select count(*) from `twoKeyJsn.json`
2016-11-28 06:12:45,602 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2016-11-28 06:12:45,602 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2016-11-28 06:12:45,603 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2016-11-28 06:12:45,603 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2016-11-28 06:12:45,603 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2016-11-28 06:12:45,633 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2016-11-28 06:12:45,669 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using
1 threads. Time: 33ms total, 33.494123ms avg, 33ms max.
2016-11-28 06:12:45,669 [27c43522-79ca-989f-3659-d7ccbc77e2e7:foreman] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using
1 threads. Earliest start: 9.54 μs, Latest start: 9.54 μs, Average
start: 9.54 μs .
2016-11-28 06:12:45,913 [27c43522-79ca-989f-3659-d7ccbc77e2e7:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 27c43522-79ca-989f-3659-d7ccbc77e2e7:0:0:
State change requested AWAITING_ALLOCATION --> RUNNING
2016-11-28 06:12:45,913 [27c43522-79ca-989f-3659-d7ccbc77e2e7:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 27c43522-79ca-989f-3659-d7ccbc77e2e7:0:0:
State to report: RUNNING
Mon Nov 28 06:12:48 UTC 2016 Terminating drillbit pid 28004
2016-11-28 06:12:48,697 [Drillbit-ShutdownHook#0] INFO
o.apache.drill.exec.server.Drillbit - Received shutdown request.
2016-11-28 06:12:55,749 [pool-6-thread-2] INFO
o.a.drill.exec.rpc.data.DataServer - closed eventLoopGroup
io.netty.channel.nio.NioEventLoopGroup@15bcfacd in 1017 ms
2016-11-28 06:12:55,750 [pool-6-thread-2] INFO
o.a.drill.exec.service.ServiceEngine - closed dataPool in 1018 ms
2016-11-28 06:12:57,749 [Drillbit-ShutdownHook#0] WARN
o.apache.drill.exec.work.WorkManager - Closing WorkManager but there are 1
running fragments.
2016-11-28 06:12:57,751 [27c43522-79ca-989f-3659-d7ccbc77e2e7:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 27c43522-79ca-989f-3659-d7ccbc77e2e7:0:0:
State change requested RUNNING --> FAILED
2016-11-28 06:12:57,751 [27c43522-79ca-989f-3659-d7ccbc77e2e7:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 27c43522-79ca-989f-3659-d7ccbc77e2e7:0:0:
State change requested FAILED --> FINISHED
2016-11-28 06:12:57,756 [27c43522-79ca-989f-3659-d7ccbc77e2e7:frag:0:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException
Fragment 0:0
[Error Id: 2df2f9a1-a7bf-4454-a31b-717ab4ebd815 on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
NullPointerException
Fragment 0:0
[Error Id: 2df2f9a1-a7bf-4454-a31b-717ab4ebd815 on centos-01.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
~[drill-common-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.9.0.jar:1.9.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_101]
at
[jira] [Created] (DRILL-5049) difference in results - correlated subquery interacting with null equality join
Khurram Faraaz created DRILL-5049:
-
Summary: difference in results - correlated subquery interacting
with null equality join
Key: DRILL-5049
URL: https://issues.apache.org/jira/browse/DRILL-5049
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.9.0
Reporter: Khurram Faraaz
Here is a query that uses null equality join. Drill 1.9.0 returns 124 records,
whereas Postgres 9.3 returns 145 records. I am on Drill 1.9.0 git commit id:
db308549
I have attached the results from Drill 1.9.0 and Postgres, please review.
{noformat}
0: jdbc:drill:schema=dfs.tmp> explain plan for
. . . . . . . . . . . . . . > SELECT *
. . . . . . . . . . . . . . > FROM `t_alltype.parquet` t1
. . . . . . . . . . . . . . > WHERE EXISTS
. . . . . . . . . . . . . . > (
. . . . . . . . . . . . . . > SELECT *
. . . . . . . . . . . . . . > FROM `t_alltype.parquet` t2
. . . . . . . . . . . . . . > WHERE t1.c4 = t2.c4 OR (t1.c4 IS
NULL AND t2.c4 IS NULL)
. . . . . . . . . . . . . . > );
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 Project(*=[$0])
00-02Project(T30¦¦*=[$0])
00-03 HashJoin(condition=[AND(=($1, $2), =($1, $3))], joinType=[inner])
00-05Project(T30¦¦*=[$0], c4=[$1])
00-07 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=maprfs:///tmp/t_alltype.parquet]],
selectionRoot=maprfs:/tmp/t_alltype.parquet, numFiles=1,
usedMetadataFile=false, columns=[`*`]]])
00-04HashAgg(group=[{0, 1}], agg#0=[MIN($2)])
00-06 Project(c40=[$1], c400=[$1], $f0=[true])
00-08HashJoin(condition=[IS NOT DISTINCT FROM($0, $1)],
joinType=[inner])
00-10 Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=maprfs:///tmp/t_alltype.parquet]],
selectionRoot=maprfs:/tmp/t_alltype.parquet, numFiles=1,
usedMetadataFile=false, columns=[`c4`]]])
00-09 Project(c40=[$0])
00-11HashAgg(group=[{0}])
00-12 Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=maprfs:///tmp/t_alltype.parquet]],
selectionRoot=maprfs:/tmp/t_alltype.parquet, numFiles=1,
usedMetadataFile=false, columns=[`c4`]]])
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-5040) Interrupted CTAS should not succeed & should not create physical file on disk
Khurram Faraaz created DRILL-5040:
-
Summary: Interrupted CTAS should not succeed & should not create
physical file on disk
Key: DRILL-5040
URL: https://issues.apache.org/jira/browse/DRILL-5040
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.9.0
Reporter: Khurram Faraaz
We should not allow CTAS to succeed (i.e create physical file on disk ) in the
case where it was interrupted. (vis Ctrl-C)
Drill 1.9.0
git commit ID : db30854
Consider the below CTAS that was interrupted using Ctrl-C
{noformat}
0: jdbc:drill:schema=dfs.tmp> create table temp_t1 as select * from
`twoKeyJsn.json`;
[ issue Ctrl-C while the above CTAS is running ]
No rows affected (7.694 seconds)
{noformat}
I verified that physical file was created on disk, even though the above CTAS
was Canceled
{noformat}
[root@centos-01 ~]# hadoop fs -ls /tmp/temp_t1*
-rwxr-xr-x 3 root root 36713198 2016-11-14 10:51 /tmp/temp_t1/0_0_0.parquet
{noformat}
We are able to do a select on the CTAS table (above) that was Canceled.
{noformat}
0: jdbc:drill:schema=dfs.tmp> select count(*) from temp_t1;
+--+
| EXPR$0 |
+--+
| 3747840 |
+--+
1 row selected (0.183 seconds)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-5039) NPE - CTAS PARTITION BY ()
Khurram Faraaz created DRILL-5039:
-
Summary: NPE - CTAS PARTITION BY ()
Key: DRILL-5039
URL: https://issues.apache.org/jira/browse/DRILL-5039
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.9.0
Reporter: Khurram Faraaz
We are seeing an NPE, when CTAS is used with PARTITION BY (
and all columns are projected in SELECT of CTAS.
Drill 1.9.0
git commit ID : db30854
{noformat}
0: jdbc:drill:schema=dfs.tmp> CREATE TABLE dfs.tmp.temp_tbl
. . . . . . . . . . . . . . > PARTITION BY (col_chr)
. . . . . . . . . . . . . . > AS
. . . . . . . . . . . . . . > SELECT col_int, col_chr, col_vrchr1, col_vrchr2 ,
col_dt, col_tim, col_tmstmp , col_flt, col_intrvl_yr , col_intrvl_day , col_bln
. . . . . . . . . . . . . . > FROM typeall_l;
Error: SYSTEM ERROR: NullPointerException
Fragment 0:0
[Error Id: ab6c199e-cb61-42dd-ae22-0090eea22ec5 on centos-01.qa.lab:31010]
(state=,code=0)
{noformat}
{noformat}
2016-11-12 19:54:14,901 [27d88c99-a64d-0317-ba3b-d78195cf85cc:frag:0:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException
Fragment 0:0
[Error Id: ab6c199e-cb61-42dd-ae22-0090eea22ec5 on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
NullPointerException
Fragment 0:0
[Error Id: ab6c199e-cb61-42dd-ae22-0090eea22ec5 on centos-01.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
~[drill-common-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.9.0.jar:1.9.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_91]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: java.lang.NullPointerException: null
at
org.apache.drill.exec.expr.fn.impl.ByteFunctionHelpers.compare(ByteFunctionHelpers.java:100)
~[vector-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.test.generated.ProjectorGen3.doEval(ProjectorTemplate.java:88)
~[na:na]
at
org.apache.drill.exec.test.generated.ProjectorGen3.projectRecords(ProjectorTemplate.java:62)
~[na:na]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:199)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:91)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
~[drill-java-exec-1.9.0.jar:1.9.0]
at
[jira] [Created] (DRILL-5033) Query on JSON that has null as value for each key
Khurram Faraaz created DRILL-5033:
-
Summary: Query on JSON that has null as value for each key
Key: DRILL-5033
URL: https://issues.apache.org/jira/browse/DRILL-5033
Project: Apache Drill
Issue Type: Bug
Components: Storage - JSON
Affects Versions: 1.9.0
Reporter: Khurram Faraaz
Drill 1.9.0 git commit ID : 83513daf
Drill returns same result with or without `store.json.all_text_mode`=true
Note that each key in the JSON has null as its value.
[root@cent01 null_eq_joins]# cat right_all_nulls.json
{
"intKey" : null,
"bgintKey": null,
"strKey": null,
"boolKey": null,
"fltKey": null,
"dblKey": null,
"timKey": null,
"dtKey": null,
"tmstmpKey": null,
"intrvldyKey": null,
"intrvlyrKey": null
}
[root@cent01 null_eq_joins]#
Querying the above JSON file results in null as query result.
- We should see each of the keys in the JSON as a column in query result.
- And in each column the value should be a null value.
Current behavior does not look right.
{noformat}
0: jdbc:drill:schema=dfs.tmp> select * from `right_all_nulls.json`;
+---+
| * |
+---+
| null |
+---+
1 row selected (0.313 seconds)
{noformat}
Adding comment from [~julianhyde]
{noformat}
IMHO it is similar but not the same as DRILL-1256. Worth logging an issue and
let [~jnadeau] (or someone) put on the record what should be the behavior of an
empty record (empty JSON map) when it is top-level (as in this case) or in a
collection.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-5029) need better error - cast interval day to int or bigint
Khurram Faraaz created DRILL-5029:
-
Summary: need better error - cast interval day to int or bigint
Key: DRILL-5029
URL: https://issues.apache.org/jira/browse/DRILL-5029
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.8.0
Reporter: Khurram Faraaz
Priority: Minor
We need a better error message, today Drill returns an AssertionError
{noformat}
0: jdbc:drill:schema=dfs.tmp> values(cast('P162M24D' as INTERVAL DAY));
+-+
| EXPR$0 |
+-+
| P24D|
+-+
1 row selected (0.419 seconds)
{noformat}
A better error would be
ERROR: cannot cast type interval to int
{noformat}
0: jdbc:drill:schema=dfs.tmp> values(cast(cast('P162M24D' as INTERVAL DAY) as
INT));
Error: SYSTEM ERROR: AssertionError: Internal error: Conversion to relational
algebra failed to preserve datatypes:
validated type:
RecordType(INTEGER NOT NULL EXPR$0) NOT NULL
converted type:
RecordType(BIGINT NOT NULL EXPR$0) NOT NULL
rel:
LogicalProject(EXPR$0=[/INT(Reinterpret(CAST('P162M24D'):INTERVAL DAY NOT
NULL), 8640)])
LogicalValues(tuples=[[{ 0 }]])
[Error Id: 662716fb-c2c3-4032-8d92-835f8b0ec7ae on centos-01.qa.lab:31010]
(state=,code=0)
{noformat}
A better message would be
ERROR: cannot cast type interval to bigint
{noformat}
0: jdbc:drill:schema=dfs.tmp> values(cast(cast('P162M24D' as INTERVAL DAY) as
BIGINT));
Error: SYSTEM ERROR: AssertionError: todo: implement syntax
SPECIAL(Reinterpret(CAST('P162M24D'):INTERVAL DAY NOT NULL))
[Error Id: ef2c31cd-dee3-4f13-aca0-05c16185f789 on centos-01.qa.lab:31010]
(state=,code=0)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-5024) CTAS with LIMIT 0 query in SELECT stmt does not create parquet file
Khurram Faraaz created DRILL-5024:
-
Summary: CTAS with LIMIT 0 query in SELECT stmt does not create
parquet file
Key: DRILL-5024
URL: https://issues.apache.org/jira/browse/DRILL-5024
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.8.0
Reporter: Khurram Faraaz
Note that CTAS was successful
{noformat}
0: jdbc:drill:schema=dfs.tmp> create table regtbl_w0rows as select * from
typeall_l LIMIT 0;
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 0 |
+---++
1 row selected (0.51 seconds)
{noformat}
But a SELECT on CTAS created file fails.
{noformat}
0: jdbc:drill:schema=dfs.tmp> select * from regtbl_w0rows;
Error: VALIDATION ERROR: From line 1, column 15 to line 1, column 27: Table
'regtbl_w0rows' not found
SQL Query null
[Error Id: 0569cf98-3800-43ee-b635-aa101b016d46 on centos-01.qa.lab:31010]
(state=,code=0)
{noformat}
DROP on the CTAS created table also fails
{noformat}
0: jdbc:drill:schema=dfs.tmp> drop table regtbl_w0rows;
Error: VALIDATION ERROR: Table [regtbl_w0rows] not found
[Error Id: fb0b1ea8-f76d-42e2-b69c-4beae2798bdf on centos-01.qa.lab:31010]
(state=,code=0)
0: jdbc:drill:schema=dfs.tmp>
{noformat}
Verified that CTAS did not create a physical file in dfs.tmp schema
{noformat}
[test@cent01 bin]# hadoop fs -ls /tmp/regtbl_w0rows
ls: `/tmp/regtbl_w0rows': No such file or directory
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-5010) Equality join condition is treated as a MergeJoin and not as a HashJoin.
Khurram Faraaz created DRILL-5010:
-
Summary: Equality join condition is treated as a MergeJoin and not
as a HashJoin.
Key: DRILL-5010
URL: https://issues.apache.org/jira/browse/DRILL-5010
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.9.0
Reporter: Khurram Faraaz
Equality join condition is treated as a MergeJoin and not as a HashJoin.
Drill 1.9.0 git commit ID: 83513daf
Projecting the join columns results in merge join, whereas it should be doing a
HashJoin.
{noformat}
0: jdbc:drill:schema=dfs.tmp> explain plan for
. . . . . . . . . . . . . . > select t1.intKey, t2.intKey from `left.json` t1,
`right_all_nulls.json` t2
WHERE t1.intKey = t2.intKey OR ( t1.intKey IS NULL AND t2.intKey IS NULL);
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 Project(intKey=[$0], intKey0=[$1])
00-02Project(intKey=[$0], intKey0=[$1])
00-03 MergeJoin(condition=[IS NOT DISTINCT FROM($0, $1)],
joinType=[inner])
00-05SelectionVectorRemover
00-07 Sort(sort0=[$0], dir0=[ASC])
00-09Scan(groupscan=[EasyGroupScan
[selectionRoot=maprfs:/tmp/left.json, numFiles=1, columns=[`intKey`],
files=[maprfs:///tmp/left.json]]])
00-04Project(intKey0=[$0])
00-06 SelectionVectorRemover
00-08Sort(sort0=[$0], dir0=[ASC])
00-10 Scan(groupscan=[EasyGroupScan
[selectionRoot=maprfs:/tmp/right_all_nulls.json, numFiles=1,
columns=[`intKey`], files=[maprfs:///tmp/right_all_nulls.json]]])
{noformat}
Note that HashAgg and HashJoin were enabled.
{noformat}
| planner.enable_hashagg | BOOLEAN | SYSTEM | DEFAULT |
null| null| true | null |
| planner.enable_hashjoin| BOOLEAN | SYSTEM | DEFAULT |
null| null| true | null |
{noformat}
Doing a SELECT results in a HashJoin in the query plan
{noformat}
0: jdbc:drill:schema=dfs.tmp> explain plan for
. . . . . . . . . . . . . . > select * from `left.json` t1,
`right_all_nulls.json` t2
WHERE t1.intKey = t2.intKey OR ( t1.intKey IS NULL AND t2.intKey IS NULL);
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 ProjectAllowDup(*=[$0], *0=[$1])
00-02Project(T46¦¦*=[$0], T47¦¦*=[$2])
00-03 HashJoin(condition=[IS NOT DISTINCT FROM($1, $3)],
joinType=[inner])
00-04Project(T47¦¦*=[$0], intKey0=[$1])
00-06 Project(T47¦¦*=[$0], intKey=[$1])
00-08Scan(groupscan=[EasyGroupScan
[selectionRoot=maprfs:/tmp/right_all_nulls.json, numFiles=1, columns=[`*`],
files=[maprfs:///tmp/right_all_nulls.json]]])
00-05Project(T46¦¦*=[$0], intKey=[$1])
00-07 Scan(groupscan=[EasyGroupScan
[selectionRoot=maprfs:/tmp/left.json, numFiles=1, columns=[`*`],
files=[maprfs:///tmp/left.json]]])
{noformat}
Data used in above queries
{noformat}
[root@centos-01 null_eq_joins]# cat left.json
{
"intKey" : 123,
"bgintKey": 1234567,
"strKey": "this is a test string",
"boolKey": true,
"fltKey": 123.786,
"dblKey": 457.984,
"timKey": "18:30:45",
"dtKey": "1997-10-21",
"tmstmpKey": "2007-04-30 13:10:02.047",
"intrvldyKey": "P9DT38833S",
"intrvlyrKey": "P255M"
}
[root@centos-01 null_equality_joins]#
[root@centos-01 null_eq_joins]# cat right_all_nulls.json
{
"intKey" : null,
"bgintKey": null,
"strKey": null,
"boolKey": null,
"fltKey": null,
"dblKey": null,
"timKey": null,
"dtKey": null,
"tmstmpKey": null,
"intrvldyKey": null,
"intrvlyrKey": null
}
[root@centos-01 null_eq_joins]#
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-5003) need a better error message
Khurram Faraaz created DRILL-5003:
-
Summary: need a better error message
Key: DRILL-5003
URL: https://issues.apache.org/jira/browse/DRILL-5003
Project: Apache Drill
Issue Type: Bug
Components: Query Planning & Optimization
Affects Versions: 1.9.0
Reporter: Khurram Faraaz
Priority: Minor
We need a better error message, Drill 1.9.0 returns an AssertionError.
{noformat}
0: jdbc:drill:schema=dfs.tmp> SELECT *
. . . . . . . . . . . . . . > FROM (
. . . . . . . . . . . . . . > select * from `t_alltype.parquet` t1,
`t_alltype.parquet` t2
. . . . . . . . . . . . . . > WHERE EXISTS(t1.c4 = t2.c4) OR (t1.c4 IS NULL AND
t2.c4 IS NULL)
. . . . . . . . . . . . . . > );
Error: SYSTEM ERROR: AssertionError: Internal error: not a query: `t1`.`c4` =
`t2`.`c4`
[Error Id: 91e7af43-27a7-4577-802a-643705aeaf4f on centos-01.qa.lab:31010]
(state=,code=0)
{noformat}
{noformat}
0: jdbc:drill:schema=dfs.tmp> SELECT *
. . . . . . . . . . . . . . > FROM (
. . . . . . . . . . . . . . > select * from `t_alltype.parquet` t1,
`t_alltype.parquet` t2
. . . . . . . . . . . . . . > WHERE EXISTS(t1.c4 = t2.c4 OR (t1.c4 IS NULL AND
t2.c4 IS NULL))
. . . . . . . . . . . . . . > );
Error: SYSTEM ERROR: AssertionError: Internal error: not a query: `t1`.`c4` =
`t2`.`c4` OR `t1`.`c4` IS NULL AND `t2`.`c4` IS NULL
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-4985) CAST in join predicate in null equality join - "query cannot be planned error"
Khurram Faraaz created DRILL-4985:
-
Summary: CAST in join predicate in null equality join - "query
cannot be planned error"
Key: DRILL-4985
URL: https://issues.apache.org/jira/browse/DRILL-4985
Project: Apache Drill
Issue Type: Bug
Components: Query Planning & Optimization
Affects Versions: 1.9.0
Reporter: Khurram Faraaz
Use of CAST in join predicate results in, "query cannot be planned error", in a
Null Equality Join.
Drill 1.9.0, git commit ID: a29f1e29
{noformat}
0: jdbc:drill:schema=dfs.tmp> SELECT t1.col_int , t2.col_int FROM typeall_l t1,
typeall_r t2 WHERE CAST(t1.col_int AS BIGINT)= CAST(t2.col_int AS BIGINT) OR (
t1.col_int IS NULL AND t2.col_int IS NULL );
Error: UNSUPPORTED_OPERATION ERROR: This query cannot be planned possibly due
to either a cartesian join or an inequality join
[Error Id: 10452a20-c0a7-45b1-8ee7-065b28484738 on centos-01.qa.lab:31010]
(state=,code=0)
{noformat}
{noformat}
0: jdbc:drill:schema=dfs.tmp> SELECT t1.col_int , t2.col_int FROM typeall_l t1,
typeall_r t2 WHERE CAST(t1.col_int AS BIGINT)= CAST(t2.col_int AS BIGINT) OR (
CAST(t1.col_int AS BIGINT) IS NULL AND CAST(t2.col_int AS BIGINT) IS NULL );
Error: UNSUPPORTED_OPERATION ERROR: This query cannot be planned possibly due
to either a cartesian join or an inequality join
[Error Id: ca8eba6c-b8c0-42d9-a0a0-16e94f887280 on centos-01.qa.lab:31010]
(state=,code=0)
{noformat}
Whereas, without the CAST to BIGINT in the join predicate, query returns
correct results.
{noformat}
0: jdbc:drill:schema=dfs.tmp> SELECT t1.col_int , t2.col_int FROM typeall_l t1,
typeall_r t2 WHERE t1.col_int = t2.col_int OR ( t1.col_int IS NULL AND
t2.col_int IS NULL );
+--+---+
| col_int | col_int0 |
+--+---+
| 30410| 30410 |
| 37420| 37420 |
| 58583| 58583 |
| 1| 1 |
| null | null |
| null | null |
| null | null |
| 20326| 20326 |
| null | null |
| null | null |
| null | null |
| 19 | 19|
| 60718| 60718 |
| null | null |
| null | null |
| null | null |
+--+---+
16 rows selected (0.304 seconds)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
