[jira] [Updated] (DRILL-6285) IllegalArgumentException: Self-suppression not permitted
[
https://issues.apache.org/jira/browse/DRILL-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-6285:
--
Affects Version/s: 1.15.0
> IllegalArgumentException: Self-suppression not permitted
>
>
> Key: DRILL-6285
> URL: https://issues.apache.org/jira/browse/DRILL-6285
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - RPC
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Khurram Faraaz
>Priority: Minor
>
> When a query is marked as Canceled on Web UI, there is this Exception in
> drillbit.log
> Drill 1.13.0 commit cac2882d5a9e22fbc251e4caf622fe30242ad557
> {noformat}
> 2018-03-21 15:35:43,198 [UserServer-1] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 254d2796-7744-cc2a-d77e-5ec030b04211:0:0: State change requested RUNNING -->
> FAILED
> 2018-03-21 15:35:43,198 [254d2796-7744-cc2a-d77e-5ec030b04211:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 254d2796-7744-cc2a-d77e-5ec030b04211:0:0: State change requested FAILED -->
> FAILED
> 2018-03-21 15:35:43,199 [254d2796-7744-cc2a-d77e-5ec030b04211:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 254d2796-7744-cc2a-d77e-5ec030b04211:0:0: State change requested FAILED -->
> FAILED
> 2018-03-21 15:35:43,202 [254d2796-7744-cc2a-d77e-5ec030b04211:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 254d2796-7744-cc2a-d77e-5ec030b04211:0:0: State change requested FAILED -->
> FAILED
> 2018-03-21 15:35:43,202 [UserServer-1] WARN
> o.apache.drill.exec.rpc.RequestIdMap - Failure while attempting to fail rpc
> response.
> java.lang.IllegalArgumentException: Self-suppression not permitted
> at java.lang.Throwable.addSuppressed(Throwable.java:1043) ~[na:1.8.0_161]
> at
> org.apache.drill.common.DeferredException.addException(DeferredException.java:88)
> ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:97)
> ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:412)
> ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.access$700(FragmentExecutor.java:55)
> ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$ExecutorStateImpl.fail(FragmentExecutor.java:426)
> ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.ops.FragmentContextImpl.fail(FragmentContextImpl.java:233)
> ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.ops.FragmentContextImpl$1.accept(FragmentContextImpl.java:100)
> ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.ops.FragmentContextImpl$1.accept(FragmentContextImpl.java:97)
> ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at org.apache.drill.exec.ops.StatusHandler.failed(StatusHandler.java:42)
> ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.rpc.RequestIdMap$RpcListener.setException(RequestIdMap.java:139)
> ~[drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:76)
> [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:66)
> [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at com.carrotsearch.hppc.IntObjectHashMap.forEach(IntObjectHashMap.java:692)
> [hppc-0.7.1.jar:na]
> at org.apache.drill.exec.rpc.RequestIdMap.channelClosed(RequestIdMap.java:62)
> [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.rpc.AbstractRemoteConnection.channelClosed(AbstractRemoteConnection.java:192)
> [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.rpc.AbstractServerConnection.channelClosed(AbstractServerConnection.java:165)
> [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:167)
> [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:144)
> [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
> [netty-common-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
> [netty-common-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
> [netty-common-4.0.48.Final.jar:4.0.48.Final]
> at
>
[jira] [Updated] (DRILL-6285) IllegalArgumentException: Self-suppression not permitted
[
https://issues.apache.org/jira/browse/DRILL-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-6285:
--
Summary: IllegalArgumentException: Self-suppression not permitted (was:
Failure while attempting to fail rpc response.)
> IllegalArgumentException: Self-suppression not permitted
>
>
> Key: DRILL-6285
> URL: https://issues.apache.org/jira/browse/DRILL-6285
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - RPC
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Priority: Minor
>
> When a query is marked as Canceled on Web UI, there is this Exception in
> drillbit.log
> Drill 1.13.0 commit cac2882d5a9e22fbc251e4caf622fe30242ad557
> {noformat}
> 2018-03-21 15:35:43,198 [UserServer-1] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 254d2796-7744-cc2a-d77e-5ec030b04211:0:0: State change requested RUNNING -->
> FAILED
> 2018-03-21 15:35:43,198 [254d2796-7744-cc2a-d77e-5ec030b04211:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 254d2796-7744-cc2a-d77e-5ec030b04211:0:0: State change requested FAILED -->
> FAILED
> 2018-03-21 15:35:43,199 [254d2796-7744-cc2a-d77e-5ec030b04211:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 254d2796-7744-cc2a-d77e-5ec030b04211:0:0: State change requested FAILED -->
> FAILED
> 2018-03-21 15:35:43,202 [254d2796-7744-cc2a-d77e-5ec030b04211:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 254d2796-7744-cc2a-d77e-5ec030b04211:0:0: State change requested FAILED -->
> FAILED
> 2018-03-21 15:35:43,202 [UserServer-1] WARN
> o.apache.drill.exec.rpc.RequestIdMap - Failure while attempting to fail rpc
> response.
> java.lang.IllegalArgumentException: Self-suppression not permitted
> at java.lang.Throwable.addSuppressed(Throwable.java:1043) ~[na:1.8.0_161]
> at
> org.apache.drill.common.DeferredException.addException(DeferredException.java:88)
> ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:97)
> ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:412)
> ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.access$700(FragmentExecutor.java:55)
> ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$ExecutorStateImpl.fail(FragmentExecutor.java:426)
> ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.ops.FragmentContextImpl.fail(FragmentContextImpl.java:233)
> ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.ops.FragmentContextImpl$1.accept(FragmentContextImpl.java:100)
> ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.ops.FragmentContextImpl$1.accept(FragmentContextImpl.java:97)
> ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at org.apache.drill.exec.ops.StatusHandler.failed(StatusHandler.java:42)
> ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.rpc.RequestIdMap$RpcListener.setException(RequestIdMap.java:139)
> ~[drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:76)
> [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:66)
> [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at com.carrotsearch.hppc.IntObjectHashMap.forEach(IntObjectHashMap.java:692)
> [hppc-0.7.1.jar:na]
> at org.apache.drill.exec.rpc.RequestIdMap.channelClosed(RequestIdMap.java:62)
> [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.rpc.AbstractRemoteConnection.channelClosed(AbstractRemoteConnection.java:192)
> [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.rpc.AbstractServerConnection.channelClosed(AbstractServerConnection.java:165)
> [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:167)
> [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:144)
> [drill-rpc-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
> [netty-common-4.0.48.Final.jar:4.0.48.Final]
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
> [netty-common-4.0.48.Final.jar:4.0.48.Final]
> at
>
[jira] [Commented] (DRILL-7275) CTAS + CTE query fails with IllegalStateException: Read batch count [%d] should be greater than zero [0]
[
https://issues.apache.org/jira/browse/DRILL-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16845648#comment-16845648
]
Khurram Faraaz commented on DRILL-7275:
---
[~arina] I understand that, the dataset will help in the investigation and
repro. However this is coming from a source that does not want to publicly
share their dataset. I can try to reproduce the issue and add the test dataset
here, once I can successfully repro the failure on my setup.
> CTAS + CTE query fails with IllegalStateException: Read batch count [%d]
> should be greater than zero [0]
>
>
> Key: DRILL-7275
> URL: https://issues.apache.org/jira/browse/DRILL-7275
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.15.0
>Reporter: Khurram Faraaz
>Assignee: salim achouche
>Priority: Major
>
> CTAS + CTE query fails with IllegalStateException: Read batch count [%d]
> should be greater than zero [0]
> Precondition check fails on line 47 in VarLenFixedEntryReader.java
> {noformat}
> 44 final int expectedDataLen = columnPrecInfo.precision;
> 45 final int entrySz = 4 + columnPrecInfo.precision;
> 46 final int readBatch = getFixedLengthMaxRecordsToRead(valuesToRead,
> entrySz);
> 47 Preconditions.checkState(readBatch > 0, "Read batch count [%d] should be
> greater than zero", readBatch);
> {noformat}
> Stack trace from drillbit.log, also has the failing query.
> {noformat}
> 2019-05-13 14:40:14,090 [23268c40-ef3a-6349-5901-5762f6888971:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 23268c40-ef3a-6349-5901-5762f6888971 issued by scoop_stc: CREATE TABLE
> TEST_TEMPLATE_SCHEMA_creid.tbl_c_EquityProxyDailyReturn AS
> WITH
> che AS (
> SELECT * FROM
> TEST_TEMPLATE_SCHEMA_creid.tbl_c_CompositeHierarchyEntry_TimeVarying
> WHERE CompositeHierarchyName = 'AxiomaRegion/AxiomaSector/VectorUniverse'
> AND state = 'DupesRemoved'
> AND CompositeLevel = 'AxiomaRegion_1/AxiomaSector_1/VectorUniverse_0'
> ),
> ef AS (SELECT * FROM
> TEST_TEMPLATE_SCHEMA_creid.tbl_c_EquityDailyReturn_FXAdjusted WHERE Status =
> 'PresentInRawData'),
> d AS (SELECT * FROM TEST_TEMPLATE_SCHEMA_creid.tbl_r_BusinessDate WHERE
> IsWeekday),
> x AS
> (
> SELECT
> che.CompositeHierarchyName,
> che.State,
> che.CompositeNodeName,
> d.`Date` AS RecordDate,
> COUNT(che.CompositeNodeName) AS countDistinctConstituents,
> COUNT(ef.VectorListingId) AS countDataPoints,
> AVG(ef.DailyReturn) AS AvgReturn,
> AVG(ef.DailyReturnUSD) AS AvgReturnUSD,
> AVG(ef.NotionalReturnUSD) AS AvgNotionalReturnUSD
> FROM d
> INNER JOIN che ON d.`Date` BETWEEN che.CompositeUltimateChildStartDate AND
> che.CompositeUltimateChildEndDate
> LEFT OUTER JOIN ef ON d.`Date` = ef.RecordDate AND 'VectorListingId_' ||
> CAST(ef.VectorListingId AS VARCHAR(100)) = che.UltimateChild
> GROUP BY che.CompositeHierarchyName, che.State, che.CompositeNodeName,
> d.`Date`, d.IsWeekday, d.IsHoliday
> )
> SELECT * FROM x
> 2019-05-13 14:40:16,971 [23268c40-ef3a-6349-5901-5762f6888971:foreman] INFO
> o.a.d.e.p.s.h.CreateTableHandler - Creating persistent table
> [tbl_c_EquityProxyDailyReturn].
> ...
> ...
> 2019-05-13 14:40:20,036 [23268c40-ef3a-6349-5901-5762f6888971:frag:6:10] INFO
> o.a.d.exec.physical.impl.ScanBatch - User Error Occurred: Error in parquet
> record reader.
> Message:
> Hadoop path: /DEV/tbl_c_EquityDailyReturn_FXAdjusted/1_32_32.parquet
> Total records read: 0
> Row group index: 0
> Records in row group: 3243
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {
> optional int64 VectorListingId;
> optional int32 RecordDate (DATE);
> required binary Status (UTF8);
> required binary CurrencyISO (UTF8);
> optional double DailyReturn;
> optional double DailyReturnUSD;
> optional double NotionalReturnUSD;
> }
> , metadata: \{drill-writer.version=2, drill.version=1.15.0.0-mapr}}, blocks:
> [BlockMetaData\{3243, 204762 [ColumnMetaData{UNCOMPRESSED [VectorListingId]
> optional int64 VectorListingId [RLE, BIT_PACKED, PLAIN], 4},
> ColumnMetaData\{UNCOMPRESSED [RecordDate] optional int32 RecordDate (DATE)
> [RLE, BIT_PACKED, PLAIN], 26021}, ColumnMetaData\{UNCOMPRESSED [Status]
> required binary Status (UTF8) [BIT_PACKED, PLAIN], 39050},
> ColumnMetaData\{UNCOMPRESSED [CurrencyISO] required binary CurrencyISO (UTF8)
> [BIT_PACKED, PLAIN], 103968}, ColumnMetaData\{UNCOMPRESSED [DailyReturn]
> optional double DailyReturn [RLE, BIT_PACKED, PLAIN], 126715},
> ColumnMetaData\{UNCOMPRESSED [DailyReturnUSD] optional double DailyReturnUSD
> [RLE, BIT_PACKED, PLAIN], 152732}, ColumnMetaData\{UNCOMPRESSED
> [NotionalReturnUSD] optional double NotionalReturnUSD [RLE, BIT_PACKED,
> PLAIN], 178749}]}]}
[jira] [Created] (DRILL-7275) CTAS + CTE query fails with IllegalStateException: Read batch count [%d] should be greater than zero [0]
Khurram Faraaz created DRILL-7275:
-
Summary: CTAS + CTE query fails with IllegalStateException: Read
batch count [%d] should be greater than zero [0]
Key: DRILL-7275
URL: https://issues.apache.org/jira/browse/DRILL-7275
Project: Apache Drill
Issue Type: Bug
Components: Storage - Parquet
Affects Versions: 1.15.0
Reporter: Khurram Faraaz
CTAS + CTE query fails with IllegalStateException: Read batch count [%d] should
be greater than zero [0]
Precondition check fails on line 47 in VarLenFixedEntryReader.java
{noformat}
44 final int expectedDataLen = columnPrecInfo.precision;
45 final int entrySz = 4 + columnPrecInfo.precision;
46 final int readBatch = getFixedLengthMaxRecordsToRead(valuesToRead, entrySz);
47 Preconditions.checkState(readBatch > 0, "Read batch count [%d] should be
greater than zero", readBatch);
{noformat}
Stack trace from drillbit.log, also has the failing query.
{noformat}
2019-05-13 14:40:14,090 [23268c40-ef3a-6349-5901-5762f6888971:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query with id
23268c40-ef3a-6349-5901-5762f6888971 issued by scoop_stc: CREATE TABLE
TEST_TEMPLATE_SCHEMA_creid.tbl_c_EquityProxyDailyReturn AS
WITH
che AS (
SELECT * FROM
TEST_TEMPLATE_SCHEMA_creid.tbl_c_CompositeHierarchyEntry_TimeVarying
WHERE CompositeHierarchyName = 'AxiomaRegion/AxiomaSector/VectorUniverse'
AND state = 'DupesRemoved'
AND CompositeLevel = 'AxiomaRegion_1/AxiomaSector_1/VectorUniverse_0'
),
ef AS (SELECT * FROM
TEST_TEMPLATE_SCHEMA_creid.tbl_c_EquityDailyReturn_FXAdjusted WHERE Status =
'PresentInRawData'),
d AS (SELECT * FROM TEST_TEMPLATE_SCHEMA_creid.tbl_r_BusinessDate WHERE
IsWeekday),
x AS
(
SELECT
che.CompositeHierarchyName,
che.State,
che.CompositeNodeName,
d.`Date` AS RecordDate,
COUNT(che.CompositeNodeName) AS countDistinctConstituents,
COUNT(ef.VectorListingId) AS countDataPoints,
AVG(ef.DailyReturn) AS AvgReturn,
AVG(ef.DailyReturnUSD) AS AvgReturnUSD,
AVG(ef.NotionalReturnUSD) AS AvgNotionalReturnUSD
FROM d
INNER JOIN che ON d.`Date` BETWEEN che.CompositeUltimateChildStartDate AND
che.CompositeUltimateChildEndDate
LEFT OUTER JOIN ef ON d.`Date` = ef.RecordDate AND 'VectorListingId_' ||
CAST(ef.VectorListingId AS VARCHAR(100)) = che.UltimateChild
GROUP BY che.CompositeHierarchyName, che.State, che.CompositeNodeName,
d.`Date`, d.IsWeekday, d.IsHoliday
)
SELECT * FROM x
2019-05-13 14:40:16,971 [23268c40-ef3a-6349-5901-5762f6888971:foreman] INFO
o.a.d.e.p.s.h.CreateTableHandler - Creating persistent table
[tbl_c_EquityProxyDailyReturn].
...
...
2019-05-13 14:40:20,036 [23268c40-ef3a-6349-5901-5762f6888971:frag:6:10] INFO
o.a.d.exec.physical.impl.ScanBatch - User Error Occurred: Error in parquet
record reader.
Message:
Hadoop path: /DEV/tbl_c_EquityDailyReturn_FXAdjusted/1_32_32.parquet
Total records read: 0
Row group index: 0
Records in row group: 3243
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {
optional int64 VectorListingId;
optional int32 RecordDate (DATE);
required binary Status (UTF8);
required binary CurrencyISO (UTF8);
optional double DailyReturn;
optional double DailyReturnUSD;
optional double NotionalReturnUSD;
}
, metadata: \{drill-writer.version=2, drill.version=1.15.0.0-mapr}}, blocks:
[BlockMetaData\{3243, 204762 [ColumnMetaData{UNCOMPRESSED [VectorListingId]
optional int64 VectorListingId [RLE, BIT_PACKED, PLAIN], 4},
ColumnMetaData\{UNCOMPRESSED [RecordDate] optional int32 RecordDate (DATE)
[RLE, BIT_PACKED, PLAIN], 26021}, ColumnMetaData\{UNCOMPRESSED [Status]
required binary Status (UTF8) [BIT_PACKED, PLAIN], 39050},
ColumnMetaData\{UNCOMPRESSED [CurrencyISO] required binary CurrencyISO (UTF8)
[BIT_PACKED, PLAIN], 103968}, ColumnMetaData\{UNCOMPRESSED [DailyReturn]
optional double DailyReturn [RLE, BIT_PACKED, PLAIN], 126715},
ColumnMetaData\{UNCOMPRESSED [DailyReturnUSD] optional double DailyReturnUSD
[RLE, BIT_PACKED, PLAIN], 152732}, ColumnMetaData\{UNCOMPRESSED
[NotionalReturnUSD] optional double NotionalReturnUSD [RLE, BIT_PACKED, PLAIN],
178749}]}]} (Error in parquet record reader.
...
...
Hadoop path: /DEV/tbl_c_EquityDailyReturn_FXAdjusted/1_32_32.parquet
Total records read: 0
Row group index: 0
Records in row group: 3243
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {
optional int64 VectorListingId;
optional int32 RecordDate (DATE);
required binary Status (UTF8);
required binary CurrencyISO (UTF8);
optional double DailyReturn;
optional double DailyReturnUSD;
optional double NotionalReturnUSD;
}
, metadata: \{drill-writer.version=2, drill.version=1.15.0.0-mapr}}, blocks:
[BlockMetaData\{3243, 204762 [ColumnMetaData{UNCOMPRESSED [VectorListingId]
optional int64 VectorListingId [RLE, BIT_PACKED, PLAIN], 4},
ColumnMetaData\{UNCOMPRESSED [RecordDate] optional int32 RecordDate (DATE)
[RLE,
[jira] [Assigned] (DRILL-7275) CTAS + CTE query fails with IllegalStateException: Read batch count [%d] should be greater than zero [0]
[
https://issues.apache.org/jira/browse/DRILL-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz reassigned DRILL-7275:
-
Assignee: salim achouche
> CTAS + CTE query fails with IllegalStateException: Read batch count [%d]
> should be greater than zero [0]
>
>
> Key: DRILL-7275
> URL: https://issues.apache.org/jira/browse/DRILL-7275
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.15.0
>Reporter: Khurram Faraaz
>Assignee: salim achouche
>Priority: Major
>
> CTAS + CTE query fails with IllegalStateException: Read batch count [%d]
> should be greater than zero [0]
> Precondition check fails on line 47 in VarLenFixedEntryReader.java
> {noformat}
> 44 final int expectedDataLen = columnPrecInfo.precision;
> 45 final int entrySz = 4 + columnPrecInfo.precision;
> 46 final int readBatch = getFixedLengthMaxRecordsToRead(valuesToRead,
> entrySz);
> 47 Preconditions.checkState(readBatch > 0, "Read batch count [%d] should be
> greater than zero", readBatch);
> {noformat}
> Stack trace from drillbit.log, also has the failing query.
> {noformat}
> 2019-05-13 14:40:14,090 [23268c40-ef3a-6349-5901-5762f6888971:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 23268c40-ef3a-6349-5901-5762f6888971 issued by scoop_stc: CREATE TABLE
> TEST_TEMPLATE_SCHEMA_creid.tbl_c_EquityProxyDailyReturn AS
> WITH
> che AS (
> SELECT * FROM
> TEST_TEMPLATE_SCHEMA_creid.tbl_c_CompositeHierarchyEntry_TimeVarying
> WHERE CompositeHierarchyName = 'AxiomaRegion/AxiomaSector/VectorUniverse'
> AND state = 'DupesRemoved'
> AND CompositeLevel = 'AxiomaRegion_1/AxiomaSector_1/VectorUniverse_0'
> ),
> ef AS (SELECT * FROM
> TEST_TEMPLATE_SCHEMA_creid.tbl_c_EquityDailyReturn_FXAdjusted WHERE Status =
> 'PresentInRawData'),
> d AS (SELECT * FROM TEST_TEMPLATE_SCHEMA_creid.tbl_r_BusinessDate WHERE
> IsWeekday),
> x AS
> (
> SELECT
> che.CompositeHierarchyName,
> che.State,
> che.CompositeNodeName,
> d.`Date` AS RecordDate,
> COUNT(che.CompositeNodeName) AS countDistinctConstituents,
> COUNT(ef.VectorListingId) AS countDataPoints,
> AVG(ef.DailyReturn) AS AvgReturn,
> AVG(ef.DailyReturnUSD) AS AvgReturnUSD,
> AVG(ef.NotionalReturnUSD) AS AvgNotionalReturnUSD
> FROM d
> INNER JOIN che ON d.`Date` BETWEEN che.CompositeUltimateChildStartDate AND
> che.CompositeUltimateChildEndDate
> LEFT OUTER JOIN ef ON d.`Date` = ef.RecordDate AND 'VectorListingId_' ||
> CAST(ef.VectorListingId AS VARCHAR(100)) = che.UltimateChild
> GROUP BY che.CompositeHierarchyName, che.State, che.CompositeNodeName,
> d.`Date`, d.IsWeekday, d.IsHoliday
> )
> SELECT * FROM x
> 2019-05-13 14:40:16,971 [23268c40-ef3a-6349-5901-5762f6888971:foreman] INFO
> o.a.d.e.p.s.h.CreateTableHandler - Creating persistent table
> [tbl_c_EquityProxyDailyReturn].
> ...
> ...
> 2019-05-13 14:40:20,036 [23268c40-ef3a-6349-5901-5762f6888971:frag:6:10] INFO
> o.a.d.exec.physical.impl.ScanBatch - User Error Occurred: Error in parquet
> record reader.
> Message:
> Hadoop path: /DEV/tbl_c_EquityDailyReturn_FXAdjusted/1_32_32.parquet
> Total records read: 0
> Row group index: 0
> Records in row group: 3243
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {
> optional int64 VectorListingId;
> optional int32 RecordDate (DATE);
> required binary Status (UTF8);
> required binary CurrencyISO (UTF8);
> optional double DailyReturn;
> optional double DailyReturnUSD;
> optional double NotionalReturnUSD;
> }
> , metadata: \{drill-writer.version=2, drill.version=1.15.0.0-mapr}}, blocks:
> [BlockMetaData\{3243, 204762 [ColumnMetaData{UNCOMPRESSED [VectorListingId]
> optional int64 VectorListingId [RLE, BIT_PACKED, PLAIN], 4},
> ColumnMetaData\{UNCOMPRESSED [RecordDate] optional int32 RecordDate (DATE)
> [RLE, BIT_PACKED, PLAIN], 26021}, ColumnMetaData\{UNCOMPRESSED [Status]
> required binary Status (UTF8) [BIT_PACKED, PLAIN], 39050},
> ColumnMetaData\{UNCOMPRESSED [CurrencyISO] required binary CurrencyISO (UTF8)
> [BIT_PACKED, PLAIN], 103968}, ColumnMetaData\{UNCOMPRESSED [DailyReturn]
> optional double DailyReturn [RLE, BIT_PACKED, PLAIN], 126715},
> ColumnMetaData\{UNCOMPRESSED [DailyReturnUSD] optional double DailyReturnUSD
> [RLE, BIT_PACKED, PLAIN], 152732}, ColumnMetaData\{UNCOMPRESSED
> [NotionalReturnUSD] optional double NotionalReturnUSD [RLE, BIT_PACKED,
> PLAIN], 178749}]}]} (Error in parquet record reader.
> ...
> ...
> Hadoop path: /DEV/tbl_c_EquityDailyReturn_FXAdjusted/1_32_32.parquet
> Total records read: 0
> Row group index: 0
> Records in row group: 3243
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {
> optional int64 VectorListingId;
>
[jira] [Assigned] (DRILL-7256) Query over empty Hive tables fails, we will need to print heap usagePercent details in error message
[
https://issues.apache.org/jira/browse/DRILL-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz reassigned DRILL-7256:
-
Assignee: Khurram Faraaz
> Query over empty Hive tables fails, we will need to print heap usagePercent
> details in error message
>
>
> Key: DRILL-7256
> URL: https://issues.apache.org/jira/browse/DRILL-7256
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
>Reporter: Khurram Faraaz
>Assignee: Khurram Faraaz
>Priority: Major
>
> The below query from Drill's web UI on Hive tables failed due to not enough
> heap memory to run this query.
> It fails intermittently from Drill web UI, and note that the two Hive tables
> used in the query are empty, meaning they have no data in them. The query
> does not fail when run from sqlline.
> The error message does not provide information about the usagePercent of heap.
> It will be useful to provide heap usagePercent information as part of the
> error message in QueryWrapper.java when usagePercent >
> HEAP_MEMORY_FAILURE_THRESHOLD
> Drill 1.15.0
> Failing query.
> {noformat}
> SELECT a.event_id
> FROM hive.cust_bhsf_ce_blob a, hive.t_fct_clinical_event b
> where
> a.event_id=b.event_id
> and a.blob_contents not like '%dd:contenttype="TESTS"%'
> and b.EVENT_RELATIONSHIP_CD='B'
> and b.EVENT_CLASS_CD in ('DOC')
> and b.entry_mode_cd='Web'
> and b.RECORD_STATUS_CD='Active'
> and b.RESULT_STATUS_CD ='Auth (Verified)'
> and substring(b.valid_until_dt_tm,1,10) >='2017-12-30'
> and substring(b.event_end_date,1,10) >='2018-01-01'
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2019-05-09 16:25:58,472 [qtp1934687-790] ERROR
> o.a.d.e.server.rest.QueryResources - Query from Web UI Failed
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: There is
> not enough heap memory to run this query using the web interface.
> Please try a query with fewer columns or with a filter or limit condition to
> limit the data returned.
> You can also try an ODBC/JDBC client.
> [Error Id: 91668f42-d88e-426b-b1fe-c0d042700500 ]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.15.0.5-mapr.jar:1.15.0.5-mapr]
> at org.apache.drill.exec.server.rest.QueryWrapper.run(QueryWrapper.java:103)
> ~[drill-java-exec-1.15.0.5-mapr.jar:1.15.0.5-mapr]
> at
> org.apache.drill.exec.server.rest.QueryResources.submitQueryJSON(QueryResources.java:72)
> ~[drill-java-exec-1.15.0.5-mapr.jar:1.15.0.5-mapr]
> at
> org.apache.drill.exec.server.rest.QueryResources.submitQuery(QueryResources.java:87)
> ~[drill-java-exec-1.15.0.5-mapr.jar:1.15.0.5-mapr]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_151]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_151]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_151]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
> at
> org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103)
> [jersey-server-2.8.jar:na]
> at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269)
> [jersey-server-2.8.jar:na]
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
> [jersey-common-2.8.jar:na]
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
> [jersey-common-2.8.jar:na]
> at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
>
[jira] [Created] (DRILL-7256) Query over empty Hive tables fails, we will need to print heap usagePercent details in error message
Khurram Faraaz created DRILL-7256:
-
Summary: Query over empty Hive tables fails, we will need to print
heap usagePercent details in error message
Key: DRILL-7256
URL: https://issues.apache.org/jira/browse/DRILL-7256
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.15.0
Reporter: Khurram Faraaz
The below query from Drill's web UI on Hive tables failed due to not enough
heap memory to run this query.
It fails intermittently from Drill web UI, and note that the two Hive tables
used in the query are empty, meaning they have no data in them. The query does
not fail when run from sqlline.
The error message does not provide information about the usagePercent of heap.
It will be useful to provide heap usagePercent information as part of the error
message in QueryWrapper.java when usagePercent > HEAP_MEMORY_FAILURE_THRESHOLD
Drill 1.15.0
Failing query.
{noformat}
SELECT a.event_id
FROM hive.cust_bhsf_ce_blob a, hive.t_fct_clinical_event b
where
a.event_id=b.event_id
and a.blob_contents not like '%dd:contenttype="TESTS"%'
and b.EVENT_RELATIONSHIP_CD='B'
and b.EVENT_CLASS_CD in ('DOC')
and b.entry_mode_cd='Web'
and b.RECORD_STATUS_CD='Active'
and b.RESULT_STATUS_CD ='Auth (Verified)'
and substring(b.valid_until_dt_tm,1,10) >='2017-12-30'
and substring(b.event_end_date,1,10) >='2018-01-01'
{noformat}
Stack trace from drillbit.log
{noformat}
2019-05-09 16:25:58,472 [qtp1934687-790] ERROR
o.a.d.e.server.rest.QueryResources - Query from Web UI Failed
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: There is not
enough heap memory to run this query using the web interface.
Please try a query with fewer columns or with a filter or limit condition to
limit the data returned.
You can also try an ODBC/JDBC client.
[Error Id: 91668f42-d88e-426b-b1fe-c0d042700500 ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.15.0.5-mapr.jar:1.15.0.5-mapr]
at org.apache.drill.exec.server.rest.QueryWrapper.run(QueryWrapper.java:103)
~[drill-java-exec-1.15.0.5-mapr.jar:1.15.0.5-mapr]
at
org.apache.drill.exec.server.rest.QueryResources.submitQueryJSON(QueryResources.java:72)
~[drill-java-exec-1.15.0.5-mapr.jar:1.15.0.5-mapr]
at
org.apache.drill.exec.server.rest.QueryResources.submitQuery(QueryResources.java:87)
~[drill-java-exec-1.15.0.5-mapr.jar:1.15.0.5-mapr]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[na:1.8.0_151]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331)
[jersey-server-2.8.jar:na]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103)
[jersey-server-2.8.jar:na]
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269)
[jersey-server-2.8.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
[jersey-common-2.8.jar:na]
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
[jersey-common-2.8.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:252)
[jersey-server-2.8.jar:na]
at
[jira] [Commented] (DRILL-6562) Plugin Management improvements
[ https://issues.apache.org/jira/browse/DRILL-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822149#comment-16822149 ] Khurram Faraaz commented on DRILL-6562: --- [~vitalii] Please add unit tests, there are no unit tests here [https://github.com/apache/drill/pull/1692] We will also have to add Functional tests to the test framework to test this. > Plugin Management improvements > -- > > Key: DRILL-6562 > URL: https://issues.apache.org/jira/browse/DRILL-6562 > Project: Apache Drill > Issue Type: Improvement > Components: Client - HTTP, Web Server >Affects Versions: 1.14.0 >Reporter: Abhishek Girish >Assignee: Vitalii Diravka >Priority: Major > Labels: doc-complete, ready-to-commit > Fix For: 1.16.0 > > Attachments: Export.png, ExportAll.png, Screenshot from 2019-03-21 > 01-18-17.png, Screenshot from 2019-03-21 02-52-50.png, Storage.png, > UpdateExport.png, create.png, image-2018-07-23-02-55-02-024.png, > image-2018-10-22-20-20-24-658.png, image-2018-10-22-20-20-59-105.png > > > Follow-up to DRILL-4580. > Provide ability to export all storage plugin configurations at once, with a > new "Export All" option on the Storage page of the Drill web UI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7163) Join query fails with java.lang.IllegalArgumentException: null
Khurram Faraaz created DRILL-7163:
-
Summary: Join query fails with java.lang.IllegalArgumentException:
null
Key: DRILL-7163
URL: https://issues.apache.org/jira/browse/DRILL-7163
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.15.0
Reporter: Khurram Faraaz
Join query fails with java.lang.IllegalArgumentException: null
Drill : 1.15.0
Failing query is
{noformat}
Select *
From
(
select
convert_from(t.itm.iUUID, 'UTF8') iUUID,
convert_from(t.UPC.UPC14, 'UTF8') UPC14,
convert_from(t.itm.upcDesc, 'UTF8') upcDesc,
convert_from(t.ris.mstBrdOid, 'UTF8') mstBrdOid,
convert_from(t.ris.vrfLgyMtch, 'UTF8') vrfLgyMtch,
convert_from(t.itm.mtch.cfdMtch, 'UTF8') cfdMtch,
convert_from(t.itm.uoM, 'UTF8') uoM,
convert_from(t.uomRec.valVal, 'UTF8') uomVal,
case when a.iUUID is null then 0 else 1 end as keyind
from hbase.`/mapr/tables/item-master` t
left outer join
(
select distinct
convert_from(t.m.iUUID, 'UTF8') iUUID
from hbase.`/mapr/tables/items` t
) a
on t.itm.iUUID = a.iUUID
) i
where (i.mstBrdOid is null
or i.vrfLgyMtch is null)
and i.keyind=1
{noformat}
Stack trace from drillbit.log
{noformat}
2019-03-27 11:45:44,563 [23646564-3d23-f32b-6f68-11d7c4dd7a19:frag:1:0] ERROR
o.a.d.e.physical.impl.BaseRootExec - Batch dump started: dumping last 2 failed
batches
2019-03-27 11:45:44,564 [23646564-3d23-f32b-6f68-11d7c4dd7a19:frag:1:0] ERROR
o.a.d.e.p.i.p.ProjectRecordBatch -
ProjectRecordBatch[projector=Projector[vector2=null, selectionVectorMode=NONE],
hasRemainder=false, remainderIndex=0, recordCount=0,
container=org.apache.drill.exec.record.VectorContainer@2133fd0e[recordCount =
0, schemaChanged = false, schema = BatchSchema [fields=[[`row_key`
(VARBINARY:REQUIRED)], [`clnDesc` (MAP:REQUIRED), children=([`bndlCnt`
(VARBINARY:OPTIONAL)], [`by` (VARBINARY:OPTIONAL)], [`desc`
(VARBINARY:OPTIONAL)], [`dt` (VARBINARY:OPTIONAL)], [`descExt`
(VARBINARY:OPTIONAL)])], [`dup` (MAP:REQUIRED), children=([`dupBy`
(VARBINARY:OPTIONAL)], [`dupDt` (VARBINARY:OPTIONAL)], [`duplicate`
(VARBINARY:OPTIONAL)], [`preferred` (VARBINARY:OPTIONAL)])], [`itm`
(MAP:REQUIRED), children=([`iUUID` (VARBINARY:OPTIONAL)], [`cfdLgyMtch`
(VARBINARY:OPTIONAL)], [`uoM` (VARBINARY:OPTIONAL)], [`upcCd`
(VARBINARY:OPTIONAL)], [`upcDesc` (VARBINARY:OPTIONAL)], [`promo`
(VARBINARY:OPTIONAL)])], [`lckSts` (MAP:REQUIRED), children=([`lckBy`
(VARBINARY:OPTIONAL)], [`lckDt` (VARBINARY:OPTIONAL)])], [`lgy` (MAP:REQUIRED),
children=([`lgyBr` (VARBINARY:OPTIONAL)])], [`obs` (MAP:REQUIRED),
children=([`POSFile` (VARBINARY:OPTIONAL)])], [`prmRec` (MAP:REQUIRED)], [`ris`
(MAP:REQUIRED), children=([`UPC` (VARBINARY:OPTIONAL)], [`brdDesc`
(VARBINARY:OPTIONAL)], [`brdExtDesc` (VARBINARY:OPTIONAL)], [`brdFamDesc`
(VARBINARY:OPTIONAL)], [`brdTypeCd` (VARBINARY:OPTIONAL)], [`flvDesc`
(VARBINARY:OPTIONAL)], [`mfgDesc` (VARBINARY:OPTIONAL)], [`modBy`
(VARBINARY:OPTIONAL)], [`modDt` (VARBINARY:OPTIONAL)], [`msaCatCd`
(VARBINARY:OPTIONAL)])], [`rjr` (MAP:REQUIRED)], [`uomRec` (MAP:REQUIRED),
children=([`valBy` (VARBINARY:OPTIONAL)], [`valDt` (VARBINARY:OPTIONAL)],
[`valVal` (VARBINARY:OPTIONAL)], [`recBy` (VARBINARY:OPTIONAL)], [`recDt`
(VARBINARY:OPTIONAL)], [`recRat` (VARBINARY:OPTIONAL)], [`recVal`
(VARBINARY:OPTIONAL)])], [`upc` (MAP:REQUIRED), children=([`UPC14`
(VARBINARY:OPTIONAL)], [`allUPCVar` (VARBINARY:OPTIONAL)])], [`$f12`
(VARBINARY:OPTIONAL)], [`iUUID` (VARCHAR:OPTIONAL)]], selectionVector=NONE],
wrappers = [org.apache.drill.exec.vector.VarBinaryVector@b23a384[field =
[`row_key` (VARBINARY:REQUIRED)], ...],
org.apache.drill.exec.vector.complex.MapVector@61c779ff,
org.apache.drill.exec.vector.complex.MapVector@575c0f96,
org.apache.drill.exec.vector.complex.MapVector@69b943fe,
org.apache.drill.exec.vector.complex.MapVector@7f90e2ce,
org.apache.drill.exec.vector.complex.MapVector@25c27442,
org.apache.drill.exec.vector.complex.MapVector@12d5ffd3,
org.apache.drill.exec.vector.complex.MapVector@3150f8c4,
org.apache.drill.exec.vector.complex.MapVector@49aefab2,
org.apache.drill.exec.vector.complex.MapVector@7f78e7a1,
org.apache.drill.exec.vector.complex.MapVector@426ea4fa,
org.apache.drill.exec.vector.complex.MapVector@74cee2ab,
org.apache.drill.exec.vector.NullableVarBinaryVector@4a0bfdea[field = [`$f12`
(VARBINARY:OPTIONAL)], ...],
org.apache.drill.exec.vector.NullableVarCharVector@72f64ee5[field = [`iUUID`
(VARCHAR:OPTIONAL)], ...]], ...]]
2019-03-27 11:45:44,565 [23646564-3d23-f32b-6f68-11d7c4dd7a19:frag:1:0] ERROR
o.a.d.e.p.impl.join.HashJoinBatch -
HashJoinBatch[container=org.apache.drill.exec.record.VectorContainer@45887d35[recordCount
= 0, schemaChanged = false, schema = BatchSchema [fields=[[`row_key`
(VARBINARY:REQUIRED)], [`clnDesc` (MAP:REQUIRED), children=([`bndlCnt`
(VARBINARY:OPTIONAL)], [`by` (VARBINARY:OPTIONAL)], [`desc`
(VARBINARY:OPTIONAL)],
[jira] [Created] (DRILL-7144) sqlline option : !set useLineContinuation false, fails with ParseException
Khurram Faraaz created DRILL-7144:
-
Summary: sqlline option : !set useLineContinuation false, fails
with ParseException
Key: DRILL-7144
URL: https://issues.apache.org/jira/browse/DRILL-7144
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.15.0, 1.13.0
Reporter: Khurram Faraaz
Assignee: Arina Ielchiieva
sqlline option does not work as intended. Returns ParseException instead.
!set useLineContinuation false
On mapr-drill-1.13.0 we hit the below Exception.
{noformat}
0: jdbc:drill:drillbit=drill-abcd-dev.dev.schw> !set useLineContinuation false
Error setting configuration: useLineContinuation:
java.lang.IllegalArgumentException: No method matching "setuseLineContinuation"
was found in sqlline.SqlLineOpts.
{noformat}
It does not work on drill-1.15.0-mapr-r1
git.branch=drill-1.15.0-mapr-r1
git.commit.id=ebc9fe49d4477b04701fdd81884d5a0b748a13ae
{noformat}
[test@test-ab bin]# ./sqlline -u
"jdbc:drill:schema=dfs.tmp;auth=MAPRSASL;drillbit=test-ab.qa.lab" -n mapr -p
mapr
Apache Drill 1.15.0.3-mapr
"Start your SQL engine."
0: jdbc:drill:schema=dfs.tmp> !set useLineContinuation false
0: jdbc:drill:schema=dfs.tmp> select * from sys.version
> select * from sys.memory
Error: PARSE ERROR: Encountered "select" at line 2, column 1.
Was expecting one of:
"ORDER" ...
"LIMIT" ...
"OFFSET" ...
"FETCH" ...
"NATURAL" ...
"JOIN" ...
"INNER" ...
"LEFT" ...
"RIGHT" ...
"FULL" ...
"CROSS" ...
"," ...
"OUTER" ...
"EXTEND" ...
"(" ...
"MATCH_RECOGNIZE" ...
"AS" ...
...
...
...
...
...
"TABLESAMPLE" ...
"WHERE" ...
"GROUP" ...
"HAVING" ...
"WINDOW" ...
"UNION" ...
"INTERSECT" ...
"EXCEPT" ...
"MINUS" ...
"." ...
"[" ...
SQL Query select * from sys.version
select * from sys.memory
^
[Error Id: 067d5402-b965-4660-8981-34491ab5a051 on test-ab.qa.lab:31010]
(state=,code=0)
{noformat}
{noformat}
[Error Id: 067d5402-b965-4660-8981-34491ab5a051 ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at org.apache.drill.exec.planner.sql.SqlConverter.parse(SqlConverter.java:185)
[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:138)
[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:110)
[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:76)
[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:584)
[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:272)
[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_151]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
Caused by: org.apache.calcite.sql.parser.SqlParseException: Encountered
"select" at line 2, column 1.
Was expecting one of:
"ORDER" ...
"LIMIT" ...
"OFFSET" ...
"FETCH" ...
...
"[" ...
at
org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.convertException(DrillParserImpl.java:350)
~[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at
org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.normalizeException(DrillParserImpl.java:131)
~[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:137)
~[calcite-core-1.17.0-drill-r2.jar:1.17.0-drill-r2]
at org.apache.calcite.sql.parser.SqlParser.parseStmt(SqlParser.java:162)
~[calcite-core-1.17.0-drill-r2.jar:1.17.0-drill-r2]
at org.apache.drill.exec.planner.sql.SqlConverter.parse(SqlConverter.java:177)
[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
... 8 common frames omitted
Caused by: org.apache.drill.exec.planner.sql.parser.impl.ParseException:
Encountered "select" at line 2, column 1.
Was expecting one of:
"ORDER" ...
"LIMIT" ...
"OFFSET" ...
"FETCH" ...
"NATURAL" ...
...
...
"[" ...
at
org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.generateParseException(DrillParserImpl.java:24076)
~[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at
org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.jj_consume_token(DrillParserImpl.java:23893)
~[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at
org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.SqlStmtEof(DrillParserImpl.java:899)
~[drill-java-exec-1.15.0.3-mapr.jar:1.15.0.3-mapr]
at
[jira] [Commented] (DRILL-7138) Implement command to describe schema for table
[
https://issues.apache.org/jira/browse/DRILL-7138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804197#comment-16804197
]
Khurram Faraaz commented on DRILL-7138:
---
[~arina] will this also be supported for views ?
> Implement command to describe schema for table
> --
>
> Key: DRILL-7138
> URL: https://issues.apache.org/jira/browse/DRILL-7138
> Project: Apache Drill
> Issue Type: Sub-task
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.16.0
>
>
> As part of schema provisioning project, it will be handy for the user to see
> if table has schema and its content.
> Command syntax:
> *describe schema for table dfs.tmp.`text_table`*
> By default schema will be output in JSON format (format schema is stored in
> .drill.schema file):
> {noformat}
> {
> "table" : "dfs.tmp.`text_table`",
> "schema" : {
> "columns" : [
> {
> "name" : "id",
> "type" : "INT",
> "mode" : "OPTIONAL"
> }
> ]
> },
> "version" : 1
> }
> {noformat}
> JSON format can be indicated explicitly:
> *describe schema for table dfs.tmp.`text_table` as json*
> Other formats:
> STATEMENT - schema will be output in the form of CREATE SCHEMA statement.
> *describe schema for table dfs.tmp.`text_table` as statement*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-7128) IllegalStateException: Read batch count [0] should be greater than zero
Khurram Faraaz created DRILL-7128:
-
Summary: IllegalStateException: Read batch count [0] should be
greater than zero
Key: DRILL-7128
URL: https://issues.apache.org/jira/browse/DRILL-7128
Project: Apache Drill
Issue Type: Bug
Components: Storage - Parquet
Affects Versions: 1.15.0
Reporter: Khurram Faraaz
Source table is a Hive table stored as parquet.
Issue is seen only when querying datacapturekey column, which is of VARCHAR
type.
Hive 2.3
MapR Drill : 1.15.0.0-mapr
commit id : 951ef156fb1025677a2ca2dcf84e11002bf4b513
{noformat}
0: jdbc:drill:drillbit=test.a.node1> describe bt_br_cc_invalid_leads ;
+-++--+
| COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
+-++--+
| wrapup | CHARACTER VARYING | YES |
| datacapturekey | CHARACTER VARYING | YES |
| leadgendate | CHARACTER VARYING | YES |
| crla1 | CHARACTER VARYING | YES |
| crla2 | CHARACTER VARYING | YES |
| invalid_lead | INTEGER | YES |
| destination_advertiser_vendor_name | CHARACTER VARYING | YES |
| source_program_key | CHARACTER VARYING | YES |
| publisher_publisher | CHARACTER VARYING | YES |
| areaname | CHARACTER VARYING | YES |
| data_abertura_ficha | CHARACTER VARYING | YES |
+-++--+
11 rows selected (1.85 seconds)
0: jdbc:drill:drillbit=test.a.node1>
// from the view definition, note that column datacapturekey is of type
VARVCHAR with precision 2000
{
"name" : "bt_br_cc_invalid_leads",
"sql" : "SELECT CAST(`wrapup` AS VARCHAR(2000)) AS `wrapup`,
CAST(`datacapturekey` AS VARCHAR(2000)) AS `datacapturekey`, CAST(`leadgendate`
AS VARCHAR(2000)) AS `leadgendate`, CAST(`crla1` AS VARCHAR(2000)) AS `crla1`,
CAST(`crla2` AS VARCHAR(2000)) AS `crla2`, CAST(`invalid_lead` AS INTEGER) AS
`invalid_lead`, CAST(`destination_advertiser_vendor_name` AS VARCHAR(2000)) AS
`destination_advertiser_vendor_name`, CAST(`source_program_key` AS
VARCHAR(2000)) AS `source_program_key`, CAST(`publisher_publisher` AS
VARCHAR(2000)) AS `publisher_publisher`, CAST(`areaname` AS VARCHAR(2000)) AS
`areaname`, CAST(`data_abertura_ficha` AS VARCHAR(2000)) AS
`data_abertura_ficha`\nFROM
`dfs`.`root`.`/user/bigtable/logs/hive/warehouse/bt_br_cc_invalid_leads`",
"fields" : [ {
"name" : "wrapup",
"type" : "VARCHAR",
"precision" : 2000,
"isNullable" : true
}, {
"name" : "datacapturekey",
"type" : "VARCHAR",
"precision" : 2000,
"isNullable" : true
...
...
// total number of rows in bt_br_cc_invalid_leads
0: jdbc:drill:drillbit=test.a.node1> select count(*) from
bt_br_cc_invalid_leads ;
+-+
| EXPR$0 |
+-+
| 20599 |
+-+
1 row selected (0.173 seconds)
{noformat}
Stack trace from drillbit.log
{noformat}
2019-03-18 12:19:01,610 [237010da-6eda-a913-0424-32f63fbe01be:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query with id
237010da-6eda-a913-0424-32f63fbe01be issued by bigtable: SELECT
`bt_br_cc_invalid_leads`.`datacapturekey` AS `datacapturekey`
FROM `dfs.drill_views`.`bt_br_cc_invalid_leads` `bt_br_cc_invalid_leads`
GROUP BY `bt_br_cc_invalid_leads`.`datacapturekey`
2019-03-18 12:19:02,495 [237010da-6eda-a913-0424-32f63fbe01be:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 237010da-6eda-a913-0424-32f63fbe01be:0:0:
State change requested AWAITING_ALLOCATION --> RUNNING
2019-03-18 12:19:02,495 [237010da-6eda-a913-0424-32f63fbe01be:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 237010da-6eda-a913-0424-32f63fbe01be:0:0:
State to report: RUNNING
2019-03-18 12:19:02,502 [237010da-6eda-a913-0424-32f63fbe01be:frag:0:0] INFO
o.a.d.exec.physical.impl.ScanBatch - User Error Occurred: Error in parquet
record reader.
Message:
Hadoop path: /user/bigtable/logs/hive/warehouse/bt_br_cc_invalid_leads/08_0
Total records read: 0
Row group index: 0
Records in row group: 1551
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message hive_schema {
optional binary wrapup (UTF8);
optional binary datacapturekey (UTF8);
optional binary leadgendate (UTF8);
optional binary crla1 (UTF8);
optional binary crla2 (UTF8);
optional binary invalid_lead (UTF8);
optional binary destination_advertiser_vendor_name (UTF8);
optional binary source_program_key (UTF8);
optional binary publisher_publisher (UTF8);
optional binary areaname (UTF8);
optional binary data_abertura_ficha (UTF8);
}
, metadata: {}}, blocks: [BlockMetaData\{1551, 139906
[ColumnMetaData{UNCOMPRESSED [wrapup] optional binary wrapup (UTF8)
[PLAIN_DICTIONARY, RLE, BIT_PACKED], 4}, ColumnMetaData\{UNCOMPRESSED
[datacapturekey] optional binary datacapturekey (UTF8) [RLE, PLAIN,
BIT_PACKED], 656}, ColumnMetaData\{UNCOMPRESSED [leadgendate] optional binary
leadgendate (UTF8) [PLAIN_DICTIONARY, RLE, BIT_PACKED], 23978},
[jira] [Assigned] (DRILL-7128) IllegalStateException: Read batch count [0] should be greater than zero
[
https://issues.apache.org/jira/browse/DRILL-7128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz reassigned DRILL-7128:
-
Assignee: salim achouche
> IllegalStateException: Read batch count [0] should be greater than zero
> ---
>
> Key: DRILL-7128
> URL: https://issues.apache.org/jira/browse/DRILL-7128
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.15.0
>Reporter: Khurram Faraaz
>Assignee: salim achouche
>Priority: Major
>
> Source table is a Hive table stored as parquet.
> Issue is seen only when querying datacapturekey column, which is of VARCHAR
> type.
> Hive 2.3
> MapR Drill : 1.15.0.0-mapr
> commit id : 951ef156fb1025677a2ca2dcf84e11002bf4b513
> {noformat}
> 0: jdbc:drill:drillbit=test.a.node1> describe bt_br_cc_invalid_leads ;
> +-++--+
> | COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
> +-++--+
> | wrapup | CHARACTER VARYING | YES |
> | datacapturekey | CHARACTER VARYING | YES |
> | leadgendate | CHARACTER VARYING | YES |
> | crla1 | CHARACTER VARYING | YES |
> | crla2 | CHARACTER VARYING | YES |
> | invalid_lead | INTEGER | YES |
> | destination_advertiser_vendor_name | CHARACTER VARYING | YES |
> | source_program_key | CHARACTER VARYING | YES |
> | publisher_publisher | CHARACTER VARYING | YES |
> | areaname | CHARACTER VARYING | YES |
> | data_abertura_ficha | CHARACTER VARYING | YES |
> +-++--+
> 11 rows selected (1.85 seconds)
> 0: jdbc:drill:drillbit=test.a.node1>
> // from the view definition, note that column datacapturekey is of type
> VARVCHAR with precision 2000
> {
> "name" : "bt_br_cc_invalid_leads",
> "sql" : "SELECT CAST(`wrapup` AS VARCHAR(2000)) AS `wrapup`,
> CAST(`datacapturekey` AS VARCHAR(2000)) AS `datacapturekey`,
> CAST(`leadgendate` AS VARCHAR(2000)) AS `leadgendate`, CAST(`crla1` AS
> VARCHAR(2000)) AS `crla1`, CAST(`crla2` AS VARCHAR(2000)) AS `crla2`,
> CAST(`invalid_lead` AS INTEGER) AS `invalid_lead`,
> CAST(`destination_advertiser_vendor_name` AS VARCHAR(2000)) AS
> `destination_advertiser_vendor_name`, CAST(`source_program_key` AS
> VARCHAR(2000)) AS `source_program_key`, CAST(`publisher_publisher` AS
> VARCHAR(2000)) AS `publisher_publisher`, CAST(`areaname` AS VARCHAR(2000)) AS
> `areaname`, CAST(`data_abertura_ficha` AS VARCHAR(2000)) AS
> `data_abertura_ficha`\nFROM
> `dfs`.`root`.`/user/bigtable/logs/hive/warehouse/bt_br_cc_invalid_leads`",
> "fields" : [ {
> "name" : "wrapup",
> "type" : "VARCHAR",
> "precision" : 2000,
> "isNullable" : true
> }, {
> "name" : "datacapturekey",
> "type" : "VARCHAR",
> "precision" : 2000,
> "isNullable" : true
> ...
> ...
> // total number of rows in bt_br_cc_invalid_leads
> 0: jdbc:drill:drillbit=test.a.node1> select count(*) from
> bt_br_cc_invalid_leads ;
> +-+
> | EXPR$0 |
> +-+
> | 20599 |
> +-+
> 1 row selected (0.173 seconds)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2019-03-18 12:19:01,610 [237010da-6eda-a913-0424-32f63fbe01be:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 237010da-6eda-a913-0424-32f63fbe01be issued by bigtable: SELECT
> `bt_br_cc_invalid_leads`.`datacapturekey` AS `datacapturekey`
> FROM `dfs.drill_views`.`bt_br_cc_invalid_leads` `bt_br_cc_invalid_leads`
> GROUP BY `bt_br_cc_invalid_leads`.`datacapturekey`
> 2019-03-18 12:19:02,495 [237010da-6eda-a913-0424-32f63fbe01be:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 237010da-6eda-a913-0424-32f63fbe01be:0:0: State change requested
> AWAITING_ALLOCATION --> RUNNING
> 2019-03-18 12:19:02,495 [237010da-6eda-a913-0424-32f63fbe01be:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 237010da-6eda-a913-0424-32f63fbe01be:0:0: State to report: RUNNING
> 2019-03-18 12:19:02,502 [237010da-6eda-a913-0424-32f63fbe01be:frag:0:0] INFO
> o.a.d.exec.physical.impl.ScanBatch - User Error Occurred: Error in parquet
> record reader.
> Message:
> Hadoop path:
> /user/bigtable/logs/hive/warehouse/bt_br_cc_invalid_leads/08_0
> Total records read: 0
> Row group index: 0
> Records in row group: 1551
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message hive_schema {
> optional binary wrapup (UTF8);
> optional binary datacapturekey (UTF8);
> optional binary leadgendate (UTF8);
> optional binary crla1 (UTF8);
> optional binary crla2 (UTF8);
> optional binary invalid_lead (UTF8);
> optional binary destination_advertiser_vendor_name (UTF8);
> optional binary source_program_key (UTF8);
> optional binary publisher_publisher (UTF8);
> optional binary
[jira] [Created] (DRILL-7100) parquet RecordBatchSizerManager : IllegalArgumentException: the requested size must be non-negative
Khurram Faraaz created DRILL-7100:
-
Summary: parquet RecordBatchSizerManager :
IllegalArgumentException: the requested size must be non-negative
Key: DRILL-7100
URL: https://issues.apache.org/jira/browse/DRILL-7100
Project: Apache Drill
Issue Type: Bug
Components: Storage - Parquet
Affects Versions: 1.15.0
Reporter: Khurram Faraaz
Table has string columns that can range from 1024 bytes to 32MB in length, we
should be able to handle such wide string columns in parquet, when querying
from Drill.
Hive Version 2.3.3
Drill Version 1.15
{noformat}
CREATE TABLE temp.cust_bhsf_ce_blob_parquet (
event_id DECIMAL,
valid_until_dt_tm string,
blob_seq_num DECIMAL,
valid_from_dt_tm string,
blob_length DECIMAL,
compression_cd DECIMAL,
blob_contents string,
updt_dt_tm string,
updt_id DECIMAL,
updt_task DECIMAL,
updt_cnt DECIMAL,
updt_applctx DECIMAL,
last_utc_ts string,
ccl_load_dt_tm string,
ccl_updt_dt_tm string )
STORED AS PARQUET;
{noformat}
The source table is stored as ORC format.
Failing query.
{noformat}
SELECT event_id, BLOB_CONTENTS FROM hive.temp.cust_bhsf_ce_blob_parquet WHERE
event_id = 3443236037
2019-03-07 14:40:17,886 [237e8c79-0e9b-45d6-9134-0da95dba462f:frag:1:269] INFO
o.a.d.exec.physical.impl.ScanBatch - User Error Occurred: the requested size
must be non-negative (the requested size must be non-negative)
org.apache.drill.common.exceptions.UserException: INTERNAL_ERROR ERROR: the
requested size must be non-negative
{noformat}
Snippet from drillbit.log file
{noformat}
[Error Id: 41a4d597-f54d-42a6-be6d-5dbeb7f642ba ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:293)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:126)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:116)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:126)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:116)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:69)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:297)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:284)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_181]
at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_181]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
[hadoop-common-2.7.0-mapr-1808.jar:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:284)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: java.lang.IllegalArgumentException: the requested size must be
non-negative
at
org.apache.drill.shaded.guava.com.google.common.base.Preconditions.checkArgument(Preconditions.java:135)
~[drill-shaded-guava-23.0.jar:23.0]
at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:224)
[jira] [Assigned] (DRILL-7100) parquet RecordBatchSizerManager : IllegalArgumentException: the requested size must be non-negative
[
https://issues.apache.org/jira/browse/DRILL-7100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz reassigned DRILL-7100:
-
Assignee: salim achouche
> parquet RecordBatchSizerManager : IllegalArgumentException: the requested
> size must be non-negative
> ---
>
> Key: DRILL-7100
> URL: https://issues.apache.org/jira/browse/DRILL-7100
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.15.0
>Reporter: Khurram Faraaz
>Assignee: salim achouche
>Priority: Major
>
> Table has string columns that can range from 1024 bytes to 32MB in length, we
> should be able to handle such wide string columns in parquet, when querying
> from Drill.
> Hive Version 2.3.3
> Drill Version 1.15
> {noformat}
> CREATE TABLE temp.cust_bhsf_ce_blob_parquet (
> event_id DECIMAL,
> valid_until_dt_tm string,
> blob_seq_num DECIMAL,
> valid_from_dt_tm string,
> blob_length DECIMAL,
> compression_cd DECIMAL,
> blob_contents string,
> updt_dt_tm string,
> updt_id DECIMAL,
> updt_task DECIMAL,
> updt_cnt DECIMAL,
> updt_applctx DECIMAL,
> last_utc_ts string,
> ccl_load_dt_tm string,
> ccl_updt_dt_tm string )
> STORED AS PARQUET;
> {noformat}
>
> The source table is stored as ORC format.
> Failing query.
> {noformat}
> SELECT event_id, BLOB_CONTENTS FROM hive.temp.cust_bhsf_ce_blob_parquet WHERE
> event_id = 3443236037
> 2019-03-07 14:40:17,886 [237e8c79-0e9b-45d6-9134-0da95dba462f:frag:1:269]
> INFO o.a.d.exec.physical.impl.ScanBatch - User Error Occurred: the requested
> size must be non-negative (the requested size must be non-negative)
> org.apache.drill.common.exceptions.UserException: INTERNAL_ERROR ERROR: the
> requested size must be non-negative
> {noformat}
> Snippet from drillbit.log file
> {noformat}
> [Error Id: 41a4d597-f54d-42a6-be6d-5dbeb7f642ba ]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:293)
> [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:126)
> [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:116)
> [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
> [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
> [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:126)
> [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:116)
> [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:69)
> [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
> [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
> [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93)
> [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
> [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:297)
> [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:284)
> [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_181]
> at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_181]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
> [hadoop-common-2.7.0-mapr-1808.jar:na]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:284)
> [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.15.0.0-mapr.jar:1.15.0.0-mapr]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
[jira] [Commented] (DRILL-7061) Selecting option to limit results to 1000 on web UI causes parse error
[
https://issues.apache.org/jira/browse/DRILL-7061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779921#comment-16779921
]
Khurram Faraaz commented on DRILL-7061:
---
[~kkhatua] this is not a duplicate, the two error messages are different. And
in DRILL-6960 the user supplies a different LIMIT value (which is 10),
different from default "1,000"
Whereas, I hit the ParseException when we retained the default value of "1,000"
for the LIMIT. The fix is to remove the comma from the default value provided
in the textbox for queryLimit, like you have mentioned.
> Selecting option to limit results to 1000 on web UI causes parse error
> --
>
> Key: DRILL-7061
> URL: https://issues.apache.org/jira/browse/DRILL-7061
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.16.0
>Reporter: Khurram Faraaz
>Assignee: Kunal Khatua
>Priority: Critical
> Fix For: 1.16.0
>
> Attachments: image-2019-02-27-14-17-24-348.png
>
>
> Selecting option to Limit results to 1,000 causes a parse error on web UI,
> screen shot is attached. Browser used was Chrome.
> Drill version => 1.16.0-SNAPSHOT
> commit = e342ff5
> Error reported on web UI when we press Submit button on web UI
> {noformat}
> Query Failed: An Error Occurred
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR: 'LIMIT
> start, count' is not allowed under the current SQL conformance level SQL
> Query -- [autoLimit: 1,000 rows] select * from ( select length(varStr) from
> dfs.`/root/many_json_files` ) limit 1,000 [Error Id:
> e252d1cc-54d4-4530-837c-a1726a5be89f on qa102-45.qa.lab:31010]{noformat}
> Stack trace from drillbit.log
> {noformat}
> 2019-02-27 21:59:18,428 [2388f7c9-2cb4-0ef8-4088-3ffcab1f0ed2:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query with id
> 2388f7c9-2cb4-0ef8-4088-3ffcab1f0ed2 issued by anonymous: -- [autoLimit:
> 1,000 rows]
> select * from (
> select length(varStr) from dfs.`/root/many_json_files`
> ) limit 1,000
> 2019-02-27 21:59:18,438 [2388f7c9-2cb4-0ef8-4088-3ffcab1f0ed2:foreman] INFO
> o.a.d.exec.planner.sql.SqlConverter - User Error Occurred: 'LIMIT start,
> count' is not allowed under the current SQL conformance level ('LIMIT start,
> count' is not allowed under the current SQL conformance level)
> org.apache.drill.common.exceptions.UserException: PARSE ERROR: 'LIMIT start,
> count' is not allowed under the current SQL conformance level
> SQL Query -- [autoLimit: 1,000 rows]
> select * from (
> select length(varStr) from dfs.`/root/many_json_files`
> ) limit 1,000
> [Error Id: 286b7236-bafd-4ddc-ab10-aaac07e5c088 ]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.SqlConverter.parse(SqlConverter.java:193)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:138)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:110)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:76)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:584)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:272)
> [drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [na:1.8.0_191]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_191]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
> Caused by: org.apache.calcite.sql.parser.SqlParseException: 'LIMIT start,
> count' is not allowed under the current SQL conformance level
> at
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.convertException(DrillParserImpl.java:357)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.normalizeException(DrillParserImpl.java:145)
> ~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
> at org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:156)
> ~[calcite-core-1.18.0-drill-r0.jar:1.18.0-drill-r0]
> at org.apache.calcite.sql.parser.SqlParser.parseStmt(SqlParser.java:181)
> ~[calcite-core-1.18.0-drill-r0.jar:1.18.0-drill-r0]
> at
> org.apache.drill.exec.planner.sql.SqlConverter.parse(SqlConverter.java:185)
>
[jira] [Assigned] (DRILL-7061) Selecting option to limit results to 1000 on web UI causes parse error
[
https://issues.apache.org/jira/browse/DRILL-7061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz reassigned DRILL-7061:
-
Assignee: Kunal Khatua
> Selecting option to limit results to 1000 on web UI causes parse error
> --
>
> Key: DRILL-7061
> URL: https://issues.apache.org/jira/browse/DRILL-7061
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.16.0
>Reporter: Khurram Faraaz
>Assignee: Kunal Khatua
>Priority: Critical
> Attachments: image-2019-02-27-14-17-24-348.png
>
>
> Selecting option to Limit results to 1,000 causes a parse error on web UI,
> screen shot is attached. Browser used was Chrome.
> Drill version => 1.16.0-SNAPSHOT
> commit = e342ff5
> Error reported on web UI when we press Submit button on web UI
> {noformat}
> Query Failed: An Error Occurred
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR: 'LIMIT
> start, count' is not allowed under the current SQL conformance level SQL
> Query -- [autoLimit: 1,000 rows] select * from ( select length(varStr) from
> dfs.`/root/many_json_files` ) limit 1,000 [Error Id:
> e252d1cc-54d4-4530-837c-a1726a5be89f on qa102-45.qa.lab:31010]{noformat}
>
> !image-2019-02-27-14-17-24-348.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-7061) Selecting option to limit results to 1000 on web UI causes parse error
[
https://issues.apache.org/jira/browse/DRILL-7061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-7061:
--
Description:
Selecting option to Limit results to 1,000 causes a parse error on web UI,
screen shot is attached. Browser used was Chrome.
Drill version => 1.16.0-SNAPSHOT
commit = e342ff5
Error reported on web UI when we press Submit button on web UI
{noformat}
Query Failed: An Error Occurred
org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR: 'LIMIT
start, count' is not allowed under the current SQL conformance level SQL Query
-- [autoLimit: 1,000 rows] select * from ( select length(varStr) from
dfs.`/root/many_json_files` ) limit 1,000 [Error Id:
e252d1cc-54d4-4530-837c-a1726a5be89f on qa102-45.qa.lab:31010]{noformat}
Stack trace from drillbit.log
{noformat}
2019-02-27 21:59:18,428 [2388f7c9-2cb4-0ef8-4088-3ffcab1f0ed2:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query with id
2388f7c9-2cb4-0ef8-4088-3ffcab1f0ed2 issued by anonymous: -- [autoLimit: 1,000
rows]
select * from (
select length(varStr) from dfs.`/root/many_json_files`
) limit 1,000
2019-02-27 21:59:18,438 [2388f7c9-2cb4-0ef8-4088-3ffcab1f0ed2:foreman] INFO
o.a.d.exec.planner.sql.SqlConverter - User Error Occurred: 'LIMIT start, count'
is not allowed under the current SQL conformance level ('LIMIT start, count' is
not allowed under the current SQL conformance level)
org.apache.drill.common.exceptions.UserException: PARSE ERROR: 'LIMIT start,
count' is not allowed under the current SQL conformance level
SQL Query -- [autoLimit: 1,000 rows]
select * from (
select length(varStr) from dfs.`/root/many_json_files`
) limit 1,000
[Error Id: 286b7236-bafd-4ddc-ab10-aaac07e5c088 ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at org.apache.drill.exec.planner.sql.SqlConverter.parse(SqlConverter.java:193)
[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:138)
[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:110)
[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:76)
[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:584)
[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:272)
[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_191]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
Caused by: org.apache.calcite.sql.parser.SqlParseException: 'LIMIT start,
count' is not allowed under the current SQL conformance level
at
org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.convertException(DrillParserImpl.java:357)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.normalizeException(DrillParserImpl.java:145)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:156)
~[calcite-core-1.18.0-drill-r0.jar:1.18.0-drill-r0]
at org.apache.calcite.sql.parser.SqlParser.parseStmt(SqlParser.java:181)
~[calcite-core-1.18.0-drill-r0.jar:1.18.0-drill-r0]
at org.apache.drill.exec.planner.sql.SqlConverter.parse(SqlConverter.java:185)
[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
... 8 common frames omitted
Caused by: org.apache.drill.exec.planner.sql.parser.impl.ParseException: 'LIMIT
start, count' is not allowed under the current SQL conformance level
at
org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.OrderedQueryOrExpr(DrillParserImpl.java:489)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.SqlStmt(DrillParserImpl.java:873)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.SqlStmtEof(DrillParserImpl.java:901)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.parser.impl.DrillParserImpl.parseSqlStmtEof(DrillParserImpl.java:201)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.parser.impl.DrillParserWithCompoundIdConverter.parseSqlStmtEof(DrillParserWithCompoundIdConverter.java:59)
~[drill-java-exec-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
at org.apache.calcite.sql.parser.SqlParser.parseQuery(SqlParser.java:148)
[jira] [Created] (DRILL-7061) Selecting option to limit results to 1000 on web UI causes parse error
Khurram Faraaz created DRILL-7061:
-
Summary: Selecting option to limit results to 1000 on web UI
causes parse error
Key: DRILL-7061
URL: https://issues.apache.org/jira/browse/DRILL-7061
Project: Apache Drill
Issue Type: Bug
Components: Web Server
Affects Versions: 1.16.0
Reporter: Khurram Faraaz
Attachments: image-2019-02-27-14-17-24-348.png
Selecting option to Limit results to 1,000 causes a parse error on web UI,
screen shot is attached. Browser used was Chrome.
Drill version => 1.16.0-SNAPSHOT
commit = e342ff5
Error reported on web UI when we press Submit button on web UI
{noformat}
Query Failed: An Error Occurred
org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR: 'LIMIT
start, count' is not allowed under the current SQL conformance level SQL Query
-- [autoLimit: 1,000 rows] select * from ( select length(varStr) from
dfs.`/root/many_json_files` ) limit 1,000 [Error Id:
e252d1cc-54d4-4530-837c-a1726a5be89f on qa102-45.qa.lab:31010]{noformat}
!image-2019-02-27-14-17-24-348.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-7023) Query fails with IndexOutOfBoundsException after upgrade from drill 1.13.0 to drill 1.14.0
Khurram Faraaz created DRILL-7023:
-
Summary: Query fails with IndexOutOfBoundsException after upgrade
from drill 1.13.0 to drill 1.14.0
Key: DRILL-7023
URL: https://issues.apache.org/jira/browse/DRILL-7023
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Query fails with IndexOutOfBoundsException after upgrade from drill 1.13.0 to
drill 1.14.0
{noformat}
2018-12-06 21:43:00,538 [23f5f79c-3777-eb37-ee46-f73be74381ef:frag:2:1] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IndexOutOfBoundsException
Fragment 2:1
[Error Id: 3b653503-b6da-4853-a395-317a169468ce on am1397.test.net:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IndexOutOfBoundsException
Fragment 2:1
[Error Id: 3b653503-b6da-4853-a395-317a169468ce on am1397.test.net:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361)
[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216)
[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327)
[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.14.0-mapr.jar:1.14.0-mapr]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_152]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
Caused by: java.lang.IndexOutOfBoundsException: null
at io.netty.buffer.DrillBuf.getBytes(DrillBuf.java:677)
~[drill-memory-base-1.14.0-mapr.jar:4.0.48.Final]
at org.apache.drill.exec.vector.BigIntVector.copyEntry(BigIntVector.java:389)
~[vector-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.test.generated.HashJoinProbeGen480.appendProbe(HashJoinProbeTemplate.java:190)
~[na:na]
at
org.apache.drill.exec.test.generated.HashJoinProbeGen480.outputOuterRow(HashJoinProbeTemplate.java:223)
~[na:na]
at
org.apache.drill.exec.test.generated.HashJoinProbeGen480.executeProbePhase(HashJoinProbeTemplate.java:357)
~[na:na]
at
org.apache.drill.exec.test.generated.HashJoinProbeGen480.probeAndProject(HashJoinProbeTemplate.java:400)
~[na:na]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.innerNext(HashJoinBatch.java:465)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:69)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:142)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
[jira] [Commented] (DRILL-6994) TIMESTAMP type DOB column in Spark parquet is treated as VARBINARY in Drill
[
https://issues.apache.org/jira/browse/DRILL-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749568#comment-16749568
]
Khurram Faraaz commented on DRILL-6994:
---
[~kkhatua] here are the parquet schema details
{noformat}
[test@md123-45 parquet]# ./parquet-schema
infer_schema_example.parquet/part-0-53f066b2-ca90-499e-a976-e5282d1b59ac-c000.snappy.parquet
message spark_schema {
optional binary Name (UTF8);
optional binary Department (UTF8);
optional int32 years_of_experience;
optional int96 DOB;
}{noformat}
> TIMESTAMP type DOB column in Spark parquet is treated as VARBINARY in Drill
> ---
>
> Key: DRILL-6994
> URL: https://issues.apache.org/jira/browse/DRILL-6994
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Priority: Major
>
> A timestamp type column in a parquet file created from Spark is treated as
> VARBINARY by Drill 1.14.0., Trying to cast DOB column to DATE results in an
> Exception, although the monthOfYear field is in the allowed range.
> Data used in the test
> {noformat}
> [test@md123 spark_data]# cat inferSchema_example.csv
> Name,Department,years_of_experience,DOB
> Sam,Software,5,1990-10-10
> Alex,Data Analytics,3,1992-10-10
> {noformat}
> Create the parquet file using the above CSV file
> {noformat}
> [test@md123 bin]# ./spark-shell
> 19/01/22 21:21:34 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> Spark context Web UI available at http://md123.qa.lab:4040
> Spark context available as 'sc' (master = local[*], app id =
> local-1548192099796).
> Spark session available as 'spark'.
> Welcome to
> __
> / __/__ ___ _/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 2.3.1-mapr-SNAPSHOT
> /_/
> Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_191)
> Type in expressions to have them evaluated.
> Type :help for more information.
> scala> import org.apache.spark.sql.\{DataFrame, SQLContext}
> import org.apache.spark.sql.\{DataFrame, SQLContext}
> scala> import org.apache.spark.\{SparkConf, SparkContext}
> import org.apache.spark.\{SparkConf, SparkContext}
> scala> val sqlContext: SQLContext = new SQLContext(sc)
> warning: there was one deprecation warning; re-run with -deprecation for
> details
> sqlContext: org.apache.spark.sql.SQLContext =
> org.apache.spark.sql.SQLContext@2e0163cb
> scala> val df =
> sqlContext.read.format("com.databricks.spark.csv").option("header",
> "true").option("inferSchema", "true").load("/apps/inferSchema_example.csv")
> df: org.apache.spark.sql.DataFrame = [Name: string, Department: string ... 2
> more fields]
> scala> df.printSchema
> test
> |-- Name: string (nullable = true)
> |-- Department: string (nullable = true)
> |-- years_of_experience: integer (nullable = true)
> |-- DOB: timestamp (nullable = true)
> scala> df.write.parquet("/apps/infer_schema_example.parquet")
> // Read the parquet file
> scala> val data =
> sqlContext.read.parquet("/apps/infer_schema_example.parquet")
> data: org.apache.spark.sql.DataFrame = [Name: string, Department: string ...
> 2 more fields]
> // Print the schema of the parquet file from Spark
> scala> data.printSchema
> test
> |-- Name: string (nullable = true)
> |-- Department: string (nullable = true)
> |-- years_of_experience: integer (nullable = true)
> |-- DOB: timestamp (nullable = true)
> // Display the contents of parquet file on spark-shell
> // register temp table and do a show on all records,to display.
> scala> data.registerTempTable("employee")
> warning: there was one deprecation warning; re-run with -deprecation for
> details
> scala> val allrecords = sqlContext.sql("SELeCT * FROM employee")
> allrecords: org.apache.spark.sql.DataFrame = [Name: string, Department:
> string ... 2 more fields]
> scala> allrecords.show()
> ++--+---+---+
> |Name| Department|years_of_experience| DOB|
> ++--+---+---+
> | Sam| Software| 5|1990-10-10 00:00:00|
> |Alex|Data Analytics| 3|1992-10-10 00:00:00|
> ++--+---+---+
> {noformat}
> Querying the parquet file from Drill 1.14.0-mapr, results in the DOB column
> (timestamp type in Spark) being treated as VARBINARY.
> {noformat}
> apache drill 1.14.0-mapr
> "a little sql for your nosql"
> 0: jdbc:drill:schema=dfs.tmp> select * from
> dfs.`/apps/infer_schema_example.parquet`;
> +---+-+--+--+
> | Name | Department | years_of_experience | DOB |
> +---+-+--+--+
> | Sam | Software |
[jira] [Updated] (DRILL-6994) TIMESTAMP type DOB column in Spark parquet is treated as VARBINARY in Drill
[
https://issues.apache.org/jira/browse/DRILL-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-6994:
--
Component/s: Execution - Data Types
> TIMESTAMP type DOB column in Spark parquet is treated as VARBINARY in Drill
> ---
>
> Key: DRILL-6994
> URL: https://issues.apache.org/jira/browse/DRILL-6994
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Priority: Major
>
> A timestamp type column in a parquet file created from Spark is treated as
> VARBINARY by Drill 1.14.0., Trying to cast DOB column to DATE results in an
> Exception, although the monthOfYear field is in the allowed range.
> Data used in the test
> {noformat}
> [test@md123 spark_data]# cat inferSchema_example.csv
> Name,Department,years_of_experience,DOB
> Sam,Software,5,1990-10-10
> Alex,Data Analytics,3,1992-10-10
> {noformat}
> Create the parquet file using the above CSV file
> {noformat}
> [test@md123 bin]# ./spark-shell
> 19/01/22 21:21:34 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> Spark context Web UI available at http://md123.qa.lab:4040
> Spark context available as 'sc' (master = local[*], app id =
> local-1548192099796).
> Spark session available as 'spark'.
> Welcome to
> __
> / __/__ ___ _/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 2.3.1-mapr-SNAPSHOT
> /_/
> Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_191)
> Type in expressions to have them evaluated.
> Type :help for more information.
> scala> import org.apache.spark.sql.\{DataFrame, SQLContext}
> import org.apache.spark.sql.\{DataFrame, SQLContext}
> scala> import org.apache.spark.\{SparkConf, SparkContext}
> import org.apache.spark.\{SparkConf, SparkContext}
> scala> val sqlContext: SQLContext = new SQLContext(sc)
> warning: there was one deprecation warning; re-run with -deprecation for
> details
> sqlContext: org.apache.spark.sql.SQLContext =
> org.apache.spark.sql.SQLContext@2e0163cb
> scala> val df =
> sqlContext.read.format("com.databricks.spark.csv").option("header",
> "true").option("inferSchema", "true").load("/apps/inferSchema_example.csv")
> df: org.apache.spark.sql.DataFrame = [Name: string, Department: string ... 2
> more fields]
> scala> df.printSchema
> test
> |-- Name: string (nullable = true)
> |-- Department: string (nullable = true)
> |-- years_of_experience: integer (nullable = true)
> |-- DOB: timestamp (nullable = true)
> scala> df.write.parquet("/apps/infer_schema_example.parquet")
> // Read the parquet file
> scala> val data =
> sqlContext.read.parquet("/apps/infer_schema_example.parquet")
> data: org.apache.spark.sql.DataFrame = [Name: string, Department: string ...
> 2 more fields]
> // Print the schema of the parquet file from Spark
> scala> data.printSchema
> test
> |-- Name: string (nullable = true)
> |-- Department: string (nullable = true)
> |-- years_of_experience: integer (nullable = true)
> |-- DOB: timestamp (nullable = true)
> // Display the contents of parquet file on spark-shell
> // register temp table and do a show on all records,to display.
> scala> data.registerTempTable("employee")
> warning: there was one deprecation warning; re-run with -deprecation for
> details
> scala> val allrecords = sqlContext.sql("SELeCT * FROM employee")
> allrecords: org.apache.spark.sql.DataFrame = [Name: string, Department:
> string ... 2 more fields]
> scala> allrecords.show()
> ++--+---+---+
> |Name| Department|years_of_experience| DOB|
> ++--+---+---+
> | Sam| Software| 5|1990-10-10 00:00:00|
> |Alex|Data Analytics| 3|1992-10-10 00:00:00|
> ++--+---+---+
> {noformat}
> Querying the parquet file from Drill 1.14.0-mapr, results in the DOB column
> (timestamp type in Spark) being treated as VARBINARY.
> {noformat}
> apache drill 1.14.0-mapr
> "a little sql for your nosql"
> 0: jdbc:drill:schema=dfs.tmp> select * from
> dfs.`/apps/infer_schema_example.parquet`;
> +---+-+--+--+
> | Name | Department | years_of_experience | DOB |
> +---+-+--+--+
> | Sam | Software | 5 | [B@2bef51f2 |
> | Alex | Data Analytics | 3 | [B@650eab8 |
> +---+-+--+--+
> 2 rows selected (0.229 seconds)
> // typeof(DOB) column returns a VARBINARY type, whereas the parquet schema in
> Spark for DOB: timestamp (nullable = true)
> 0: jdbc:drill:schema=dfs.tmp> select typeof(DOB) from
>
[jira] [Created] (DRILL-6994) TIMESTAMP type DOB column in Spark parquet is treated as VARBINARY in Drill
Khurram Faraaz created DRILL-6994:
-
Summary: TIMESTAMP type DOB column in Spark parquet is treated as
VARBINARY in Drill
Key: DRILL-6994
URL: https://issues.apache.org/jira/browse/DRILL-6994
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
A timestamp type column in a parquet file created from Spark is treated as
VARBINARY by Drill 1.14.0., Trying to cast DOB column to DATE results in an
Exception, although the monthOfYear field is in the allowed range.
Data used in the test
{noformat}
[test@md123 spark_data]# cat inferSchema_example.csv
Name,Department,years_of_experience,DOB
Sam,Software,5,1990-10-10
Alex,Data Analytics,3,1992-10-10
{noformat}
Create the parquet file using the above CSV file
{noformat}
[test@md123 bin]# ./spark-shell
19/01/22 21:21:34 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://md123.qa.lab:4040
Spark context available as 'sc' (master = local[*], app id =
local-1548192099796).
Spark session available as 'spark'.
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.3.1-mapr-SNAPSHOT
/_/
Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_191)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import org.apache.spark.sql.\{DataFrame, SQLContext}
import org.apache.spark.sql.\{DataFrame, SQLContext}
scala> import org.apache.spark.\{SparkConf, SparkContext}
import org.apache.spark.\{SparkConf, SparkContext}
scala> val sqlContext: SQLContext = new SQLContext(sc)
warning: there was one deprecation warning; re-run with -deprecation for details
sqlContext: org.apache.spark.sql.SQLContext =
org.apache.spark.sql.SQLContext@2e0163cb
scala> val df =
sqlContext.read.format("com.databricks.spark.csv").option("header",
"true").option("inferSchema", "true").load("/apps/inferSchema_example.csv")
df: org.apache.spark.sql.DataFrame = [Name: string, Department: string ... 2
more fields]
scala> df.printSchema
test
|-- Name: string (nullable = true)
|-- Department: string (nullable = true)
|-- years_of_experience: integer (nullable = true)
|-- DOB: timestamp (nullable = true)
scala> df.write.parquet("/apps/infer_schema_example.parquet")
// Read the parquet file
scala> val data = sqlContext.read.parquet("/apps/infer_schema_example.parquet")
data: org.apache.spark.sql.DataFrame = [Name: string, Department: string ... 2
more fields]
// Print the schema of the parquet file from Spark
scala> data.printSchema
test
|-- Name: string (nullable = true)
|-- Department: string (nullable = true)
|-- years_of_experience: integer (nullable = true)
|-- DOB: timestamp (nullable = true)
// Display the contents of parquet file on spark-shell
// register temp table and do a show on all records,to display.
scala> data.registerTempTable("employee")
warning: there was one deprecation warning; re-run with -deprecation for details
scala> val allrecords = sqlContext.sql("SELeCT * FROM employee")
allrecords: org.apache.spark.sql.DataFrame = [Name: string, Department: string
... 2 more fields]
scala> allrecords.show()
++--+---+---+
|Name| Department|years_of_experience| DOB|
++--+---+---+
| Sam| Software| 5|1990-10-10 00:00:00|
|Alex|Data Analytics| 3|1992-10-10 00:00:00|
++--+---+---+
{noformat}
Querying the parquet file from Drill 1.14.0-mapr, results in the DOB column
(timestamp type in Spark) being treated as VARBINARY.
{noformat}
apache drill 1.14.0-mapr
"a little sql for your nosql"
0: jdbc:drill:schema=dfs.tmp> select * from
dfs.`/apps/infer_schema_example.parquet`;
+---+-+--+--+
| Name | Department | years_of_experience | DOB |
+---+-+--+--+
| Sam | Software | 5 | [B@2bef51f2 |
| Alex | Data Analytics | 3 | [B@650eab8 |
+---+-+--+--+
2 rows selected (0.229 seconds)
// typeof(DOB) column returns a VARBINARY type, whereas the parquet schema in
Spark for DOB: timestamp (nullable = true)
0: jdbc:drill:schema=dfs.tmp> select typeof(DOB) from
dfs.`/apps/infer_schema_example.parquet`;
++
| EXPR$0 |
++
| VARBINARY |
| VARBINARY |
++
2 rows selected (0.199 seconds)
{noformat}
// CAST to DATE type results in Exception, though the monthOfYear is in the
range [1,12]
{noformat}
0: jdbc:drill:schema=dfs.tmp> select cast(DOB as DATE) from
dfs.`/apps/infer_schema_example.parquet`;
Error: SYSTEM ERROR: IllegalFieldValueException: Value 0 for monthOfYear must
be in the range [1,12]
Fragment
[jira] [Commented] (DRILL-6937) sys.functions table needs a fix in the names column
[
https://issues.apache.org/jira/browse/DRILL-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748292#comment-16748292
]
Khurram Faraaz commented on DRILL-6937:
---
One question is, why do we have two different entries in the sys.functions
table, one with name being ">" and other with name being "greater_than", all
other columns for the two functions seem to have identical values, except for
the name.
{noformat}
| > | BIGINT-REQUIRED,BIGINT-REQUIRED | BIT | built-in | false |
| greater_than | BIGINT-REQUIRED,BIGINT-REQUIRED | BIT | built-in | false |
{noformat}
> sys.functions table needs a fix in the names column
> ---
>
> Key: DRILL-6937
> URL: https://issues.apache.org/jira/browse/DRILL-6937
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
>Reporter: Khurram Faraaz
>Assignee: Kunal Khatua
>Priority: Minor
> Fix For: 1.16.0
>
>
> The function names in the name column of sys.functions in some cases, are the
> operators, this is not the expected behavior, the name column should have
> actual names and not the operators.
> I am on Drill 1.15.0 commit : 8743e8f1e8d5bca4d67c94d07a8560ad356ff2b6
> {noformat}
> Apache Drill 1.15.0
> "Data is the new oil. Ready to Drill some?"
> 0: jdbc:drill:schema=dfs.tmp> select count(*) from sys.functions;
> +-+
> | EXPR$0 |
> +-+
> | 2846 |
> +-+
> 1 row selected (0.327 seconds)
> 0: jdbc:drill:schema=dfs.tmp>
> {noformat}
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select distinct name from sys.functions limit
> 12;
> ++
> | name |
> ++
> | != |
> | $sum0 |
> | && |
> | - |
> | /int |
> | < |
> | <= |
> | <> |
> | = |
> | == |
> | > |
> | >= |
> ++
> 12 rows selected (0.175 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6990) IllegalStateException: The current reader doesn't support getting next information
Khurram Faraaz created DRILL-6990:
-
Summary: IllegalStateException: The current reader doesn't support
getting next information
Key: DRILL-6990
URL: https://issues.apache.org/jira/browse/DRILL-6990
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Attachments: parqt_nestedArray.parquet.tar
Reading a parquet file created from Spark, returns IllegalStateException: The
current reader doesn't support getting next information
Drill 1.14.0, parquet file created from Spark is attached here.
//Steps to create parquet file from Spark 2.3.1
[root@ba102-495 ~]# cd /opt/mapr/spark/spark-2.3.1
[root@ba102-495 spark-2.3.1]# cd bin
[root@ba102-495 bin]# ./spark-shell
19/01/21 22:57:05 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://qa102-45.qa.lab:4040
Spark context available as 'sc' (master = local[*], app id =
local-1548111430809).
Spark session available as 'spark'.
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.3.1-mapr-SNAPSHOT
/_/
Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_191)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import spark.implicits._
import spark.implicits._
scala> val df = spark.read.json("/apps/nestedDataJson.json")
df: org.apache.spark.sql.DataFrame = [id: bigint, nested_array:
array>]
scala> df.write.parquet("/apps/parqt_nestedArray.parquet")
Data used in test
{noformat}
[root@ba102-495 ~]# cat nestedDataJson.json
{"id":19,"nested_array":[[1,2,3,4],[5,6,7,8],[9,10,12]]}
{"id":14121,"nested_array":[[1,3,4],[5,6,8],[9,11,12]]}
{"id":18894,"nested_array":[[1,3,4],[5,6,7,8],[9,10,11,12]]}
{"id":12499,"nested_array":[[1,4],[5,7,8],[9,11,12]]}
{"id":120,"nested_array":[[1,4],[5,7,8],[9,10,11,12]]}
{"id":12,"nested_array":[[1,2,3,4],[5,6,7,8],[11,12]]}
{"id":13,"nested_array":[[1,2,3,4],[5,8],[9,10,11,12]]}
{"id":14,"nested_array":[[1,2,3,4],[5,68],[9,10,11,12]]}
{"id":123,"nested_array":[[1,2,3,4],[5,8],[9,10,11,12]]}
{"id":124,"nested_array":[[1,2,4],[5,6,7,8],[9,10,11,12]]}
{"id":134,"nested_array":[[1,4],[5,8],[9,12]]}
{noformat}
>From drillbit.log
{noformat}
Query Failed: An Error Occurred
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
IllegalStateException: The current reader doesn't support getting next
information. Fragment 0:0 [Error Id: c16c70dd-6565-463f-83a7-118ccd8442e2 on
ba102-495.qa.lab:31010]
...
...
2019-01-21 23:08:11,268 [23b9af24-10b9-ad11-5583-ecc3e0c562e6:frag:0:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: The
current reader doesn't support getting next information.
Fragment 0:0
[Error Id: c16c70dd-6565-463f-83a7-118ccd8442e2 on ba102-495.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: The current reader doesn't support getting next
information.
Fragment 0:0
[Error Id: c16c70dd-6565-463f-83a7-118ccd8442e2 on ba102-495.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361)
[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216)
[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327)
[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.14.0-mapr.jar:1.14.0-mapr]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: java.lang.IllegalStateException: The current reader doesn't support
getting next information.
at
org.apache.drill.exec.vector.complex.impl.AbstractBaseReader.next(AbstractBaseReader.java:64)
~[vector-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.vector.complex.impl.SingleMapReaderImpl.next(SingleMapReaderImpl.java:31)
~[vector-1.14.0-mapr.jar:1.14.0-mapr]
at
org.apache.drill.exec.test.generated.ProjectorGen971.doEval(ProjectorTemplate.java:35)
~[na:na]
at
org.apache.drill.exec.test.generated.ProjectorGen971.projectRecords(ProjectorTemplate.java:67)
~[na:na]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:231)
~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at
[jira] [Created] (DRILL-6979) Add autofocus attribute to username on login page, and to query textbox on Query tab.
Khurram Faraaz created DRILL-6979: - Summary: Add autofocus attribute to username on login page, and to query textbox on Query tab. Key: DRILL-6979 URL: https://issues.apache.org/jira/browse/DRILL-6979 Project: Apache Drill Issue Type: Improvement Components: Web Server Affects Versions: 1.16.0 Reporter: Khurram Faraaz Assignee: Khurram Faraaz Add autofocus attribute to username on login page, and to query textbox on Query tab. The two text boxes that need the change are in these files ./exec/java-exec/src/main/resources/rest/query/query.ftl ./exec/java-exec/src/main/resources/rest/login.ftl -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6146) UNION with empty input on any one side returns incorrect results
[
https://issues.apache.org/jira/browse/DRILL-6146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-6146:
--
Priority: Critical (was: Major)
> UNION with empty input on any one side returns incorrect results
>
>
> Key: DRILL-6146
> URL: https://issues.apache.org/jira/browse/DRILL-6146
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning Optimization
>Affects Versions: 1.12.0
>Reporter: Khurram Faraaz
>Assignee: Vitalii Diravka
>Priority: Critical
>
> When any one side of the UNION has an empty file as input, Drill returns
> incorrect results.
>
> table t3 does not have any data inserted into its rows. Postgress returns 1
> as the result for both the queries, whereas Drill does not.
>
> {noformat}
> postgres=# create table t3(id int, name varchar(25));
> CREATE TABLE
> postgres=# select * from (values(1)) t union select id from t3;
> 1
>
> postgres=# select id from t3 union select * from (values(1)) t;
> 1
> {noformat}
>
>
> Results from Drill 1.12.0-mapr, note we return result 1 as result after the
> union.
> We have a directory named empty_JSON_f , and it has a single empty JSON file
> (that JSON file has no content in it, it is empty).
>
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select * from (values(1)) UNION select id from
> empty_JSON_f;
> +-+
> | EXPR$0 |
> +-+
> | 1 |
> +-+
> 1 row selected (2.272 seconds){noformat}
> However, in this query we return null and loose the value 1 from the right
> hand side, after the union, this doesn't seem correct
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select id from empty_JSON_f UNION select * from
> (values(1));
> +---+
> | id |
> +---+
> | null |
> +---+
> 1 row selected (0.33 seconds){noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6146) UNION with empty input on any one side returns incorrect results
[
https://issues.apache.org/jira/browse/DRILL-6146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-6146:
--
Affects Version/s: 1.14.0
> UNION with empty input on any one side returns incorrect results
>
>
> Key: DRILL-6146
> URL: https://issues.apache.org/jira/browse/DRILL-6146
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning Optimization
>Affects Versions: 1.12.0, 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Vitalii Diravka
>Priority: Critical
>
> When any one side of the UNION has an empty file as input, Drill returns
> incorrect results.
>
> table t3 does not have any data inserted into its rows. Postgress returns 1
> as the result for both the queries, whereas Drill does not.
>
> {noformat}
> postgres=# create table t3(id int, name varchar(25));
> CREATE TABLE
> postgres=# select * from (values(1)) t union select id from t3;
> 1
>
> postgres=# select id from t3 union select * from (values(1)) t;
> 1
> {noformat}
>
>
> Results from Drill 1.12.0-mapr, note we return result 1 as result after the
> union.
> We have a directory named empty_JSON_f , and it has a single empty JSON file
> (that JSON file has no content in it, it is empty).
>
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select * from (values(1)) UNION select id from
> empty_JSON_f;
> +-+
> | EXPR$0 |
> +-+
> | 1 |
> +-+
> 1 row selected (2.272 seconds){noformat}
> However, in this query we return null and loose the value 1 from the right
> hand side, after the union, this doesn't seem correct
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select id from empty_JSON_f UNION select * from
> (values(1));
> +---+
> | id |
> +---+
> | null |
> +---+
> 1 row selected (0.33 seconds){noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6146) UNION with empty input on any one side returns incorrect results
[
https://issues.apache.org/jira/browse/DRILL-6146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742818#comment-16742818
]
Khurram Faraaz commented on DRILL-6146:
---
The issue can be reproduced on Drill 1.14.0.
When LHS has an empty table ( in this case the directory empty_dir has an empty
file f1.json )
Whereas, when we flip the LHS and RHS in the failing query, it returns a
result, with one record, having the value 1
{noformat}
[test@test102-45 ~]# hadoop fs -ls /tmp/empty_dir
Found 1 items
-rwxr-xr-x 3 test test 0 2019-01-15 06:59 /tmp/empty_dir/f1.json
failing query => select id from dfs.tmp.`empty_dir` union select * from
(values(1));
00-00Screen : rowType = RecordType(ANY id): rowcount = 1.0, cumulative cost
= {8.1 rows, 20.1 cpu, 0.0 io, 0.0 network, 16.0 memory}, id = 1216
00-01 Project(id=[$0]) : rowType = RecordType(ANY id): rowcount = 1.0,
cumulative cost = {8.0 rows, 20.0 cpu, 0.0 io, 0.0 network, 16.0 memory}, id =
1215
00-02StreamAgg(group=[{0}]) : rowType = RecordType(ANY id): rowcount =
1.0, cumulative cost = {7.0 rows, 19.0 cpu, 0.0 io, 0.0 network, 16.0 memory},
id = 1214
00-03 Sort(sort0=[$0], dir0=[ASC]) : rowType = RecordType(ANY id):
rowcount = 2.0, cumulative cost = {5.0 rows, 11.0 cpu, 0.0 io, 0.0 network,
16.0 memory}, id = 1213
00-04UnionAll(all=[true]) : rowType = RecordType(ANY id): rowcount
= 2.0, cumulative cost = {3.0 rows, 3.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
id = 1212
00-06 Scan(table=[[dfs, tmp, empty_dir]], groupscan=[EasyGroupScan
[selectionRoot=maprfs:/tmp/empty_dir, numFiles=1, columns=[`id`],
files=[maprfs:///tmp/empty_dir/f1.json]]]) : rowType = RecordType(ANY id):
rowcount = 1.0, cumulative cost = {0.0 rows, 0.0 cpu, 0.0 io, 0.0 network, 0.0
memory}, id = 1210
00-05 Values(tuples=[[{ 1 }]]) : rowType = RecordType(INTEGER
EXPR$0): rowcount = 1.0, cumulative cost = {1.0 rows, 1.0 cpu, 0.0 io, 0.0
network, 0.0 memory}, id = 1211{noformat}
> UNION with empty input on any one side returns incorrect results
>
>
> Key: DRILL-6146
> URL: https://issues.apache.org/jira/browse/DRILL-6146
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning Optimization
>Affects Versions: 1.12.0
>Reporter: Khurram Faraaz
>Assignee: Vitalii Diravka
>Priority: Major
>
> When any one side of the UNION has an empty file as input, Drill returns
> incorrect results.
>
> table t3 does not have any data inserted into its rows. Postgress returns 1
> as the result for both the queries, whereas Drill does not.
>
> {noformat}
> postgres=# create table t3(id int, name varchar(25));
> CREATE TABLE
> postgres=# select * from (values(1)) t union select id from t3;
> 1
>
> postgres=# select id from t3 union select * from (values(1)) t;
> 1
> {noformat}
>
>
> Results from Drill 1.12.0-mapr, note we return result 1 as result after the
> union.
> We have a directory named empty_JSON_f , and it has a single empty JSON file
> (that JSON file has no content in it, it is empty).
>
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select * from (values(1)) UNION select id from
> empty_JSON_f;
> +-+
> | EXPR$0 |
> +-+
> | 1 |
> +-+
> 1 row selected (2.272 seconds){noformat}
> However, in this query we return null and loose the value 1 from the right
> hand side, after the union, this doesn't seem correct
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select id from empty_JSON_f UNION select * from
> (values(1));
> +---+
> | id |
> +---+
> | null |
> +---+
> 1 row selected (0.33 seconds){noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6518) DESCRIBE command on Drill created parquet table does not return results
[
https://issues.apache.org/jira/browse/DRILL-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742538#comment-16742538
]
Khurram Faraaz commented on DRILL-6518:
---
[~arina] Do we know when Drill metastore will become part of Apache Drill
master ?
> DESCRIBE command on Drill created parquet table does not return results
> ---
>
> Key: DRILL-6518
> URL: https://issues.apache.org/jira/browse/DRILL-6518
> Project: Apache Drill
> Issue Type: New Feature
> Components: Storage - Parquet
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Priority: Major
> Fix For: Future
>
> Attachments: 0_0_0.parquet, item.drill.parquet_metadata
>
>
> Describe command on a Drill (1.14.0) created parquet table, does not return
> the table description.
> Parquet file and parquet metadata cache file for item table is attached here,
> it has the details of column types in the metadata cache file.
> {noformat}
> Apache Drill 1.14.0-SNAPSHOT
> commit : b447260e49dc4a8c906f5b310c037fe6dd77166f
> {noformat}
> {noformat}
> // DESCRIBE commands returns no information about the table.
> 0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> describe
> dfs.`/drill/testdata/tpcds_sf1/parquet/item`;
> +--++--+
> | COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
> +--++--+
> +--++--+
> No rows selected (0.221 seconds)
> 0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> refresh table metadata
> dfs.`/drill/testdata/tpcds_sf1/parquet/item`;
> +---+--+
> | ok | summary |
> +---+--+
> | true | Successfully updated metadata for table
> /drill/testdata/tpcds_sf1/parquet/item. |
> +---+--+
> 1 row selected (0.173 seconds)
> 0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> describe
> dfs.`/drill/testdata/tpcds_sf1/parquet/item`;
> +--++--+
> | COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
> +--++--+
> +--++--+
> No rows selected (0.229 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6961) Error Occurred: Cannot connect to the db. query INFORMATION_SCHEMA.VIEWS : Maybe you have incorrect connection params or db unavailable now (timeout)
Khurram Faraaz created DRILL-6961:
-
Summary: Error Occurred: Cannot connect to the db. query
INFORMATION_SCHEMA.VIEWS : Maybe you have incorrect connection params or db
unavailable now (timeout)
Key: DRILL-6961
URL: https://issues.apache.org/jira/browse/DRILL-6961
Project: Apache Drill
Issue Type: Improvement
Components: Storage - Information Schema
Affects Versions: 1.13.0
Reporter: Khurram Faraaz
Trying to query drill information_schema.views table returns error. Disabling
openTSDB plugin resolves the problem.
Drill 1.13.0
Failing query :
{noformat}
SELECT TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, VIEW_DEFINITION FROM
INFORMATION_SCHEMA.`VIEWS` where VIEW_DEFINITION not like 'kraken';
{noformat}
Stack Trace from drillbit.log
{noformat}
2019-01-07 15:36:21,975 [23cc39aa-2618-e9f0-e77e-4fafa6edc314:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query id
23cc39aa-2618-e9f0-e77e-4fafa6edc314: SELECT TABLE_CATALOG, TABLE_SCHEMA,
TABLE_NAME, VIEW_DEFINITION FROM INFORMATION_SCHEMA.`VIEWS` where
VIEW_DEFINITION not like 'kraken'
2019-01-07 15:36:35,221 [23cc39aa-2618-e9f0-e77e-4fafa6edc314:frag:0:0] INFO
o.a.d.e.s.o.c.services.ServiceImpl - User Error Occurred: Cannot connect to the
db. Maybe you have incorrect connection params or db unavailable now (timeout)
org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Cannot
connect to the db. Maybe you have incorrect connection params or db unavailable
now
[Error Id: f8b4c074-ba62-4691-b142-a8ea6e4f6b2a ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl.getTableNames(ServiceImpl.java:107)
[drill-opentsdb-storage-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl.getAllMetricNames(ServiceImpl.java:70)
[drill-opentsdb-storage-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory$OpenTSDBSchema.getTableNames(OpenTSDBSchemaFactory.java:78)
[drill-opentsdb-storage-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.calcite.jdbc.SimpleCalciteSchema.addImplicitTableToBuilder(SimpleCalciteSchema.java:106)
[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
at org.apache.calcite.jdbc.CalciteSchema.getTableNames(CalciteSchema.java:318)
[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
at
org.apache.calcite.jdbc.CalciteSchema$SchemaPlusImpl.getTableNames(CalciteSchema.java:587)
[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
at
org.apache.calcite.jdbc.CalciteSchema$SchemaPlusImpl.getTableNames(CalciteSchema.java:548)
[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
at
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.visitTables(InfoSchemaRecordGenerator.java:227)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:216)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:209)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:196)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.store.ischema.InfoSchemaTableType.getRecordReader(InfoSchemaTableType.java:58)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:34)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:30)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at org.apache.drill.exec.physical.impl.ImplCreator$2.run(ImplCreator.java:146)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at org.apache.drill.exec.physical.impl.ImplCreator$2.run(ImplCreator.java:142)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_144]
at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_144]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1633)
[hadoop-common-2.7.0-mapr-1710.jar:na]
at
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:142)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:182)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:137)
[drill-java-exec-1.13.0-mapr.jar:1.13.0-mapr]
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:182)
[jira] [Created] (DRILL-6942) Provide ability to sort list of profiles on Drill web UI
Khurram Faraaz created DRILL-6942: - Summary: Provide ability to sort list of profiles on Drill web UI Key: DRILL-6942 URL: https://issues.apache.org/jira/browse/DRILL-6942 Project: Apache Drill Issue Type: Improvement Components: Web Server Affects Versions: 1.15.0 Reporter: Khurram Faraaz We need to provide a way to sort the query profiles, on the profiles tab on Drill web UI. The use case is when users run many queries (several hundred queries), they want a way to list the queries that have taken the longest time (i.e. duration) to complete. All queries that completed, failed or were canceled and took very long, all such queries should be listed on the top in the profiles tab page on Drill web ui. An option to list the query profiles based on total duration should be provided to user. That way user can easily identify such long running queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DRILL-6942) Provide ability to sort list of profiles on Drill web UI
[ https://issues.apache.org/jira/browse/DRILL-6942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Khurram Faraaz reassigned DRILL-6942: - Assignee: Kunal Khatua > Provide ability to sort list of profiles on Drill web UI > > > Key: DRILL-6942 > URL: https://issues.apache.org/jira/browse/DRILL-6942 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.15.0 >Reporter: Khurram Faraaz >Assignee: Kunal Khatua >Priority: Major > > We need to provide a way to sort the query profiles, on the profiles tab on > Drill web UI. > The use case is when users run many queries (several hundred queries), they > want a way to list the queries that have taken the longest time (i.e. > duration) to complete. All queries that completed, failed or were canceled > and took very long, all such queries should be listed on the top in the > profiles tab page on Drill web ui. > An option to list the query profiles based on total duration should be > provided to user. That way user can easily identify such long running queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6937) sys.functions table needs a fix in the names column
Khurram Faraaz created DRILL-6937:
-
Summary: sys.functions table needs a fix in the names column
Key: DRILL-6937
URL: https://issues.apache.org/jira/browse/DRILL-6937
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.15.0
Reporter: Khurram Faraaz
The function names in the name column of sys.functions in some cases, are the
operators, this is not the expected behavior, the name column should have
actual names and not the operators.
I am on Drill 1.15.0 commit : 8743e8f1e8d5bca4d67c94d07a8560ad356ff2b6
{noformat}
Apache Drill 1.15.0
"Data is the new oil. Ready to Drill some?"
0: jdbc:drill:schema=dfs.tmp> select count(*) from sys.functions;
+-+
| EXPR$0 |
+-+
| 2846 |
+-+
1 row selected (0.327 seconds)
0: jdbc:drill:schema=dfs.tmp>
{noformat}
{noformat}
0: jdbc:drill:schema=dfs.tmp> select distinct name from sys.functions limit 12;
++
| name |
++
| != |
| $sum0 |
| && |
| - |
| /int |
| < |
| <= |
| <> |
| = |
| == |
| > |
| >= |
++
12 rows selected (0.175 seconds)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (DRILL-6937) sys.functions table needs a fix in the names column
[
https://issues.apache.org/jira/browse/DRILL-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz reassigned DRILL-6937:
-
Assignee: Kunal Khatua
> sys.functions table needs a fix in the names column
> ---
>
> Key: DRILL-6937
> URL: https://issues.apache.org/jira/browse/DRILL-6937
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
>Reporter: Khurram Faraaz
>Assignee: Kunal Khatua
>Priority: Major
>
> The function names in the name column of sys.functions in some cases, are the
> operators, this is not the expected behavior, the name column should have
> actual names and not the operators.
> I am on Drill 1.15.0 commit : 8743e8f1e8d5bca4d67c94d07a8560ad356ff2b6
> {noformat}
> Apache Drill 1.15.0
> "Data is the new oil. Ready to Drill some?"
> 0: jdbc:drill:schema=dfs.tmp> select count(*) from sys.functions;
> +-+
> | EXPR$0 |
> +-+
> | 2846 |
> +-+
> 1 row selected (0.327 seconds)
> 0: jdbc:drill:schema=dfs.tmp>
> {noformat}
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select distinct name from sys.functions limit
> 12;
> ++
> | name |
> ++
> | != |
> | $sum0 |
> | && |
> | - |
> | /int |
> | < |
> | <= |
> | <> |
> | = |
> | == |
> | > |
> | >= |
> ++
> 12 rows selected (0.175 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6933) Ctrl+Enter meta combo not working on Windows & mac OS
[ https://issues.apache.org/jira/browse/DRILL-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16730443#comment-16730443 ] Khurram Faraaz commented on DRILL-6933: --- [~kkhatua] yes that is correct, impersonation is not enabled on my setup. > Ctrl+Enter meta combo not working on Windows & mac OS > - > > Key: DRILL-6933 > URL: https://issues.apache.org/jira/browse/DRILL-6933 > Project: Apache Drill > Issue Type: Bug > Components: Web Server >Affects Versions: 1.15.0 > Environment: Windows 10 Pro >Reporter: Bridget Bevens >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.16.0 > > > On my Windows machine, when I enter query text (in the query editor on the > Query page in the Drill Web UI) and then press Ctrl+Enter, the query is not > submitted. However, when I click Submit, the query is submitted and runs okay. > I've tried for all radio buttons, SQL, Logical, Physical and Ctrl+Enter does > not work for me. > I've also tried from the query profile, where you can edit and resubmit a > query and that did not work for me either. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DRILL-6933) Ctrl+Enter meta combo not working on Windows & mac OS
[ https://issues.apache.org/jira/browse/DRILL-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Khurram Faraaz reassigned DRILL-6933: - Assignee: Kunal Khatua > Ctrl+Enter meta combo not working on Windows & mac OS > - > > Key: DRILL-6933 > URL: https://issues.apache.org/jira/browse/DRILL-6933 > Project: Apache Drill > Issue Type: Bug > Components: Web Server >Affects Versions: 1.15.0 > Environment: Windows 10 Pro >Reporter: Bridget Bevens >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.15.0 > > > On my Windows machine, when I enter query text (in the query editor on the > Query page in the Drill Web UI) and then press Ctrl+Enter, the query is not > submitted. However, when I click Submit, the query is submitted and runs okay. > I've tried for all radio buttons, SQL, Logical, Physical and Ctrl+Enter does > not work for me. > I've also tried from the query profile, where you can edit and resubmit a > query and that did not work for me either. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6933) Ctrl+Enter meta combo not working on Windows & mac OS
[ https://issues.apache.org/jira/browse/DRILL-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Khurram Faraaz updated DRILL-6933: -- Summary: Ctrl+Enter meta combo not working on Windows & mac OS (was: Ctrl+Enter meta combo not working on Windows) > Ctrl+Enter meta combo not working on Windows & mac OS > - > > Key: DRILL-6933 > URL: https://issues.apache.org/jira/browse/DRILL-6933 > Project: Apache Drill > Issue Type: Bug > Components: Web Server >Affects Versions: 1.15.0 > Environment: Windows 10 Pro >Reporter: Bridget Bevens >Priority: Major > Fix For: 1.15.0 > > > On my Windows machine, when I enter query text (in the query editor on the > Query page in the Drill Web UI) and then press Ctrl+Enter, the query is not > submitted. However, when I click Submit, the query is submitted and runs okay. > I've tried for all radio buttons, SQL, Logical, Physical and Ctrl+Enter does > not work for me. > I've also tried from the query profile, where you can edit and resubmit a > query and that did not work for me either. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6933) Ctrl+Enter meta combo not working on Windows
[ https://issues.apache.org/jira/browse/DRILL-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Khurram Faraaz updated DRILL-6933: -- Priority: Major (was: Minor) > Ctrl+Enter meta combo not working on Windows > > > Key: DRILL-6933 > URL: https://issues.apache.org/jira/browse/DRILL-6933 > Project: Apache Drill > Issue Type: Bug > Components: Web Server >Affects Versions: 1.15.0 > Environment: Windows 10 Pro >Reporter: Bridget Bevens >Priority: Major > Fix For: 1.15.0 > > > On my Windows machine, when I enter query text (in the query editor on the > Query page in the Drill Web UI) and then press Ctrl+Enter, the query is not > submitted. However, when I click Submit, the query is submitted and runs okay. > I've tried for all radio buttons, SQL, Logical, Physical and Ctrl+Enter does > not work for me. > I've also tried from the query profile, where you can edit and resubmit a > query and that did not work for me either. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6933) Ctrl+Enter meta combo not working on Windows
[ https://issues.apache.org/jira/browse/DRILL-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16730057#comment-16730057 ] Khurram Faraaz commented on DRILL-6933: --- [~bbevens] this (Meta+Enter) to submit query from web UI, does not work on the Mac too for me. I am on mac OS High Sierra version 10.13.3 Apache Drill 1.15.0 commit 8743e8f1e8d5bca4d67c94d07a8560ad356ff2b6 > Ctrl+Enter meta combo not working on Windows > > > Key: DRILL-6933 > URL: https://issues.apache.org/jira/browse/DRILL-6933 > Project: Apache Drill > Issue Type: Bug > Components: Web Server >Affects Versions: 1.15.0 > Environment: Windows 10 Pro >Reporter: Bridget Bevens >Priority: Minor > Fix For: 1.15.0 > > > On my Windows machine, when I enter query text (in the query editor on the > Query page in the Drill Web UI) and then press Ctrl+Enter, the query is not > submitted. However, when I click Submit, the query is submitted and runs okay. > I've tried for all radio buttons, SQL, Logical, Physical and Ctrl+Enter does > not work for me. > I've also tried from the query profile, where you can edit and resubmit a > query and that did not work for me either. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6816) NPE - Concurrent query execution using PreparedStatement
[
https://issues.apache.org/jira/browse/DRILL-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713212#comment-16713212
]
Khurram Faraaz commented on DRILL-6816:
---
[~KazydubB] and [~vitalii] The docs say we should not use executeQuery method
cannot be called on a {{PreparedStatement}} or {{CallableStatement}}.
Then in my test and in [~vitalii]'s test how did its use succeed ?, I am
confused.
> NPE - Concurrent query execution using PreparedStatement
> -
>
> Key: DRILL-6816
> URL: https://issues.apache.org/jira/browse/DRILL-6816
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Vitalii Diravka
>Priority: Major
> Attachments: test_tbl.json
>
>
> Concurrent query execution from JDBC program using PreparedStatement results
> in NPE.
> Queries that were executed concurrently are (part of a query file),
> {noformat}
> select id from `test_tbl.json`
> select count(id) from `test_tbl.json`
> select count(*) from `test_tbl.json`
> select * from `test_tbl.json`
> {noformat}
> Drill 1.14.0
> git.commit.id=35a1ae23c9b280b9e73cb0f6f01808c996515454
> MapR version => 6.1.0.20180911143226.GA (secure cluster)
> JDBC driver used was org.apache.drill.jdbc.Driver
> Executing the above queries concurrently using a Statement object results in
> successful query execution.
> {noformat}
> Statement stmt = conn.createStatement();
> ResultSet rs = stmt.executeQuery(query);
> {noformat}
> However, when the same queries listed above are executed using a
> PreparedStatement object we see an NPE
> {noformat}
> PreparedStatement prdstmnt = conn.prepareStatement(query);
> prdstmnt.executeUpdate();
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 17:04:32.941 [pool-1-thread-3] DEBUG o.a.d.j.impl.DrillStatementRegistry -
> Adding to open-statements registry:
> org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@35757005
> 17:04:32.941 [pool-1-thread-2] DEBUG o.a.d.j.impl.DrillStatementRegistry -
> Adding to open-statements registry:
> org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d4413b8
> 17:04:32.956 [pool-1-thread-1] DEBUG o.a.d.j.impl.DrillStatementRegistry -
> Adding to open-statements registry:
> org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@5eb3b9ab
> 17:04:32.956 [pool-1-thread-4] DEBUG o.a.d.j.impl.DrillStatementRegistry -
> Adding to open-statements registry:
> org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d9367d0
> java.lang.NullPointerException
> at java.util.Objects.requireNonNull(Objects.java:203)
> at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
> at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1143)
> at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1150)
> at
> org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:511)
> at
> org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:146)
> at
> org.apache.drill.jdbc.impl.DrillPreparedStatementImpl.executeLargeUpdate(DrillPreparedStatementImpl.java:512)
> at
> org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:142)
> at RunQuery.executeQuery(RunQuery.java:61)
> at RunQuery.run(RunQuery.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-4055) Query over nested empty directory results in IOB Exception
[
https://issues.apache.org/jira/browse/DRILL-4055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711854#comment-16711854
]
Khurram Faraaz commented on DRILL-4055:
---
[~vitalii] Please add a unit test or a Functional test to cover this scenario,
thanks.
> Query over nested empty directory results in IOB Exception
> --
>
> Key: DRILL-4055
> URL: https://issues.apache.org/jira/browse/DRILL-4055
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.3.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Assignee: Vitalii Diravka
>Priority: Major
> Fix For: 1.13.0
>
>
> SELECT * over a nested empty directory results in IOB Exception. We need a
> better error message that states that the directory being queried is empty.
> git.commit.id=3a73f098
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select * from `nested_dirs/data/parquet`;
> Error: VALIDATION ERROR: Index: 0, Size: 0
> [Error Id: 18b21dac-199d-4e5f-8a28-7c91723e1b63 on centos-04.qa.lab:31010]
> (state=,code=0)
> The below command does not return any results, which is expected since there
> are no files in the directory, it is empty.
> [root@centos-01 ~]# hadoop fs -ls /tmp/nested_dirs/data/parquet
> [root@centos-01 ~]#
> Stack trace from drillbit.log
> org.apache.drill.common.exceptions.UserException: VALIDATION ERROR: Index: 0,
> Size: 0
> [Error Id: 18b21dac-199d-4e5f-8a28-7c91723e1b63 ]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
> ~[drill-common-1.3.0.jar:1.3.0]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:187)
> [drill-java-exec-1.3.0.jar:1.3.0]
> at
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:905)
> [drill-java-exec-1.3.0.jar:1.3.0]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:244)
> [drill-java-exec-1.3.0.jar:1.3.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_85]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_85]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_85]
> Caused by: org.apache.calcite.tools.ValidationException:
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at
> org.apache.calcite.prepare.PlannerImpl.validate(PlannerImpl.java:179)
> ~[calcite-core-1.4.0-drill-r8.jar:1.4.0-drill-r8]
> at
> org.apache.calcite.prepare.PlannerImpl.validateAndGetType(PlannerImpl.java:188)
> ~[calcite-core-1.4.0-drill-r8.jar:1.4.0-drill-r8]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode(DefaultSqlHandler.java:447)
> ~[drill-java-exec-1.3.0.jar:1.3.0]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert(DefaultSqlHandler.java:190)
> ~[drill-java-exec-1.3.0.jar:1.3.0]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:159)
> ~[drill-java-exec-1.3.0.jar:1.3.0]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:184)
> [drill-java-exec-1.3.0.jar:1.3.0]
> ... 5 common frames omitted
> Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635) ~[na:1.7.0_85]
> at java.util.ArrayList.get(ArrayList.java:411) ~[na:1.7.0_85]
> at
> org.apache.drill.exec.store.dfs.FileSelection.getFirstPath(FileSelection.java:126)
> ~[drill-java-exec-1.3.0.jar:1.3.0]
> at
> org.apache.drill.exec.store.dfs.BasicFormatMatcher.isReadable(BasicFormatMatcher.java:79)
> ~[drill-java-exec-1.3.0.jar:1.3.0]
> at
> org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create(WorkspaceSchemaFactory.java:340)
> ~[drill-java-exec-1.3.0.jar:1.3.0]
> at
> org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create(WorkspaceSchemaFactory.java:155)
> ~[drill-java-exec-1.3.0.jar:1.3.0]
> at
> org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.getNewEntry(ExpandingConcurrentMap.java:96)
> ~[drill-java-exec-1.3.0.jar:1.3.0]
> at
> org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.get(ExpandingConcurrentMap.java:90)
> ~[drill-java-exec-1.3.0.jar:1.3.0]
> at
> org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.getTable(WorkspaceSchemaFactory.java:278)
> ~[drill-java-exec-1.3.0.jar:1.3.0
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6816) NPE - Concurrent query execution using PreparedStatement
[
https://issues.apache.org/jira/browse/DRILL-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708041#comment-16708041
]
Khurram Faraaz commented on DRILL-6816:
---
[~vitalii] In the case of, CREATE TABLE AS SELECT * FROM ;
For such a query we will have to use stmt.executeUpdate() method, to execute it
?
> NPE - Concurrent query execution using PreparedStatement
> -
>
> Key: DRILL-6816
> URL: https://issues.apache.org/jira/browse/DRILL-6816
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Vitalii Diravka
>Priority: Major
> Attachments: test_tbl.json
>
>
> Concurrent query execution from JDBC program using PreparedStatement results
> in NPE.
> Queries that were executed concurrently are (part of a query file),
> {noformat}
> select id from `test_tbl.json`
> select count(id) from `test_tbl.json`
> select count(*) from `test_tbl.json`
> select * from `test_tbl.json`
> {noformat}
> Drill 1.14.0
> git.commit.id=35a1ae23c9b280b9e73cb0f6f01808c996515454
> MapR version => 6.1.0.20180911143226.GA (secure cluster)
> JDBC driver used was org.apache.drill.jdbc.Driver
> Executing the above queries concurrently using a Statement object results in
> successful query execution.
> {noformat}
> Statement stmt = conn.createStatement();
> ResultSet rs = stmt.executeQuery(query);
> {noformat}
> However, when the same queries listed above are executed using a
> PreparedStatement object we see an NPE
> {noformat}
> PreparedStatement prdstmnt = conn.prepareStatement(query);
> prdstmnt.executeUpdate();
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 17:04:32.941 [pool-1-thread-3] DEBUG o.a.d.j.impl.DrillStatementRegistry -
> Adding to open-statements registry:
> org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@35757005
> 17:04:32.941 [pool-1-thread-2] DEBUG o.a.d.j.impl.DrillStatementRegistry -
> Adding to open-statements registry:
> org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d4413b8
> 17:04:32.956 [pool-1-thread-1] DEBUG o.a.d.j.impl.DrillStatementRegistry -
> Adding to open-statements registry:
> org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@5eb3b9ab
> 17:04:32.956 [pool-1-thread-4] DEBUG o.a.d.j.impl.DrillStatementRegistry -
> Adding to open-statements registry:
> org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d9367d0
> java.lang.NullPointerException
> at java.util.Objects.requireNonNull(Objects.java:203)
> at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
> at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1143)
> at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1150)
> at
> org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:511)
> at
> org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:146)
> at
> org.apache.drill.jdbc.impl.DrillPreparedStatementImpl.executeLargeUpdate(DrillPreparedStatementImpl.java:512)
> at
> org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:142)
> at RunQuery.executeQuery(RunQuery.java:61)
> at RunQuery.run(RunQuery.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6816) NPE - Concurrent query execution using PreparedStatement
[
https://issues.apache.org/jira/browse/DRILL-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707903#comment-16707903
]
Khurram Faraaz commented on DRILL-6816:
---
thanks [~vitalii] yes I tried and stmt.executeQuery() works just fine.
> NPE - Concurrent query execution using PreparedStatement
> -
>
> Key: DRILL-6816
> URL: https://issues.apache.org/jira/browse/DRILL-6816
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Vitalii Diravka
>Priority: Major
> Attachments: test_tbl.json
>
>
> Concurrent query execution from JDBC program using PreparedStatement results
> in NPE.
> Queries that were executed concurrently are (part of a query file),
> {noformat}
> select id from `test_tbl.json`
> select count(id) from `test_tbl.json`
> select count(*) from `test_tbl.json`
> select * from `test_tbl.json`
> {noformat}
> Drill 1.14.0
> git.commit.id=35a1ae23c9b280b9e73cb0f6f01808c996515454
> MapR version => 6.1.0.20180911143226.GA (secure cluster)
> JDBC driver used was org.apache.drill.jdbc.Driver
> Executing the above queries concurrently using a Statement object results in
> successful query execution.
> {noformat}
> Statement stmt = conn.createStatement();
> ResultSet rs = stmt.executeQuery(query);
> {noformat}
> However, when the same queries listed above are executed using a
> PreparedStatement object we see an NPE
> {noformat}
> PreparedStatement prdstmnt = conn.prepareStatement(query);
> prdstmnt.executeUpdate();
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 17:04:32.941 [pool-1-thread-3] DEBUG o.a.d.j.impl.DrillStatementRegistry -
> Adding to open-statements registry:
> org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@35757005
> 17:04:32.941 [pool-1-thread-2] DEBUG o.a.d.j.impl.DrillStatementRegistry -
> Adding to open-statements registry:
> org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d4413b8
> 17:04:32.956 [pool-1-thread-1] DEBUG o.a.d.j.impl.DrillStatementRegistry -
> Adding to open-statements registry:
> org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@5eb3b9ab
> 17:04:32.956 [pool-1-thread-4] DEBUG o.a.d.j.impl.DrillStatementRegistry -
> Adding to open-statements registry:
> org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d9367d0
> java.lang.NullPointerException
> at java.util.Objects.requireNonNull(Objects.java:203)
> at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
> at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1143)
> at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1150)
> at
> org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:511)
> at
> org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:146)
> at
> org.apache.drill.jdbc.impl.DrillPreparedStatementImpl.executeLargeUpdate(DrillPreparedStatementImpl.java:512)
> at
> org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:142)
> at RunQuery.executeQuery(RunQuery.java:61)
> at RunQuery.run(RunQuery.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6816) NPE - Concurrent query execution using PreparedStatement
[
https://issues.apache.org/jira/browse/DRILL-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690157#comment-16690157
]
Khurram Faraaz commented on DRILL-6816:
---
[~vitalii] I haven't tried with executeQuery method yet, I will run the test
using executeQuery and share results here.
> NPE - Concurrent query execution using PreparedStatement
> -
>
> Key: DRILL-6816
> URL: https://issues.apache.org/jira/browse/DRILL-6816
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Priority: Major
> Attachments: test_tbl.json
>
>
> Concurrent query execution from JDBC program using PreparedStatement results
> in NPE.
> Queries that were executed concurrently are (part of a query file),
> {noformat}
> select id from `test_tbl.json`
> select count(id) from `test_tbl.json`
> select count(*) from `test_tbl.json`
> select * from `test_tbl.json`
> {noformat}
> Drill 1.14.0
> git.commit.id=35a1ae23c9b280b9e73cb0f6f01808c996515454
> MapR version => 6.1.0.20180911143226.GA (secure cluster)
> JDBC driver used was org.apache.drill.jdbc.Driver
> Executing the above queries concurrently using a Statement object results in
> successful query execution.
> {noformat}
> Statement stmt = conn.createStatement();
> ResultSet rs = stmt.executeQuery(query);
> {noformat}
> However, when the same queries listed above are executed using a
> PreparedStatement object we see an NPE
> {noformat}
> PreparedStatement prdstmnt = conn.prepareStatement(query);
> prdstmnt.executeUpdate();
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 17:04:32.941 [pool-1-thread-3] DEBUG o.a.d.j.impl.DrillStatementRegistry -
> Adding to open-statements registry:
> org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@35757005
> 17:04:32.941 [pool-1-thread-2] DEBUG o.a.d.j.impl.DrillStatementRegistry -
> Adding to open-statements registry:
> org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d4413b8
> 17:04:32.956 [pool-1-thread-1] DEBUG o.a.d.j.impl.DrillStatementRegistry -
> Adding to open-statements registry:
> org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@5eb3b9ab
> 17:04:32.956 [pool-1-thread-4] DEBUG o.a.d.j.impl.DrillStatementRegistry -
> Adding to open-statements registry:
> org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d9367d0
> java.lang.NullPointerException
> at java.util.Objects.requireNonNull(Objects.java:203)
> at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
> at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1143)
> at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1150)
> at
> org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:511)
> at
> org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:146)
> at
> org.apache.drill.jdbc.impl.DrillPreparedStatementImpl.executeLargeUpdate(DrillPreparedStatementImpl.java:512)
> at
> org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:142)
> at RunQuery.executeQuery(RunQuery.java:61)
> at RunQuery.run(RunQuery.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6816) NPE - Concurrent query execution using PreparedStatement
[
https://issues.apache.org/jira/browse/DRILL-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670756#comment-16670756
]
Khurram Faraaz commented on DRILL-6816:
---
Also reproducible from a single Thread
Stack trace for NPE
{noformat}
14:14:01.174 [pool-1-thread-1] INFO o.a.drill.exec.client.DrillClient - Foreman
drillbit is qa102-45
14:14:01.174 [pool-1-thread-1] INFO o.a.drill.exec.client.DrillClient -
Successfully connected to server qa102-45:31010
14:14:01.256 [pool-1-thread-1] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@2e35015b
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1143)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1150)
at
org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:511)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:146)
at
org.apache.drill.jdbc.impl.DrillPreparedStatementImpl.executeLargeUpdate(DrillPreparedStatementImpl.java:512)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:142)
at RunQuery.run(RunQuery.java:34)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
null
java.sql.SQLException
at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
at
org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:520)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:146)
at
org.apache.drill.jdbc.impl.DrillPreparedStatementImpl.executeLargeUpdate(DrillPreparedStatementImpl.java:512)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:142)
at RunQuery.run(RunQuery.java:34)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1143)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1150)
at
org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:511)
... 9 more
14:14:01.257 [pool-1-thread-1] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Auto-closing (via open-statements registry):
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@2e35015b
14:14:01.257 [pool-1-thread-1] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Removing from open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@2e35015b
14:14:01.258 [pool-1-thread-1] DEBUG o.apache.drill.exec.rpc.BasicClient -
Closing client
{noformat}
> NPE - Concurrent query execution using PreparedStatement
> -
>
> Key: DRILL-6816
> URL: https://issues.apache.org/jira/browse/DRILL-6816
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Priority: Major
>
> Concurrent query execution from JDBC program using PreparedStatement results
> in NPE.
> Queries that were executed concurrently are (part of a query file),
> {noformat}
> select id from `test_tbl.json`
> select count(id) from `test_tbl.json`
> select count(*) from `test_tbl.json`
> select * from `test_tbl.json`
> {noformat}
> Drill 1.14.0
> git.commit.id=35a1ae23c9b280b9e73cb0f6f01808c996515454
> MapR version => 6.1.0.20180911143226.GA (secure cluster)
> JDBC driver used was org.apache.drill.jdbc.Driver
> Executing the above queries concurrently using a Statement object results in
> successful query execution.
> {noformat}
> Statement stmt = conn.createStatement();
> ResultSet rs = stmt.executeQuery(query);
> {noformat}
> However, when the same queries listed above are executed using a
>
[jira] [Updated] (DRILL-6816) NPE - Concurrent query execution using PreparedStatement
[
https://issues.apache.org/jira/browse/DRILL-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-6816:
--
Description:
Concurrent query execution from JDBC program using PreparedStatement results in
NPE.
Queries that were executed concurrently are (part of a query file),
{noformat}
select id from `test_tbl.json`
select count(id) from `test_tbl.json`
select count(*) from `test_tbl.json`
select * from `test_tbl.json`
{noformat}
Drill 1.14.0
git.commit.id=35a1ae23c9b280b9e73cb0f6f01808c996515454
MapR version => 6.1.0.20180911143226.GA (secure cluster)
JDBC driver used was org.apache.drill.jdbc.Driver
Executing the above queries concurrently using a Statement object results in
successful query execution.
{noformat}
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(query);
{noformat}
However, when the same queries listed above are executed using a
PreparedStatement object we see an NPE
{noformat}
PreparedStatement prdstmnt = conn.prepareStatement(query);
prdstmnt.executeUpdate();
{noformat}
Stack trace from drillbit.log
{noformat}
17:04:32.941 [pool-1-thread-3] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@35757005
17:04:32.941 [pool-1-thread-2] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d4413b8
17:04:32.956 [pool-1-thread-1] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@5eb3b9ab
17:04:32.956 [pool-1-thread-4] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d9367d0
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1143)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1150)
at
org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:511)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:146)
at
org.apache.drill.jdbc.impl.DrillPreparedStatementImpl.executeLargeUpdate(DrillPreparedStatementImpl.java:512)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:142)
at RunQuery.executeQuery(RunQuery.java:61)
at RunQuery.run(RunQuery.java:30)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
was:
Concurrent query execution from JDBC program using PreparedStatement results in
NPE.
Queries that were executed concurrently are (part of a query file),
{noformat}
select id from `test_tbl.json`
select count(id) from `test_tbl.json`
select count(*) from `test_tbl.json`
select * from `test_tbl.json`
{noformat}
Drill 1.14.0
git.commit.id=35a1ae23c9b280b9e73cb0f6f01808c996515454
MapR version => 6.1.0.20180911143226.GA (secure cluster)
Executing the above queries concurrently using a Statement object results in
successful query execution.
{noformat}
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(query);
{noformat}
However, when the same queries listed above are executed using a
PreparedStatement object we see an NPE
{noformat}
PreparedStatement prdstmnt = conn.prepareStatement(query);
prdstmnt.executeUpdate();
{noformat}
Stack trace from drillbit.log
{noformat}
17:04:32.941 [pool-1-thread-3] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@35757005
17:04:32.941 [pool-1-thread-2] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d4413b8
17:04:32.956 [pool-1-thread-1] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@5eb3b9ab
17:04:32.956 [pool-1-thread-4] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d9367d0
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
[jira] [Updated] (DRILL-6816) NPE - Concurrent query execution using PreparedStatement
[
https://issues.apache.org/jira/browse/DRILL-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-6816:
--
Description:
Concurrent query execution from JDBC program using PreparedStatement results in
NPE.
Queries that were executed concurrently are (part of a query file),
{noformat}
select id from `test_tbl.json`
select count(id) from `test_tbl.json`
select count(*) from `test_tbl.json`
select * from `test_tbl.json`
{noformat}
Drill 1.14.0
git.commit.id=35a1ae23c9b280b9e73cb0f6f01808c996515454
MapR version => 6.1.0.20180911143226.GA (secure cluster)
Executing the above queries concurrently using a Statement object results in
successful query execution.
{noformat}
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(query);
{noformat}
However, when the same queries listed above are executed using a
PreparedStatement object we see an NPE
{noformat}
PreparedStatement prdstmnt = conn.prepareStatement(query);
prdstmnt.executeUpdate();
{noformat}
Stack trace from drillbit.log
{noformat}
17:04:32.941 [pool-1-thread-3] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@35757005
17:04:32.941 [pool-1-thread-2] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d4413b8
17:04:32.956 [pool-1-thread-1] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@5eb3b9ab
17:04:32.956 [pool-1-thread-4] DEBUG o.a.d.j.impl.DrillStatementRegistry -
Adding to open-statements registry:
org.apache.drill.jdbc.impl.DrillJdbc41Factory$DrillJdbc41PreparedStatement@d9367d0
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1143)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1150)
at
org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:511)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:146)
at
org.apache.drill.jdbc.impl.DrillPreparedStatementImpl.executeLargeUpdate(DrillPreparedStatementImpl.java:512)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:142)
at RunQuery.executeQuery(RunQuery.java:61)
at RunQuery.run(RunQuery.java:30)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
was:
Concurrent query execution from JDBC program using PreparedStatement results in
NPE.
Queries that were executed concurrently are (part of a query file),
{noformat}
select id from `test_tbl.json`
select count(id) from `test_tbl.json`
select count(*) from `test_tbl.json`
select * from `test_tbl.json`
{noformat}
Drill 1.14.0
git.commit.id=35a1ae23c9b280b9e73cb0f6f01808c996515454
MapR version => 6.1.0.20180911143226.GA (secure cluster)
Executing the above queries concurrently using a Statement object results in
successful query execution.
{noformat}
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(query);
{noformat}
However, when the same queries listed above are executed using a
PreparedStatement object we see an NPE
{noformat}
PreparedStatement prdstmnt = conn.prepareStatement(query);
prdstmnt.executeUpdate();
{noformat}
Stack trace from drillbit.log
{noformat}
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1143)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1150)
at
org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:511)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:146)
at
org.apache.drill.jdbc.impl.DrillPreparedStatementImpl.executeLargeUpdate(DrillPreparedStatementImpl.java:512)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:142)
at RunQuery.executeQuery(RunQuery.java:61)
at RunQuery.run(RunQuery.java:30)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
[jira] [Created] (DRILL-6816) NPE - Concurrent query execution using PreparedStatement
Khurram Faraaz created DRILL-6816:
-
Summary: NPE - Concurrent query execution using PreparedStatement
Key: DRILL-6816
URL: https://issues.apache.org/jira/browse/DRILL-6816
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Concurrent query execution from JDBC program using PreparedStatement results in
NPE.
Queries that were executed concurrently are (part of a query file),
{noformat}
select id from `test_tbl.json`
select count(id) from `test_tbl.json`
select count(*) from `test_tbl.json`
select * from `test_tbl.json`
{noformat}
Drill 1.14.0
git.commit.id=35a1ae23c9b280b9e73cb0f6f01808c996515454
MapR version => 6.1.0.20180911143226.GA (secure cluster)
Executing the above queries concurrently using a Statement object results in
successful query execution.
{noformat}
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(query);
{noformat}
However, when the same queries listed above are executed using a
PreparedStatement object we see an NPE
{noformat}
PreparedStatement prdstmnt = conn.prepareStatement(query);
prdstmnt.executeUpdate();
{noformat}
Stack trace from drillbit.log
{noformat}
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at org.apache.calcite.avatica.Meta$MetaResultSet.create(Meta.java:577)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1143)
at org.apache.drill.jdbc.impl.DrillMetaImpl.execute(DrillMetaImpl.java:1150)
at
org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:511)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:146)
at
org.apache.drill.jdbc.impl.DrillPreparedStatementImpl.executeLargeUpdate(DrillPreparedStatementImpl.java:512)
at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:142)
at RunQuery.executeQuery(RunQuery.java:61)
at RunQuery.run(RunQuery.java:30)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DRILL-6563) TPCDS query 10 has regressed
[ https://issues.apache.org/jira/browse/DRILL-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Khurram Faraaz closed DRILL-6563. - verified. > TPCDS query 10 has regressed > - > > Key: DRILL-6563 > URL: https://issues.apache.org/jira/browse/DRILL-6563 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning Optimization >Affects Versions: 1.14.0 >Reporter: Khurram Faraaz >Assignee: Pritesh Maker >Priority: Major > Fix For: 1.15.0 > > Attachments: 24ca3c6c-90e1-a4bf-6c6f-3f981fa2d043.sys.drill, > query10.fast_plan_old_commit, tpcds_query_10_plan_slow_140d09e.pdf, > tpcds_query_plan_10_140d09e.txt > > > TPC-DS query 10 has regressed in performance from taking 3.5 seconds to > execute on Apache Drill 1.14.0 commit b92f599 , to 07 min 51.851 sec to > complete execution on Apache Drill 1.14.0 commit 140d09e. Query was executed > over SF1 parquet views on a 4 node cluster. > Query plan from old and newer commit is attached here, with the query profile > from newer commit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6563) TPCDS query 10 has regressed
[ https://issues.apache.org/jira/browse/DRILL-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595405#comment-16595405 ] Khurram Faraaz commented on DRILL-6563: --- Verified fix on Apache Drill 1.15.0 commit 8ddc9d7. TPC-DS query 10 completes execution in about five seconds, on a 4 node cluster against SF1 parquet views data. > TPCDS query 10 has regressed > - > > Key: DRILL-6563 > URL: https://issues.apache.org/jira/browse/DRILL-6563 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning Optimization >Affects Versions: 1.14.0 >Reporter: Khurram Faraaz >Assignee: Pritesh Maker >Priority: Major > Fix For: 1.15.0 > > Attachments: 24ca3c6c-90e1-a4bf-6c6f-3f981fa2d043.sys.drill, > query10.fast_plan_old_commit, tpcds_query_10_plan_slow_140d09e.pdf, > tpcds_query_plan_10_140d09e.txt > > > TPC-DS query 10 has regressed in performance from taking 3.5 seconds to > execute on Apache Drill 1.14.0 commit b92f599 , to 07 min 51.851 sec to > complete execution on Apache Drill 1.14.0 commit 140d09e. Query was executed > over SF1 parquet views on a 4 node cluster. > Query plan from old and newer commit is attached here, with the query profile > from newer commit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-6563) TPCDS query 10 has regressed
[ https://issues.apache.org/jira/browse/DRILL-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Khurram Faraaz resolved DRILL-6563. --- Resolution: Fixed > TPCDS query 10 has regressed > - > > Key: DRILL-6563 > URL: https://issues.apache.org/jira/browse/DRILL-6563 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning Optimization >Affects Versions: 1.14.0 >Reporter: Khurram Faraaz >Assignee: Pritesh Maker >Priority: Major > Fix For: 1.15.0 > > Attachments: 24ca3c6c-90e1-a4bf-6c6f-3f981fa2d043.sys.drill, > query10.fast_plan_old_commit, tpcds_query_10_plan_slow_140d09e.pdf, > tpcds_query_plan_10_140d09e.txt > > > TPC-DS query 10 has regressed in performance from taking 3.5 seconds to > execute on Apache Drill 1.14.0 commit b92f599 , to 07 min 51.851 sec to > complete execution on Apache Drill 1.14.0 commit 140d09e. Query was executed > over SF1 parquet views on a 4 node cluster. > Query plan from old and newer commit is attached here, with the query profile > from newer commit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6453) TPC-DS query 72 has regressed
[
https://issues.apache.org/jira/browse/DRILL-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594057#comment-16594057
]
Khurram Faraaz commented on DRILL-6453:
---
Verified that TPC-DS query 72 completes execution and returns results in 4
minutes and 6 seconds on a 4 node cluster, against parquet views SF1 data.
Apache Drill 1.15.0 commit 8ddc9d7
> TPC-DS query 72 has regressed
> -
>
> Key: DRILL-6453
> URL: https://issues.apache.org/jira/browse/DRILL-6453
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Timothy Farkas
>Priority: Blocker
> Labels: ready-to-commit
> Fix For: 1.15.0
>
> Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill,
> jstack_29173_June_10_2018.txt, jstack_29173_June_10_2018.txt,
> jstack_29173_June_10_2018_b.txt, jstack_29173_June_10_2018_b.txt,
> jstack_29173_June_10_2018_c.txt, jstack_29173_June_10_2018_c.txt,
> jstack_29173_June_10_2018_d.txt, jstack_29173_June_10_2018_d.txt,
> jstack_29173_June_10_2018_e.txt, jstack_29173_June_10_2018_e.txt
>
>
> TPC-DS query 72 seems to have regressed, query profile for the case where it
> Canceled after 2 hours on Drill 1.14.0 is attached here.
> {noformat}
> On, Drill 1.14.0-SNAPSHOT
> commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took
> around 55 seconds to execute)
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> TPC-DS query 72 executed successfully & took 47 seconds to complete execution.
> {noformat}
> {noformat}
> TPC-DS data in the below run has date values stored as DATE datatype and not
> VARCHAR type
> On, Drill 1.14.0-SNAPSHOT
> commit : 82e1a12
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> and
> alter system set `exec.hashjoin.num_partitions` = 1;
> TPC-DS query 72 executed for 2 hrs and 11 mins and did not complete, I had to
> Cancel it by stopping the Foreman drillbit.
> As a result several minor fragments are reported to be in
> CANCELLATION_REQUESTED state on UI.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DRILL-6453) TPC-DS query 72 has regressed
[
https://issues.apache.org/jira/browse/DRILL-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz closed DRILL-6453.
-
Verified!
> TPC-DS query 72 has regressed
> -
>
> Key: DRILL-6453
> URL: https://issues.apache.org/jira/browse/DRILL-6453
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Timothy Farkas
>Priority: Blocker
> Labels: ready-to-commit
> Fix For: 1.15.0
>
> Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill,
> jstack_29173_June_10_2018.txt, jstack_29173_June_10_2018.txt,
> jstack_29173_June_10_2018_b.txt, jstack_29173_June_10_2018_b.txt,
> jstack_29173_June_10_2018_c.txt, jstack_29173_June_10_2018_c.txt,
> jstack_29173_June_10_2018_d.txt, jstack_29173_June_10_2018_d.txt,
> jstack_29173_June_10_2018_e.txt, jstack_29173_June_10_2018_e.txt
>
>
> TPC-DS query 72 seems to have regressed, query profile for the case where it
> Canceled after 2 hours on Drill 1.14.0 is attached here.
> {noformat}
> On, Drill 1.14.0-SNAPSHOT
> commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took
> around 55 seconds to execute)
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> TPC-DS query 72 executed successfully & took 47 seconds to complete execution.
> {noformat}
> {noformat}
> TPC-DS data in the below run has date values stored as DATE datatype and not
> VARCHAR type
> On, Drill 1.14.0-SNAPSHOT
> commit : 82e1a12
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> and
> alter system set `exec.hashjoin.num_partitions` = 1;
> TPC-DS query 72 executed for 2 hrs and 11 mins and did not complete, I had to
> Cancel it by stopping the Foreman drillbit.
> As a result several minor fragments are reported to be in
> CANCELLATION_REQUESTED state on UI.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6453) TPC-DS query 72 has regressed
[
https://issues.apache.org/jira/browse/DRILL-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543613#comment-16543613
]
Khurram Faraaz commented on DRILL-6453:
---
Results of executing the simplified query with the first three joins starting
from the leaf level in the plan, of TPC-DS query 72.
Total time taken for below query to complete was 07 min 46.719 sec
{noformat}
SELECT
Count(*) total_cnt
FROM catalog_sales
JOIN inventory
ON ( cs_item_sk = inv_item_sk )
JOIN customer_demographics
ON ( cs_bill_cdemo_sk = cd_demo_sk )
JOIN household_demographics
ON ( cs_bill_hdemo_sk = hd_demo_sk )
WHERE inv_quantity_on_hand < cs_quantity
AND hd_buy_potential = '501-1000'
AND cd_marital_status = 'M'
LIMIT 100
{noformat}
{noformat}
00-00 Screen : rowType = RecordType(BIGINT total_cnt): rowcount = 100.0,
cumulative cost = \{9.7136055E7 rows, 6.08208382E8 cpu, 0.0 io, 9.4473289728E10
network, 3.04611648E7 memory}, id = 2694
00-01 Project(total_cnt=[$0]) : rowType = RecordType(BIGINT total_cnt):
rowcount = 100.0, cumulative cost = \{9.7136045E7 rows, 6.08208372E8 cpu, 0.0
io, 9.4473289728E10 network, 3.04611648E7 memory}, id = 2693
00-02 SelectionVectorRemover : rowType = RecordType(BIGINT total_cnt): rowcount
= 100.0, cumulative cost = \{9.7135945E7 rows, 6.08208272E8 cpu, 0.0 io,
9.4473289728E10 network, 3.04611648E7 memory}, id = 2692
00-03 Limit(fetch=[100]) : rowType = RecordType(BIGINT total_cnt): rowcount =
100.0, cumulative cost = \{9.7135845E7 rows, 6.08208172E8 cpu, 0.0 io,
9.4473289728E10 network, 3.04611648E7 memory}, id = 2691
00-04 StreamAgg(group=[{}], total_cnt=[$SUM0($0)]) : rowType =
RecordType(BIGINT total_cnt): rowcount = 1.0, cumulative cost = \{9.7135745E7
rows, 6.08207772E8 cpu, 0.0 io, 9.4473289728E10 network, 3.04611648E7 memory},
id = 2690
00-05 StreamAgg(group=[{}], total_cnt=[COUNT()]) : rowType = RecordType(BIGINT
total_cnt): rowcount = 1.0, cumulative cost = \{9.7135744E7 rows, 6.0820776E8
cpu, 0.0 io, 9.4473289728E10 network, 3.04611648E7 memory}, id = 2689
00-06 Project($f0=[0]) : rowType = RecordType(INTEGER $f0): rowcount =
5872500.0, cumulative cost = \{9.1263244E7 rows, 5.3773776E8 cpu, 0.0 io,
9.4473289728E10 network, 3.04611648E7 memory}, id = 2688
00-07 HashJoin(condition=[=($0, $1)], joinType=[inner]) : rowType =
RecordType(ANY cs_bill_hdemo_sk, ANY hd_demo_sk): rowcount = 5872500.0,
cumulative cost = \{8.5390744E7 rows, 5.1424776E8 cpu, 0.0 io, 9.4473289728E10
network, 3.04611648E7 memory}, id = 2687
00-09 Project(cs_bill_hdemo_sk=[$1]) : rowType = RecordType(ANY
cs_bill_hdemo_sk): rowcount = 5872500.0, cumulative cost = \{7.9500604E7 rows,
4.4371944E8 cpu, 0.0 io, 9.4473289728E10 network, 3.04421568E7 memory}, id =
2682
00-11 HashJoin(condition=[=($0, $2)], joinType=[inner]) : rowType =
RecordType(ANY cs_bill_cdemo_sk, ANY cs_bill_hdemo_sk, ANY cd_demo_sk):
rowcount = 5872500.0, cumulative cost = \{7.3628104E7 rows, 4.3784694E8 cpu,
0.0 io, 9.4473289728E10 network, 3.04421568E7 memory}, id = 2681
00-14 Project(cs_bill_cdemo_sk=[$0], cs_bill_hdemo_sk=[$1]) : rowType =
RecordType(ANY cs_bill_cdemo_sk, ANY cs_bill_hdemo_sk): rowcount = 5872500.0,
cumulative cost = \{6.3049644E7 rows, 3.5181846E8 cpu, 0.0 io, 9.4473289728E10
network, 2.53712448E7 memory}, id = 2676
00-17 SelectionVectorRemover : rowType = RecordType(ANY cs_bill_cdemo_sk, ANY
cs_bill_hdemo_sk, ANY cs_item_sk, ANY cs_quantity, ANY inv_item_sk, ANY
inv_quantity_on_hand): rowcount = 5872500.0, cumulative cost = \{5.7177144E7
rows, 3.4007346E8 cpu, 0.0 io, 9.4473289728E10 network, 2.53712448E7 memory},
id = 2675
00-19 Filter(condition=[<($5, $3)]) : rowType = RecordType(ANY
cs_bill_cdemo_sk, ANY cs_bill_hdemo_sk, ANY cs_item_sk, ANY cs_quantity, ANY
inv_item_sk, ANY inv_quantity_on_hand): rowcount = 5872500.0, cumulative cost =
\{5.1304644E7 rows, 3.3420096E8 cpu, 0.0 io, 9.4473289728E10 network,
2.53712448E7 memory}, id = 2674
00-21 Project(cs_bill_cdemo_sk=[$2], cs_bill_hdemo_sk=[$3], cs_item_sk=[$4],
cs_quantity=[$5], inv_item_sk=[$0], inv_quantity_on_hand=[$1]) : rowType =
RecordType(ANY cs_bill_cdemo_sk, ANY cs_bill_hdemo_sk, ANY cs_item_sk, ANY
cs_quantity, ANY inv_item_sk, ANY inv_quantity_on_hand): rowcount = 1.1745E7,
cumulative cost = \{3.9559644E7 rows, 2.6373096E8 cpu, 0.0 io, 9.4473289728E10
network, 2.53712448E7 memory}, id = 2673
00-22 HashJoin(condition=[=($4, $0)], joinType=[inner]) : rowType =
RecordType(ANY inv_item_sk, ANY inv_quantity_on_hand, ANY cs_bill_cdemo_sk, ANY
cs_bill_hdemo_sk, ANY cs_item_sk, ANY cs_quantity): rowcount = 1.1745E7,
cumulative cost = \{2.7814644E7 rows, 1.9326096E8 cpu, 0.0 io, 9.4473289728E10
network, 2.53712448E7 memory}, id = 2672
00-24 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=/drill/testdata/tpcds_sf1/parquet/inventory]],
selectionRoot=/drill/testdata/tpcds_sf1/parquet/inventory, numFiles=1,
numRowGroups=1,
[jira] [Commented] (DRILL-6453) TPC-DS query 72 has regressed
[
https://issues.apache.org/jira/browse/DRILL-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543559#comment-16543559
]
Khurram Faraaz commented on DRILL-6453:
---
[~amansinha100] I am working on it, executing the simplified query with the
first three joins starting from the leaf level in the plan.
> TPC-DS query 72 has regressed
> -
>
> Key: DRILL-6453
> URL: https://issues.apache.org/jira/browse/DRILL-6453
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Boaz Ben-Zvi
>Priority: Blocker
> Fix For: 1.14.0
>
> Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill,
> jstack_29173_June_10_2018.txt, jstack_29173_June_10_2018.txt,
> jstack_29173_June_10_2018_b.txt, jstack_29173_June_10_2018_b.txt,
> jstack_29173_June_10_2018_c.txt, jstack_29173_June_10_2018_c.txt,
> jstack_29173_June_10_2018_d.txt, jstack_29173_June_10_2018_d.txt,
> jstack_29173_June_10_2018_e.txt, jstack_29173_June_10_2018_e.txt
>
>
> TPC-DS query 72 seems to have regressed, query profile for the case where it
> Canceled after 2 hours on Drill 1.14.0 is attached here.
> {noformat}
> On, Drill 1.14.0-SNAPSHOT
> commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took
> around 55 seconds to execute)
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> TPC-DS query 72 executed successfully & took 47 seconds to complete execution.
> {noformat}
> {noformat}
> TPC-DS data in the below run has date values stored as DATE datatype and not
> VARCHAR type
> On, Drill 1.14.0-SNAPSHOT
> commit : 82e1a12
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> and
> alter system set `exec.hashjoin.num_partitions` = 1;
> TPC-DS query 72 executed for 2 hrs and 11 mins and did not complete, I had to
> Cancel it by stopping the Foreman drillbit.
> As a result several minor fragments are reported to be in
> CANCELLATION_REQUESTED state on UI.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6595) IllegalAccessError: tried to access field org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files from class org.apache.drill.exec.store.
[
https://issues.apache.org/jira/browse/DRILL-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542110#comment-16542110
]
Khurram Faraaz commented on DRILL-6595:
---
Steps to reproduce are,
Run a query (tpcds query 72) on parquet SF1 data (views) from 4 threads
concurrently and the Exception is written in the drillbit.out file. Ensure that
you have run REFRESH TABLE METADATA command on all TPC-DS parquet tables,
before executing the concurrent queries, on a 4 node cluster.
> IllegalAccessError: tried to access field
> org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files
> from class org.apache.drill.exec.store.parquet.metadata.Metadata_V3
>
>
> Key: DRILL-6595
> URL: https://issues.apache.org/jira/browse/DRILL-6595
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Priority: Minor
>
> java.lang.IllegalAccessError reported in drillbit.out file
> Apache Drill 1.14.0 git.commit.id.abbrev=b0314a3
> {noformat}
> Jul 11, 2018 1:33:39 PM
> com.fasterxml.jackson.module.afterburner.deser.OptimizedSettableBeanProperty
> _reportProblem
> WARNING: Disabling Afterburner deserialization for class
> org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3
> (field #1; mutator
> com.fasterxml.jackson.module.afterburner.deser.SettableObjectFieldProperty),
> due to access error (type java.lang.IllegalAccessError, message=tried to
> access field
> org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files
> from class
> org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer48257508)
> java.lang.IllegalAccessError: tried to access field
> org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files
> from class
> org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer48257508
> at
> org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer48257508.objectField(org/apache/drill/exec/store/parquet/metadata/Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer.java)
> at
> com.fasterxml.jackson.module.afterburner.deser.SettableObjectFieldProperty.deserializeAndSet(SettableObjectFieldProperty.java:50)
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288)
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:189)
> at
> com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:120)
> at
> com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:130)
> at
> com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:97)
> at
> com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:254)
> at
> com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:68)
> at
> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4001)
> at
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3058)
> at
> org.apache.drill.exec.store.parquet.metadata.Metadata.readBlockMeta(Metadata.java:617)
> at
> org.apache.drill.exec.store.parquet.metadata.Metadata.readBlockMeta(Metadata.java:156)
> at
> org.apache.drill.exec.store.parquet.ParquetGroupScan.expandSelectionFromMetadataCache(ParquetGroupScan.java:374)
> at
> org.apache.drill.exec.store.parquet.ParquetGroupScan.expandIfNecessary(ParquetGroupScan.java:337)
> at
> org.apache.drill.exec.store.parquet.ParquetGroupScan.(ParquetGroupScan.java:121)
> at
> org.apache.drill.exec.store.parquet.ParquetGroupScan.(ParquetGroupScan.java:102)
> at
> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(ParquetFormatPlugin.java:180)
> at
> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(ParquetFormatPlugin.java:70)
> at
> org.apache.drill.exec.store.dfs.FileSystemPlugin.getPhysicalScan(FileSystemPlugin.java:136)
> at
> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:114)
> at
> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:109)
> at
> org.apache.drill.exec.planner.logical.DrillTable.getGroupScan(DrillTable.java:99)
> at
>
[jira] [Created] (DRILL-6602) Drill query cancellation should log the originator
Khurram Faraaz created DRILL-6602: - Summary: Drill query cancellation should log the originator Key: DRILL-6602 URL: https://issues.apache.org/jira/browse/DRILL-6602 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Reporter: Khurram Faraaz The Drill query Cancel functionality does not log the origin of the Cancel request. (1). security issue, we don't know who (which user) canceled the query. (2). debugging such a canceled query can be a pain to find the origin and the root cause of why a query was canceled. On Apache Drill 1.14.0 git.commit.id.abbrev=b0314a3, we have observed this problem, and it can be consistently reproduced. There is no information at all about the origin of why the query was canceled in the drillbit.log -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6590) DATA_WRITE ERROR: Hash Join failed to write to output file: /tmp/drill/spill/24bac407
[
https://issues.apache.org/jira/browse/DRILL-6590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541235#comment-16541235
]
Khurram Faraaz commented on DRILL-6590:
---
There is in the stacktrace a java.nio.channels.ClosedByInterruptException:
null, which may explain the reason.
{noformat}
2018-07-11 20:38:03,974 [24b9591c-5ed1-bccf-8512-0b7707559b29:frag:4:14] INFO
o.a.d.e.p.impl.common.HashPartition - User Error Occurred: Hash Join failed to
write to output file:
/tmp/drill/spill/24b9591c-5ed1-bccf-8512-0b7707559b29_HashJoin_4-22-14/spill5_outer
(null)
org.apache.drill.common.exceptions.UserException: DATA_WRITE ERROR: Hash Join
failed to write to output file:
/tmp/drill/spill/24b9591c-5ed1-bccf-8512-0b7707559b29_HashJoin_4-22-14/spill5_outer
[Error Id: 97755089-0ac9-4a9f-9d0d-95a9a996894e ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.common.HashPartition.spillThisPartition(HashPartition.java:350)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.common.HashPartition.completeABatch(HashPartition.java:263)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.common.HashPartition.completeAnOuterBatch(HashPartition.java:237)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.common.HashPartition.appendOuterRow(HashPartition.java:232)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.test.generated.HashJoinProbeGen50.executeProbePhase(HashJoinProbeTemplate.java:306)
[na:na]
at
org.apache.drill.exec.test.generated.HashJoinProbeGen50.probeAndProject(HashJoinProbeTemplate.java:393)
[na:na]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.innerNext(HashJoinBatch.java:348)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.sniffNonEmptyBatch(HashJoinBatch.java:274)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.prefetchFirstBatchFromBothSides(HashJoinBatch.java:236)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.buildSchema(HashJoinBatch.java:216)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:152)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.sniffNonEmptyBatch(HashJoinBatch.java:274)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.prefetchFirstBatchFromBothSides(HashJoinBatch.java:236)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.buildSchema(HashJoinBatch.java:216)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:152)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.sniffNonEmptyBatch(HashJoinBatch.java:274)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
...
...
Caused by: java.nio.channels.ClosedByInterruptException: null
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
~[na:1.8.0_161]
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:216) ~[na:1.8.0_161]
at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) ~[na:1.8.0_161]
at java.nio.channels.Channels.writeFully(Channels.java:101) ~[na:1.8.0_161]
at java.nio.channels.Channels.access$000(Channels.java:61) ~[na:1.8.0_161]
at java.nio.channels.Channels$1.write(Channels.java:174) ~[na:1.8.0_161]
at
com.google.protobuf.CodedOutputStream.refreshBuffer(CodedOutputStream.java:833)
~[protobuf-java-2.5.0.jar:na]
at com.google.protobuf.CodedOutputStream.flush(CodedOutputStream.java:843)
~[protobuf-java-2.5.0.jar:na]
at
com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:91)
~[protobuf-java-2.5.0.jar:na]
at
[jira] [Created] (DRILL-6595) IllegalAccessError: tried to access field org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files from class org.apache.drill.exec.store.pa
Khurram Faraaz created DRILL-6595:
-
Summary: IllegalAccessError: tried to access field
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files
from class org.apache.drill.exec.store.parquet.metadata.Metadata_V3
Key: DRILL-6595
URL: https://issues.apache.org/jira/browse/DRILL-6595
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
java.lang.IllegalAccessError reported in drillbit.out file
Apache Drill 1.14.0 git.commit.id.abbrev=b0314a3
{noformat}
Jul 11, 2018 1:33:39 PM
com.fasterxml.jackson.module.afterburner.deser.OptimizedSettableBeanProperty
_reportProblem
WARNING: Disabling Afterburner deserialization for class
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3
(field #1; mutator
com.fasterxml.jackson.module.afterburner.deser.SettableObjectFieldProperty),
due to access error (type java.lang.IllegalAccessError, message=tried to access
field
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files
from class
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer48257508)
java.lang.IllegalAccessError: tried to access field
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3.files
from class
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer48257508
at
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer48257508.objectField(org/apache/drill/exec/store/parquet/metadata/Metadata_V3$ParquetTableMetadata_v3$Access4JacksonDeserializer.java)
at
com.fasterxml.jackson.module.afterburner.deser.SettableObjectFieldProperty.deserializeAndSet(SettableObjectFieldProperty.java:50)
at
com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288)
at
com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:189)
at
com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:120)
at
com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:130)
at
com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:97)
at
com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:254)
at
com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:68)
at
com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4001)
at
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3058)
at
org.apache.drill.exec.store.parquet.metadata.Metadata.readBlockMeta(Metadata.java:617)
at
org.apache.drill.exec.store.parquet.metadata.Metadata.readBlockMeta(Metadata.java:156)
at
org.apache.drill.exec.store.parquet.ParquetGroupScan.expandSelectionFromMetadataCache(ParquetGroupScan.java:374)
at
org.apache.drill.exec.store.parquet.ParquetGroupScan.expandIfNecessary(ParquetGroupScan.java:337)
at
org.apache.drill.exec.store.parquet.ParquetGroupScan.(ParquetGroupScan.java:121)
at
org.apache.drill.exec.store.parquet.ParquetGroupScan.(ParquetGroupScan.java:102)
at
org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(ParquetFormatPlugin.java:180)
at
org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(ParquetFormatPlugin.java:70)
at
org.apache.drill.exec.store.dfs.FileSystemPlugin.getPhysicalScan(FileSystemPlugin.java:136)
at
org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:114)
at
org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:109)
at
org.apache.drill.exec.planner.logical.DrillTable.getGroupScan(DrillTable.java:99)
at
org.apache.drill.exec.planner.logical.DrillPushProjectIntoScanRule.canPushProjectIntoScan(DrillPushProjectIntoScanRule.java:125)
at
org.apache.drill.exec.planner.logical.DrillPushProjectIntoScanRule.onMatch(DrillPushProjectIntoScanRule.java:81)
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:652)
at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:368)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:426)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:366)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRawDrel(DefaultSqlHandler.java:255)
at
[jira] [Created] (DRILL-6590) DATA_WRITE ERROR: Hash Join failed to write to output file: /tmp/drill/spill/24bac407
Khurram Faraaz created DRILL-6590:
-
Summary: DATA_WRITE ERROR: Hash Join failed to write to output
file: /tmp/drill/spill/24bac407
Key: DRILL-6590
URL: https://issues.apache.org/jira/browse/DRILL-6590
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Apache Drill 1.14.0 git.commit.id.abbrev=eb946b0
There was enough space on /tmp, however Hash Join failed to write to spill file
[test@qa102-45 drill-1.14.0]# clush -a df -h /tmp
: Filesystem Size Used Avail Use% Mounted on
: /dev/mapper/vg_root-lv_root 500G 150G 351G 30% /
: Filesystem Size Used Avail Use% Mounted on
: /dev/mapper/vg_root-lv_root 500G 17G 484G 4% /
: Filesystem Size Used Avail Use% Mounted on
: /dev/mapper/vg_root-lv_root 500G 14G 487G 3% /
: Filesystem Size Used Avail Use% Mounted on
: /dev/mapper/vg_root-lv_root 500G 13G 488G 3% /
Stack trace from drillbit.log
{noformat}
2018-07-10 18:17:51,953 [BitServer-10] WARN o.a.d.exec.rpc.control.WorkEventBus
- A fragment message arrived but there was no registered listener for that
message: profile {
state: FAILED
error {
error_id: "6e258de2-2d4f-4b48-967d-df1b329955cd"
endpoint {
address: "qa102-48.qa.lab"
user_port: 31010
control_port: 31011
data_port: 31012
version: "1.14.0-SNAPSHOT"
state: STARTUP
}
error_type: DATA_WRITE
message: "DATA_WRITE ERROR: Hash Join failed to write to output file:
/tmp/drill/spill/24bac407-2adb-5763-ed08-cb5714dca2c0_HashJoin_4-22-53/spill15_outer\n\nFragment
4:53\n\n[Error Id: 6e258de2-2d4f-4b48-967d-df1b329955cd on
qa102-48.qa.lab:31010]"
exception {
exception_class: "java.nio.channels.ClosedByInterruptException"
stack_trace {
class_name: "..."
line_number: 0
method_name: "..."
is_native_method: false
}
stack_trace {
class_name: "com.google.protobuf.CodedOutputStream"
file_name: "CodedOutputStream.java"
line_number: 833
method_name: "refreshBuffer"
is_native_method: false
}
stack_trace {
class_name: "com.google.protobuf.CodedOutputStream"
file_name: "CodedOutputStream.java"
line_number: 843
method_name: "flush"
is_native_method: false
}
stack_trace {
class_name: "com.google.protobuf.AbstractMessageLite"
file_name: "AbstractMessageLite.java"
line_number: 91
method_name: "writeDelimitedTo"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.cache.VectorSerializer$Writer"
file_name: "VectorSerializer.java"
line_number: 97
method_name: "write"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.common.HashPartition"
file_name: "HashPartition.java"
line_number: 346
method_name: "spillThisPartition"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.common.HashPartition"
file_name: "HashPartition.java"
line_number: 263
method_name: "completeABatch"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.common.HashPartition"
file_name: "HashPartition.java"
line_number: 237
method_name: "completeAnOuterBatch"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.common.HashPartition"
file_name: "HashPartition.java"
line_number: 232
method_name: "appendOuterRow"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.test.generated.HashJoinProbeGen49"
file_name: "HashJoinProbeTemplate.java"
line_number: 306
method_name: "executeProbePhase"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.test.generated.HashJoinProbeGen49"
file_name: "HashJoinProbeTemplate.java"
line_number: 393
method_name: "probeAndProject"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.join.HashJoinBatch"
file_name: "HashJoinBatch.java"
line_number: 357
method_name: "innerNext"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.record.AbstractRecordBatch"
file_name: "AbstractRecordBatch.java"
line_number: 172
method_name: "next"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.record.AbstractRecordBatch"
file_name: "AbstractRecordBatch.java"
line_number: 119
method_name: "next"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.join.HashJoinBatch"
file_name: "HashJoinBatch.java"
line_number: 274
method_name: "sniffNonEmptyBatch"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.join.HashJoinBatch"
file_name: "HashJoinBatch.java"
line_number: 236
method_name: "prefetchFirstBatchFromBothSides"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.join.HashJoinBatch"
file_name: "HashJoinBatch.java"
line_number: 216
method_name: "buildSchema"
is_native_method: false
[jira] [Comment Edited] (DRILL-6453) TPC-DS query 72 has regressed
[
https://issues.apache.org/jira/browse/DRILL-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16539436#comment-16539436
]
Khurram Faraaz edited comment on DRILL-6453 at 7/11/18 1:38 AM:
Query 72 fails (is marked as Canceled) , it cancels it self after running for
2hrs and 11 mins.
Drill 1.14.0 git.commit.id.abbrev=eb946b0
jstack is attached here, from few mins just before the query canceled itself
and after it was canceled.[^jstack_29173_June_10_2018_b.txt]
was (Author: khfaraaz):
Query 72 fails (is marked as Canceled) , it cancels it self after running for
2hrs and 11 mins.
jstack is attached here, from few mins just before the query canceled itself
and after it was canceled.[^jstack_29173_June_10_2018_b.txt]
> TPC-DS query 72 has regressed
> -
>
> Key: DRILL-6453
> URL: https://issues.apache.org/jira/browse/DRILL-6453
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Boaz Ben-Zvi
>Priority: Blocker
> Fix For: 1.14.0
>
> Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill,
> jstack_29173_June_10_2018.txt, jstack_29173_June_10_2018.txt,
> jstack_29173_June_10_2018_b.txt, jstack_29173_June_10_2018_b.txt,
> jstack_29173_June_10_2018_c.txt, jstack_29173_June_10_2018_c.txt,
> jstack_29173_June_10_2018_d.txt, jstack_29173_June_10_2018_d.txt,
> jstack_29173_June_10_2018_e.txt, jstack_29173_June_10_2018_e.txt
>
>
> TPC-DS query 72 seems to have regressed, query profile for the case where it
> Canceled after 2 hours on Drill 1.14.0 is attached here.
> {noformat}
> On, Drill 1.14.0-SNAPSHOT
> commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took
> around 55 seconds to execute)
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> TPC-DS query 72 executed successfully & took 47 seconds to complete execution.
> {noformat}
> {noformat}
> TPC-DS data in the below run has date values stored as DATE datatype and not
> VARCHAR type
> On, Drill 1.14.0-SNAPSHOT
> commit : 82e1a12
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> and
> alter system set `exec.hashjoin.num_partitions` = 1;
> TPC-DS query 72 executed for 2 hrs and 11 mins and did not complete, I had to
> Cancel it by stopping the Foreman drillbit.
> As a result several minor fragments are reported to be in
> CANCELLATION_REQUESTED state on UI.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6453) TPC-DS query 72 has regressed
[
https://issues.apache.org/jira/browse/DRILL-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-6453:
--
Attachment: jstack_29173_June_10_2018.txt
jstack_29173_June_10_2018_e.txt
jstack_29173_June_10_2018_d.txt
jstack_29173_June_10_2018_c.txt
jstack_29173_June_10_2018_b.txt
> TPC-DS query 72 has regressed
> -
>
> Key: DRILL-6453
> URL: https://issues.apache.org/jira/browse/DRILL-6453
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Boaz Ben-Zvi
>Priority: Blocker
> Fix For: 1.14.0
>
> Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill,
> jstack_29173_June_10_2018.txt, jstack_29173_June_10_2018.txt,
> jstack_29173_June_10_2018_b.txt, jstack_29173_June_10_2018_b.txt,
> jstack_29173_June_10_2018_c.txt, jstack_29173_June_10_2018_c.txt,
> jstack_29173_June_10_2018_d.txt, jstack_29173_June_10_2018_d.txt,
> jstack_29173_June_10_2018_e.txt, jstack_29173_June_10_2018_e.txt
>
>
> TPC-DS query 72 seems to have regressed, query profile for the case where it
> Canceled after 2 hours on Drill 1.14.0 is attached here.
> {noformat}
> On, Drill 1.14.0-SNAPSHOT
> commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took
> around 55 seconds to execute)
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> TPC-DS query 72 executed successfully & took 47 seconds to complete execution.
> {noformat}
> {noformat}
> TPC-DS data in the below run has date values stored as DATE datatype and not
> VARCHAR type
> On, Drill 1.14.0-SNAPSHOT
> commit : 82e1a12
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> and
> alter system set `exec.hashjoin.num_partitions` = 1;
> TPC-DS query 72 executed for 2 hrs and 11 mins and did not complete, I had to
> Cancel it by stopping the Foreman drillbit.
> As a result several minor fragments are reported to be in
> CANCELLATION_REQUESTED state on UI.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6453) TPC-DS query 72 has regressed
[
https://issues.apache.org/jira/browse/DRILL-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16539436#comment-16539436
]
Khurram Faraaz commented on DRILL-6453:
---
Query 72 fails (is marked as Canceled) , it cancels it self after running for
2hrs and 11 mins.
jstack is attached here, from few mins just before the query canceled itself
and after it was canceled.[^jstack_29173_June_10_2018_b.txt]
> TPC-DS query 72 has regressed
> -
>
> Key: DRILL-6453
> URL: https://issues.apache.org/jira/browse/DRILL-6453
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Boaz Ben-Zvi
>Priority: Blocker
> Fix For: 1.14.0
>
> Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill,
> jstack_29173_June_10_2018.txt, jstack_29173_June_10_2018_b.txt,
> jstack_29173_June_10_2018_c.txt, jstack_29173_June_10_2018_d.txt,
> jstack_29173_June_10_2018_e.txt
>
>
> TPC-DS query 72 seems to have regressed, query profile for the case where it
> Canceled after 2 hours on Drill 1.14.0 is attached here.
> {noformat}
> On, Drill 1.14.0-SNAPSHOT
> commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took
> around 55 seconds to execute)
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> TPC-DS query 72 executed successfully & took 47 seconds to complete execution.
> {noformat}
> {noformat}
> TPC-DS data in the below run has date values stored as DATE datatype and not
> VARCHAR type
> On, Drill 1.14.0-SNAPSHOT
> commit : 82e1a12
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> and
> alter system set `exec.hashjoin.num_partitions` = 1;
> TPC-DS query 72 executed for 2 hrs and 11 mins and did not complete, I had to
> Cancel it by stopping the Foreman drillbit.
> As a result several minor fragments are reported to be in
> CANCELLATION_REQUESTED state on UI.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6453) TPC-DS query 72 has regressed
[
https://issues.apache.org/jira/browse/DRILL-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-6453:
--
Attachment: jstack_29173_June_10_2018.txt
jstack_29173_June_10_2018_e.txt
jstack_29173_June_10_2018_d.txt
jstack_29173_June_10_2018_c.txt
jstack_29173_June_10_2018_b.txt
> TPC-DS query 72 has regressed
> -
>
> Key: DRILL-6453
> URL: https://issues.apache.org/jira/browse/DRILL-6453
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Boaz Ben-Zvi
>Priority: Blocker
> Fix For: 1.14.0
>
> Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill,
> jstack_29173_June_10_2018.txt, jstack_29173_June_10_2018_b.txt,
> jstack_29173_June_10_2018_c.txt, jstack_29173_June_10_2018_d.txt,
> jstack_29173_June_10_2018_e.txt
>
>
> TPC-DS query 72 seems to have regressed, query profile for the case where it
> Canceled after 2 hours on Drill 1.14.0 is attached here.
> {noformat}
> On, Drill 1.14.0-SNAPSHOT
> commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took
> around 55 seconds to execute)
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> TPC-DS query 72 executed successfully & took 47 seconds to complete execution.
> {noformat}
> {noformat}
> TPC-DS data in the below run has date values stored as DATE datatype and not
> VARCHAR type
> On, Drill 1.14.0-SNAPSHOT
> commit : 82e1a12
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> and
> alter system set `exec.hashjoin.num_partitions` = 1;
> TPC-DS query 72 executed for 2 hrs and 11 mins and did not complete, I had to
> Cancel it by stopping the Foreman drillbit.
> As a result several minor fragments are reported to be in
> CANCELLATION_REQUESTED state on UI.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (DRILL-6583) UI usability issue
[ https://issues.apache.org/jira/browse/DRILL-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Khurram Faraaz reassigned DRILL-6583: - Assignee: Kunal Khatua > UI usability issue > -- > > Key: DRILL-6583 > URL: https://issues.apache.org/jira/browse/DRILL-6583 > Project: Apache Drill > Issue Type: Bug > Components: Web Server >Affects Versions: 1.14.0 >Reporter: Khurram Faraaz >Assignee: Kunal Khatua >Priority: Minor > Attachments: UI_usability_issue_AD_1_14_0.png > > > When a query is under execution, on the web UI we see this text which is > actually a set of different links that help navigate to different pages on > the UI, below that query's profile. > Apache Drill 1.14.0 > git.commit.id.abbrev=f481a7c > They all appear on a single line with no spacing and a typo, the formatting > of the text for those links needs to be changed / improved. > Attached is a screenshot for the issue, look for "FirstPrevious1NextLast" on > the bottom left of the screenshot. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6583) UI usability issue
Khurram Faraaz created DRILL-6583: - Summary: UI usability issue Key: DRILL-6583 URL: https://issues.apache.org/jira/browse/DRILL-6583 Project: Apache Drill Issue Type: Bug Components: Web Server Affects Versions: 1.14.0 Reporter: Khurram Faraaz Attachments: UI_usability_issue_AD_1_14_0.png When a query is under execution, on the web UI we see this text which is actually a set of different links that help navigate to different pages on the UI, below that query's profile. Apache Drill 1.14.0 git.commit.id.abbrev=f481a7c They all appear on a single line with no spacing and a typo, the formatting of the text for those links needs to be changed / improved. Attached is a screenshot for the issue, look for "FirstPrevious1NextLast" on the bottom left of the screenshot. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6517) IllegalStateException: Record count not set for this vector container
[
https://issues.apache.org/jira/browse/DRILL-6517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532224#comment-16532224
]
Khurram Faraaz commented on DRILL-6517:
---
[~sachouche]
The easiest way to repro this Exception is, run TPC-DS query 72 on a 4 node
cluster (physical machines) and after few mins ( > 5 mins), Cancel the query
from the Web UI.
Click on the query on the profiles tab while it is under execution, then click
on Edit Query tab, and then choose Cancel option, to Cancel the query. You will
notice that the query is Canceled and in the drillbit.log you will see the
Exception, "IllegalStateException: Record count not set for this vector
container".
> IllegalStateException: Record count not set for this vector container
> -
>
> Key: DRILL-6517
> URL: https://issues.apache.org/jira/browse/DRILL-6517
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: salim achouche
>Priority: Critical
> Fix For: 1.14.0
>
> Attachments: 24d7b377-7589-7928-f34f-57d02061acef.sys.drill
>
>
> TPC-DS query is Canceled after 2 hrs and 47 mins and we see an
> IllegalStateException: Record count not set for this vector container, in
> drillbit.log
> Steps to reproduce the problem, query profile
> (24d7b377-7589-7928-f34f-57d02061acef) is attached here.
> {noformat}
> In drill-env.sh set max direct memory to 12G on all 4 nodes in cluster
> export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-"12G"}
> and set these options from sqlline,
> alter system set `planner.memory.max_query_memory_per_node` = 10737418240;
> alter system set `drill.exec.hashagg.fallback.enabled` = true;
> To run the query (replace IP-ADDRESS with your foreman node's IP address)
> cd /opt/mapr/drill/drill-1.14.0/bin
> ./sqlline -u
> "jdbc:drill:schema=dfs.tpcds_sf1_parquet_views;drillbit=" -f
> /root/query72.sql
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2018-06-18 20:08:51,912 [24d7b377-7589-7928-f34f-57d02061acef:frag:4:49]
> ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
> IllegalStateException: Record count not set for this vector container
> Fragment 4:49
> [Error Id: 73177a1c-f7aa-4c9e-99e1-d6e1280e3f27 on qa102-45.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalStateException: Record count not set for this vector container
> Fragment 4:49
> [Error Id: 73177a1c-f7aa-4c9e-99e1-d6e1280e3f27 on qa102-45.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361)
> [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216)
> [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327)
> [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [na:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
> Caused by: java.lang.IllegalStateException: Record count not set for this
> vector container
> at com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> ~[guava-18.0.jar:na]
> at
> org.apache.drill.exec.record.VectorContainer.getRecordCount(VectorContainer.java:394)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.getRecordCount(RemovingRecordBatch.java:49)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.RecordBatchSizer.(RecordBatchSizer.java:690)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.RecordBatchSizer.(RecordBatchSizer.java:662)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.JoinBatchMemoryManager.update(JoinBatchMemoryManager.java:73)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.JoinBatchMemoryManager.update(JoinBatchMemoryManager.java:79)
> ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
>
[jira] [Commented] (DRILL-6453) TPC-DS query 72 has regressed
[
https://issues.apache.org/jira/browse/DRILL-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531980#comment-16531980
]
Khurram Faraaz commented on DRILL-6453:
---
[~ben-zvi] [~priteshm]
Apache Drill 1.14.0 on a 4 node cluster, TPC-DS query 72 fails (in Canceled
state) after running for 2hrs and 11 mins
and we see the same Exception as before towards the end of the drillbit.log
file.
git.commit.id.abbrev=f481a7c
{noformat}
message: "SYSTEM ERROR: IllegalStateException: Record count not set for this
vector container\n\nFragment 4:87\n\n[Error Id:
ed305d45-742f-48df-b1ad-6813bb5fdfc4 on qa102-48.qa.lab:31010]"
exception {
exception_class: "java.lang.IllegalStateException"
message: "Record count not set for this vector container"
stack_trace {
class_name: "com.google.common.base.Preconditions"
file_name: "Preconditions.java"
line_number: 173
method_name: "checkState"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.record.VectorContainer"
file_name: "VectorContainer.java"
line_number: 394
method_name: "getRecordCount"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch"
file_name: "RemovingRecordBatch.java"
line_number: 49
method_name: "getRecordCount"
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.record.RecordBatchSizer"
file_name: "RecordBatchSizer.java"
line_number: 714
method_name: ""
is_native_method: false
}
stack_trace {
class_name: "org.apache.drill.exec.record.RecordBatchSizer"
file_name: "RecordBatchSizer.java"
line_number: 686
method_name: ""
is_native_method: false
}
stack_trace {
class_name{ "org.apache.drill.exec.record.JoinBatchMemoryManager"
file_name: "JoinBatchMemoryManager.java"
line_number: 74
method_name: "update"
{noformat}
> TPC-DS query 72 has regressed
> -
>
> Key: DRILL-6453
> URL: https://issues.apache.org/jira/browse/DRILL-6453
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Boaz Ben-Zvi
>Priority: Blocker
> Fix For: 1.14.0
>
> Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill
>
>
> TPC-DS query 72 seems to have regressed, query profile for the case where it
> Canceled after 2 hours on Drill 1.14.0 is attached here.
> {noformat}
> On, Drill 1.14.0-SNAPSHOT
> commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took
> around 55 seconds to execute)
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> TPC-DS query 72 executed successfully & took 47 seconds to complete execution.
> {noformat}
> {noformat}
> TPC-DS data in the below run has date values stored as DATE datatype and not
> VARCHAR type
> On, Drill 1.14.0-SNAPSHOT
> commit : 82e1a12
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> and
> alter system set `exec.hashjoin.num_partitions` = 1;
> TPC-DS query 72 executed for 2 hrs and 11 mins and did not complete, I had to
> Cancel it by stopping the Foreman drillbit.
> As a result several minor fragments are reported to be in
> CANCELLATION_REQUESTED state on UI.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6453) TPC-DS query 72 has regressed
[
https://issues.apache.org/jira/browse/DRILL-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530592#comment-16530592
]
Khurram Faraaz commented on DRILL-6453:
---
[~ben-zvi] [~priteshm] the Exception message is the same.
However, we still need to find out why the query takes so long, over 2 hours to
execute and then fails.
I will re run the test on latest apache master to verify, if this is fixed.
> TPC-DS query 72 has regressed
> -
>
> Key: DRILL-6453
> URL: https://issues.apache.org/jira/browse/DRILL-6453
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Boaz Ben-Zvi
>Priority: Blocker
> Fix For: 1.14.0
>
> Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill
>
>
> TPC-DS query 72 seems to have regressed, query profile for the case where it
> Canceled after 2 hours on Drill 1.14.0 is attached here.
> {noformat}
> On, Drill 1.14.0-SNAPSHOT
> commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took
> around 55 seconds to execute)
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> TPC-DS query 72 executed successfully & took 47 seconds to complete execution.
> {noformat}
> {noformat}
> TPC-DS data in the below run has date values stored as DATE datatype and not
> VARCHAR type
> On, Drill 1.14.0-SNAPSHOT
> commit : 82e1a12
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> and
> alter system set `exec.hashjoin.num_partitions` = 1;
> TPC-DS query 72 executed for 2 hrs and 11 mins and did not complete, I had to
> Cancel it by stopping the Foreman drillbit.
> As a result several minor fragments are reported to be in
> CANCELLATION_REQUESTED state on UI.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6567) Jenkins Regression: TPCDS query 93 fails with INTERNAL_ERROR ERROR: java.lang.reflect.UndeclaredThrowableException.
[
https://issues.apache.org/jira/browse/DRILL-6567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528606#comment-16528606
]
Khurram Faraaz commented on DRILL-6567:
---
[~rhou] the query does not return any results, when executed over SF1 parquet
views (from Drill created parquet tables.)
on Drill 1.14.0 git.commit.id.abbrev=f481a7c
The issue is seen when running the query against Hive created parquet.
{noformat}
apache drill 1.14.0-SNAPSHOT
"the only truly happy people are children, the creative minority and drill
users"
0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> SELECT ss_customer_sk,
. . . . . . . . . . . . . . . . . . . . . . .> Sum(act_sales) sumsales
. . . . . . . . . . . . . . . . . . . . . . .> FROM (SELECT ss_item_sk,
. . . . . . . . . . . . . . . . . . . . . . .> ss_ticket_number,
. . . . . . . . . . . . . . . . . . . . . . .> ss_customer_sk,
. . . . . . . . . . . . . . . . . . . . . . .> CASE
. . . . . . . . . . . . . . . . . . . . . . .> WHEN sr_return_quantity IS NOT
NULL THEN
. . . . . . . . . . . . . . . . . . . . . . .> ( ss_quantity -
sr_return_quantity ) * ss_sales_price
. . . . . . . . . . . . . . . . . . . . . . .> ELSE ( ss_quantity *
ss_sales_price )
. . . . . . . . . . . . . . . . . . . . . . .> END act_sales
. . . . . . . . . . . . . . . . . . . . . . .> FROM store_sales
. . . . . . . . . . . . . . . . . . . . . . .> LEFT OUTER JOIN store_returns
. . . . . . . . . . . . . . . . . . . . . . .> ON ( sr_item_sk = ss_item_sk
. . . . . . . . . . . . . . . . . . . . . . .> AND sr_ticket_number =
ss_ticket_number ),
. . . . . . . . . . . . . . . . . . . . . . .> reason
. . . . . . . . . . . . . . . . . . . . . . .> WHERE sr_reason_sk = r_reason_sk
. . . . . . . . . . . . . . . . . . . . . . .> AND r_reason_desc = 'reason 38')
t
. . . . . . . . . . . . . . . . . . . . . . .> GROUP BY ss_customer_sk
. . . . . . . . . . . . . . . . . . . . . . .> ORDER BY sumsales,
. . . . . . . . . . . . . . . . . . . . . . .> ss_customer_sk
. . . . . . . . . . . . . . . . . . . . . . .> LIMIT 100;
+-+---+
| ss_customer_sk | sumsales |
+-+---+
+-+---+
No rows selected (2.173 seconds){noformat}
> Jenkins Regression: TPCDS query 93 fails with INTERNAL_ERROR ERROR:
> java.lang.reflect.UndeclaredThrowableException.
> ---
>
> Key: DRILL-6567
> URL: https://issues.apache.org/jira/browse/DRILL-6567
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.14.0
>Reporter: Robert Hou
>Assignee: Khurram Faraaz
>Priority: Critical
> Fix For: 1.14.0
>
>
> This is TPCDS Query 93.
> Query:
> /root/drillAutomation/framework-master/framework/resources/Advanced/tpcds/tpcds_sf100/hive/parquet/query93.sql
> SELECT ss_customer_sk,
> Sum(act_sales) sumsales
> FROM (SELECT ss_item_sk,
> ss_ticket_number,
> ss_customer_sk,
> CASE
> WHEN sr_return_quantity IS NOT NULL THEN
> ( ss_quantity - sr_return_quantity ) * ss_sales_price
> ELSE ( ss_quantity * ss_sales_price )
> END act_sales
> FROM store_sales
> LEFT OUTER JOIN store_returns
> ON ( sr_item_sk = ss_item_sk
> AND sr_ticket_number = ss_ticket_number ),
> reason
> WHERE sr_reason_sk = r_reason_sk
> AND r_reason_desc = 'reason 38') t
> GROUP BY ss_customer_sk
> ORDER BY sumsales,
> ss_customer_sk
> LIMIT 100;
> Here is the stack trace:
> 2018-06-29 07:00:32 INFO DrillTestLogger:348 -
> Exception:
> java.sql.SQLException: INTERNAL_ERROR ERROR:
> java.lang.reflect.UndeclaredThrowableException
> Setup failed for null
> Fragment 4:56
> [Error Id: 3c72c14d-9362-4a9b-affb-5cf937bed89e on atsqa6c82.qa.lab:31010]
> (org.apache.drill.common.exceptions.ExecutionSetupException)
> java.lang.reflect.UndeclaredThrowableException
>
> org.apache.drill.common.exceptions.ExecutionSetupException.fromThrowable():30
> org.apache.drill.exec.store.hive.readers.HiveAbstractReader.setup():327
> org.apache.drill.exec.physical.impl.ScanBatch.getNextReaderIfHas():245
> org.apache.drill.exec.physical.impl.ScanBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
>
> org.apache.drill.exec.physical.impl.join.HashJoinBatch.sniffNonEmptyBatch():276
>
> org.apache.drill.exec.physical.impl.join.HashJoinBatch.prefetchFirstBatchFromBothSides():238
> org.apache.drill.exec.physical.impl.join.HashJoinBatch.buildSchema():218
> org.apache.drill.exec.record.AbstractRecordBatch.next():152
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63
>
>
[jira] [Commented] (DRILL-6568) Jenkins Regression: TPCDS query 68 fails with IllegalStateException: Unexpected EMIT outcome received in buildSchema phase
[ https://issues.apache.org/jira/browse/DRILL-6568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528605#comment-16528605 ] Khurram Faraaz commented on DRILL-6568: --- [~rhou] this issue is the same as https://issues.apache.org/jira/browse/DRILL-6548 > Jenkins Regression: TPCDS query 68 fails with IllegalStateException: > Unexpected EMIT outcome received in buildSchema phase > -- > > Key: DRILL-6568 > URL: https://issues.apache.org/jira/browse/DRILL-6568 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.14.0 >Reporter: Robert Hou >Assignee: Khurram Faraaz >Priority: Critical > Fix For: 1.14.0 > > > This is TPCDS Query 68. > Query: > /root/drillAutomation/framework-master/framework/resources/Advanced/tpcds/tpcds_sf1/original/maprdb/json/query68.sql > SELECT c_last_name, > c_first_name, > ca_city, > bought_city, > ss_ticket_number, > extended_price, > extended_tax, > list_price > FROM (SELECT ss_ticket_number, > ss_customer_sk, > ca_city bought_city, > Sum(ss_ext_sales_price) extended_price, > Sum(ss_ext_list_price) list_price, > Sum(ss_ext_tax) extended_tax > FROM store_sales, > date_dim, > store, > household_demographics, > customer_address > WHERE store_sales.ss_sold_date_sk = date_dim.d_date_sk > AND store_sales.ss_store_sk = store.s_store_sk > AND store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk > AND store_sales.ss_addr_sk = customer_address.ca_address_sk > AND date_dim.d_dom BETWEEN 1 AND 2 > AND ( household_demographics.hd_dep_count = 8 > OR household_demographics.hd_vehicle_count = 3 ) > AND date_dim.d_year IN ( 1998, 1998 + 1, 1998 + 2 ) > AND store.s_city IN ( 'Fairview', 'Midway' ) > GROUP BY ss_ticket_number, > ss_customer_sk, > ss_addr_sk, > ca_city) dn, > customer, > customer_address current_addr > WHERE ss_customer_sk = c_customer_sk > AND customer.c_current_addr_sk = current_addr.ca_address_sk > AND current_addr.ca_city <> bought_city > ORDER BY c_last_name, > ss_ticket_number > LIMIT 100; > Here is the stack trace: > 2018-06-29 07:00:32 INFO DrillTestLogger:348 - > Exception: > java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Unexpected EMIT > outcome received in buildSchema phase > Fragment 0:0 > [Error Id: edbe3477-805e-4f1f-8405-d5c194dc28c2 on atsqa6c87.qa.lab:31010] > (java.lang.IllegalStateException) Unexpected EMIT outcome received in > buildSchema phase > org.apache.drill.exec.physical.impl.TopN.TopNBatch.buildSchema():178 > org.apache.drill.exec.record.AbstractRecordBatch.next():152 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63 > org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.innerNext():87 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63 > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():147 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.physical.impl.BaseRootExec.next():103 > > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():83 > org.apache.drill.exec.physical.impl.BaseRootExec.next():93 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():294 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():281 > java.security.AccessController.doPrivileged():-2 > javax.security.auth.Subject.doAs():422 > org.apache.hadoop.security.UserGroupInformation.doAs():1595 > org.apache.drill.exec.work.fragment.FragmentExecutor.run():281 > org.apache.drill.common.SelfCleaningRunnable.run():38 > java.util.concurrent.ThreadPoolExecutor.runWorker():1149 > java.util.concurrent.ThreadPoolExecutor$Worker.run():624 > java.lang.Thread.run():748 > at >
[jira] [Resolved] (DRILL-6568) Jenkins Regression: TPCDS query 68 fails with IllegalStateException: Unexpected EMIT outcome received in buildSchema phase
[ https://issues.apache.org/jira/browse/DRILL-6568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Khurram Faraaz resolved DRILL-6568. --- Resolution: Duplicate > Jenkins Regression: TPCDS query 68 fails with IllegalStateException: > Unexpected EMIT outcome received in buildSchema phase > -- > > Key: DRILL-6568 > URL: https://issues.apache.org/jira/browse/DRILL-6568 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.14.0 >Reporter: Robert Hou >Assignee: Khurram Faraaz >Priority: Critical > Fix For: 1.14.0 > > > This is TPCDS Query 68. > Query: > /root/drillAutomation/framework-master/framework/resources/Advanced/tpcds/tpcds_sf1/original/maprdb/json/query68.sql > SELECT c_last_name, > c_first_name, > ca_city, > bought_city, > ss_ticket_number, > extended_price, > extended_tax, > list_price > FROM (SELECT ss_ticket_number, > ss_customer_sk, > ca_city bought_city, > Sum(ss_ext_sales_price) extended_price, > Sum(ss_ext_list_price) list_price, > Sum(ss_ext_tax) extended_tax > FROM store_sales, > date_dim, > store, > household_demographics, > customer_address > WHERE store_sales.ss_sold_date_sk = date_dim.d_date_sk > AND store_sales.ss_store_sk = store.s_store_sk > AND store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk > AND store_sales.ss_addr_sk = customer_address.ca_address_sk > AND date_dim.d_dom BETWEEN 1 AND 2 > AND ( household_demographics.hd_dep_count = 8 > OR household_demographics.hd_vehicle_count = 3 ) > AND date_dim.d_year IN ( 1998, 1998 + 1, 1998 + 2 ) > AND store.s_city IN ( 'Fairview', 'Midway' ) > GROUP BY ss_ticket_number, > ss_customer_sk, > ss_addr_sk, > ca_city) dn, > customer, > customer_address current_addr > WHERE ss_customer_sk = c_customer_sk > AND customer.c_current_addr_sk = current_addr.ca_address_sk > AND current_addr.ca_city <> bought_city > ORDER BY c_last_name, > ss_ticket_number > LIMIT 100; > Here is the stack trace: > 2018-06-29 07:00:32 INFO DrillTestLogger:348 - > Exception: > java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Unexpected EMIT > outcome received in buildSchema phase > Fragment 0:0 > [Error Id: edbe3477-805e-4f1f-8405-d5c194dc28c2 on atsqa6c87.qa.lab:31010] > (java.lang.IllegalStateException) Unexpected EMIT outcome received in > buildSchema phase > org.apache.drill.exec.physical.impl.TopN.TopNBatch.buildSchema():178 > org.apache.drill.exec.record.AbstractRecordBatch.next():152 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63 > org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.innerNext():87 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63 > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():147 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.physical.impl.BaseRootExec.next():103 > > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():83 > org.apache.drill.exec.physical.impl.BaseRootExec.next():93 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():294 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():281 > java.security.AccessController.doPrivileged():-2 > javax.security.auth.Subject.doAs():422 > org.apache.hadoop.security.UserGroupInformation.doAs():1595 > org.apache.drill.exec.work.fragment.FragmentExecutor.run():281 > org.apache.drill.common.SelfCleaningRunnable.run():38 > java.util.concurrent.ThreadPoolExecutor.runWorker():1149 > java.util.concurrent.ThreadPoolExecutor$Worker.run():624 > java.lang.Thread.run():748 > at > org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:528) > at >
[jira] [Commented] (DRILL-6563) TPCDS query 10 has regressed
[ https://issues.apache.org/jira/browse/DRILL-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528394#comment-16528394 ] Khurram Faraaz commented on DRILL-6563: --- [~ppenumarthy] Executed the same query (query 10) on same data and setup on 4 nodes [https://github.com/ppadma/drill/commit/9a9d3595925341504298396a3279ed5e20da5015] The query took only 3.75 seconds. > TPCDS query 10 has regressed > - > > Key: DRILL-6563 > URL: https://issues.apache.org/jira/browse/DRILL-6563 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning Optimization >Affects Versions: 1.14.0 >Reporter: Khurram Faraaz >Assignee: Pritesh Maker >Priority: Major > Fix For: 1.14.0 > > Attachments: 24ca3c6c-90e1-a4bf-6c6f-3f981fa2d043.sys.drill, > query10.fast_plan_old_commit, tpcds_query_10_plan_slow_140d09e.pdf, > tpcds_query_plan_10_140d09e.txt > > > TPC-DS query 10 has regressed in performance from taking 3.5 seconds to > execute on Apache Drill 1.14.0 commit b92f599 , to 07 min 51.851 sec to > complete execution on Apache Drill 1.14.0 commit 140d09e. Query was executed > over SF1 parquet views on a 4 node cluster. > Query plan from old and newer commit is attached here, with the query profile > from newer commit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6563) TPCDS query 10 has regressed
[ https://issues.apache.org/jira/browse/DRILL-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528381#comment-16528381 ] Khurram Faraaz commented on DRILL-6563: --- Yes, HashJoin is spilling. > TPCDS query 10 has regressed > - > > Key: DRILL-6563 > URL: https://issues.apache.org/jira/browse/DRILL-6563 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning Optimization >Affects Versions: 1.14.0 >Reporter: Khurram Faraaz >Assignee: Pritesh Maker >Priority: Major > Fix For: 1.14.0 > > Attachments: 24ca3c6c-90e1-a4bf-6c6f-3f981fa2d043.sys.drill, > query10.fast_plan_old_commit, tpcds_query_10_plan_slow_140d09e.pdf, > tpcds_query_plan_10_140d09e.txt > > > TPC-DS query 10 has regressed in performance from taking 3.5 seconds to > execute on Apache Drill 1.14.0 commit b92f599 , to 07 min 51.851 sec to > complete execution on Apache Drill 1.14.0 commit 140d09e. Query was executed > over SF1 parquet views on a 4 node cluster. > Query plan from old and newer commit is attached here, with the query profile > from newer commit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6563) TPCDS query 10 has regressed
[ https://issues.apache.org/jira/browse/DRILL-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528349#comment-16528349 ] Khurram Faraaz commented on DRILL-6563: --- HASH_PARTITION_SENDER and HASH_JOIN operators take up most of the time in the query profile, and there are extra BroadcastExchange operators in the (slow) query plan. > TPCDS query 10 has regressed > - > > Key: DRILL-6563 > URL: https://issues.apache.org/jira/browse/DRILL-6563 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning Optimization >Affects Versions: 1.14.0 >Reporter: Khurram Faraaz >Priority: Major > Attachments: 24ca3c6c-90e1-a4bf-6c6f-3f981fa2d043.sys.drill, > query10.fast_plan_old_commit, tpcds_query_10_plan_slow_140d09e.pdf, > tpcds_query_plan_10_140d09e.txt > > > TPC-DS query 10 has regressed in performance from taking 3.5 seconds to > execute on Apache Drill 1.14.0 commit b92f599 , to 07 min 51.851 sec to > complete execution on Apache Drill 1.14.0 commit 140d09e. Query was executed > over SF1 parquet views on a 4 node cluster. > Query plan from old and newer commit is attached here, with the query profile > from newer commit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6563) TPCDS query 10 has regressed
Khurram Faraaz created DRILL-6563: - Summary: TPCDS query 10 has regressed Key: DRILL-6563 URL: https://issues.apache.org/jira/browse/DRILL-6563 Project: Apache Drill Issue Type: Bug Components: Query Planning Optimization Affects Versions: 1.14.0 Reporter: Khurram Faraaz Attachments: 24ca3c6c-90e1-a4bf-6c6f-3f981fa2d043.sys.drill, query10.fast_plan_old_commit, tpcds_query_10_plan_slow_140d09e.pdf, tpcds_query_plan_10_140d09e.txt TPC-DS query 10 has regressed in performance from taking 3.5 seconds to execute on Apache Drill 1.14.0 commit b92f599 , to 07 min 51.851 sec to complete execution on Apache Drill 1.14.0 commit 140d09e. Query was executed over SF1 parquet views on a 4 node cluster. Query plan from old and newer commit is attached here, with the query profile from newer commit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6548) IllegalStateException: Unexpected EMIT outcome received in buildSchema phase
Khurram Faraaz created DRILL-6548:
-
Summary: IllegalStateException: Unexpected EMIT outcome received
in buildSchema phase
Key: DRILL-6548
URL: https://issues.apache.org/jira/browse/DRILL-6548
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Assignee: Sorabh Hamirwasia
On a four node Apache Drill 1.14.0 master branch against TPC-DS SF1 parquet
data (parquet views)
git.commit.id.abbrev=b92f599
TPC-DS query 69 fails with IllegalStateException: Unexpected EMIT outcome
received in buildSchema phase
Failing query is,
{noformat}
2018-06-27 15:24:39,493 [24cbf157-e95c-42ab-7307-f75f5943a277:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query id
24cbf157-e95c-42ab-7307-f75f5943a277: SELECT cd_gender,
cd_marital_status,
cd_education_status,
Count(*) cnt1,
cd_purchase_estimate,
Count(*) cnt2,
cd_credit_rating,
FROM customer c,
customer_address ca,
customer_demographics
WHERE c.c_current_addr_sk = ca.ca_address_sk
AND ca_state IN ( 'KS', 'AZ', 'NE' )
AND cd_demo_sk = c.c_current_cdemo_sk
AND EXISTS (SELECT *
FROM store_sales,
date_dim
WHERE c.c_customer_sk = ss_customer_sk
AND ss_sold_date_sk = d_date_sk
AND d_year = 2004
AND d_moy BETWEEN 3 AND 3 + 2)
AND ( NOT EXISTS (SELECT *
FROM web_sales,
date_dim
WHERE c.c_customer_sk = ws_bill_customer_sk
AND ws_sold_date_sk = d_date_sk
AND d_year = 2004
AND d_moy BETWEEN 3 AND 3 + 2)
AND NOT EXISTS (SELECT *
FROM catalog_sales,
date_dim
WHERE c.c_customer_sk = cs_ship_customer_sk
AND cs_sold_date_sk = d_date_sk
AND d_year = 2004
AND d_moy BETWEEN 3 AND 3 + 2) )
GROUP BY cd_gender,
cd_marital_status,
cd_education_status,
cd_purchase_estimate,
cd_credit_rating
ORDER BY cd_gender,
cd_marital_status,
cd_education_status,
cd_purchase_estimate,
cd_credit_rating
cd_credit_rating
LIMIT 100
{noformat}
Stack trace from drillbit.log
{noformat}
2018-06-27 15:24:42,130 [24cbf157-e95c-42ab-7307-f75f5943a277:frag:0:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException:
Unexpected EMIT outcome received in buildSchema phase
Fragment 0:0
[Error Id: ba1a35e0-807e-4bab-b820-8aa6aad80e87 on qa102-45.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Unexpected EMIT outcome received in buildSchema phase
Fragment 0:0
[Error Id: ba1a35e0-807e-4bab-b820-8aa6aad80e87 on qa102-45.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.lang.IllegalStateException: Unexpected EMIT outcome received in
buildSchema phase
at
org.apache.drill.exec.physical.impl.TopN.TopNBatch.buildSchema(TopNBatch.java:178)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:152)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
[jira] [Commented] (DRILL-6453) TPC-DS query 72 has regressed
[
https://issues.apache.org/jira/browse/DRILL-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524094#comment-16524094
]
Khurram Faraaz commented on DRILL-6453:
---
[~priteshm]
Results of the other test on Apache Drill 1.14.0 commit b92f599, with the two
options set to below values,
alter system set `planner.memory.max_query_memory_per_node` = 8589934592;
alter system set `drill.exec.hashagg.fallback.enabled` = true;
And in drill-env.sh we set direct memory to
export DRILL_MAX_DIRECT_MEMORY=$\{DRILL_MAX_DIRECT_MEMORY:-"12G"}
TPC-DS query 72 is reported to be in Canceled state on UI after having run for
2 hr 11 min 21.958 sec
The last Exception we see in the drillbit.log is this
IllegalStateException: Record count not set for this vector container
> TPC-DS query 72 has regressed
> -
>
> Key: DRILL-6453
> URL: https://issues.apache.org/jira/browse/DRILL-6453
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Boaz Ben-Zvi
>Priority: Blocker
> Fix For: 1.14.0
>
> Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill
>
>
> TPC-DS query 72 seems to have regressed, query profile for the case where it
> Canceled after 2 hours on Drill 1.14.0 is attached here.
> {noformat}
> On, Drill 1.14.0-SNAPSHOT
> commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took
> around 55 seconds to execute)
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> TPC-DS query 72 executed successfully & took 47 seconds to complete execution.
> {noformat}
> {noformat}
> TPC-DS data in the below run has date values stored as DATE datatype and not
> VARCHAR type
> On, Drill 1.14.0-SNAPSHOT
> commit : 82e1a12
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> and
> alter system set `exec.hashjoin.num_partitions` = 1;
> TPC-DS query 72 executed for 2 hrs and 11 mins and did not complete, I had to
> Cancel it by stopping the Foreman drillbit.
> As a result several minor fragments are reported to be in
> CANCELLATION_REQUESTED state on UI.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6539) Record count not set for this vector container error
[ https://issues.apache.org/jira/browse/DRILL-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524076#comment-16524076 ] Khurram Faraaz commented on DRILL-6539: --- [~sachouche] you should try to run the test on a 4 node cluster to hit the issue, since you ran the test a single node (mac) you may not hit it. I can share test cluster details for you to try and repro the issue. > Record count not set for this vector container error > - > > Key: DRILL-6539 > URL: https://issues.apache.org/jira/browse/DRILL-6539 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.13.0 >Reporter: Padma Penumarthy >Assignee: Padma Penumarthy >Priority: Major > Fix For: 1.14.0 > > > This error is randomly seen when executing queries. > [Error Id: 6a2a49e5-28d9-4587-ab8b-5262c07f8fdc on drill196:31010] > (java.lang.IllegalStateException) Record count not set for this vector > container > com.google.common.base.Preconditions.checkState():173 > org.apache.drill.exec.record.VectorContainer.getRecordCount():394 > org.apache.drill.exec.record.RecordBatchSizer.():681 > org.apache.drill.exec.record.RecordBatchSizer.():665 > > org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.getActualSize():441 > > org.apache.drill.exec.physical.impl.common.HashTableTemplate.getActualSize():882 > > org.apache.drill.exec.physical.impl.common.HashTableTemplate.makeDebugString():891 > > org.apache.drill.exec.physical.impl.common.HashPartition.makeDebugString():578 > > org.apache.drill.exec.physical.impl.join.HashJoinBatch.makeDebugString():937 > > org.apache.drill.exec.physical.impl.join.HashJoinBatch.executeBuildPhase():754 > org.apache.drill.exec.physical.impl.join.HashJoinBatch.innerNext():335 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63 > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():137 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63 > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():137 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.test.generated.HashAggregatorGen89497.doWork():617 > org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():176 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.test.generated.HashAggregatorGen89497.doWork():617 > org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():176 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > > org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.loadBatch():403 > > org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.load():354 > > org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.innerNext():299 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63 > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():137 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.physical.impl.BaseRootExec.next():103 > > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():83 > org.apache.drill.exec.physical.impl.BaseRootExec.next():93 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():294 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():281 > java.security.AccessController.doPrivileged():-2 > javax.security.auth.Subject.doAs():422 >
[jira] [Commented] (DRILL-6453) TPC-DS query 72 has regressed
[
https://issues.apache.org/jira/browse/DRILL-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523255#comment-16523255
]
Khurram Faraaz commented on DRILL-6453:
---
[~priteshm] on latest apache master commit b92f599 with DEFAULT values for
system options and default
DRILL_MAX_DIRECT_MEMORY we see that query is in Canceled state after running
for 2 hr 11 mins.
As part of the other test, we will increase DRILL_MAX_DIRECT_MEMORY to 12G and
also set below options and share test results.
alter system set `planner.memory.max_query_memory_per_node` = 10737418240;
alter system set `drill.exec.hashagg.fallback.enabled` = true;
> TPC-DS query 72 has regressed
> -
>
> Key: DRILL-6453
> URL: https://issues.apache.org/jira/browse/DRILL-6453
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Boaz Ben-Zvi
>Priority: Blocker
> Fix For: 1.14.0
>
> Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill
>
>
> TPC-DS query 72 seems to have regressed, query profile for the case where it
> Canceled after 2 hours on Drill 1.14.0 is attached here.
> {noformat}
> On, Drill 1.14.0-SNAPSHOT
> commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took
> around 55 seconds to execute)
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> TPC-DS query 72 executed successfully & took 47 seconds to complete execution.
> {noformat}
> {noformat}
> TPC-DS data in the below run has date values stored as DATE datatype and not
> VARCHAR type
> On, Drill 1.14.0-SNAPSHOT
> commit : 82e1a12
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> and
> alter system set `exec.hashjoin.num_partitions` = 1;
> TPC-DS query 72 executed for 2 hrs and 11 mins and did not complete, I had to
> Cancel it by stopping the Foreman drillbit.
> As a result several minor fragments are reported to be in
> CANCELLATION_REQUESTED state on UI.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6539) Record count not set for this vector container error
[ https://issues.apache.org/jira/browse/DRILL-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523247#comment-16523247 ] Khurram Faraaz commented on DRILL-6539: --- This issue is same as https://issues.apache.org/jira/browse/DRILL-6517 we can consistently reproduce this error when TPC-DS query 72 is executed on latest Apache master 1.14.0 commit b92f599 with DEFAULT values for system options. > Record count not set for this vector container error > - > > Key: DRILL-6539 > URL: https://issues.apache.org/jira/browse/DRILL-6539 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.13.0 >Reporter: Padma Penumarthy >Assignee: Padma Penumarthy >Priority: Major > Fix For: 1.14.0 > > > This error is randomly seen when executing queries. > [Error Id: 6a2a49e5-28d9-4587-ab8b-5262c07f8fdc on drill196:31010] > (java.lang.IllegalStateException) Record count not set for this vector > container > com.google.common.base.Preconditions.checkState():173 > org.apache.drill.exec.record.VectorContainer.getRecordCount():394 > org.apache.drill.exec.record.RecordBatchSizer.():681 > org.apache.drill.exec.record.RecordBatchSizer.():665 > > org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.getActualSize():441 > > org.apache.drill.exec.physical.impl.common.HashTableTemplate.getActualSize():882 > > org.apache.drill.exec.physical.impl.common.HashTableTemplate.makeDebugString():891 > > org.apache.drill.exec.physical.impl.common.HashPartition.makeDebugString():578 > > org.apache.drill.exec.physical.impl.join.HashJoinBatch.makeDebugString():937 > > org.apache.drill.exec.physical.impl.join.HashJoinBatch.executeBuildPhase():754 > org.apache.drill.exec.physical.impl.join.HashJoinBatch.innerNext():335 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63 > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():137 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63 > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():137 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.test.generated.HashAggregatorGen89497.doWork():617 > org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():176 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.test.generated.HashAggregatorGen89497.doWork():617 > org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():176 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > > org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.loadBatch():403 > > org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.load():354 > > org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.innerNext():299 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.record.AbstractRecordBatch.next():119 > org.apache.drill.exec.record.AbstractRecordBatch.next():109 > org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63 > > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():137 > org.apache.drill.exec.record.AbstractRecordBatch.next():172 > org.apache.drill.exec.physical.impl.BaseRootExec.next():103 > > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():83 > org.apache.drill.exec.physical.impl.BaseRootExec.next():93 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():294 > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():281 > java.security.AccessController.doPrivileged():-2 >
[jira] [Commented] (DRILL-6518) DESCRIBE command on Drill created parquet table does not return results
[
https://issues.apache.org/jira/browse/DRILL-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518715#comment-16518715
]
Khurram Faraaz commented on DRILL-6518:
---
SHOW TABLES; command does not return any results. Thanks Vlad for looking into
this.
{noformat}
[test@tqa1102-45 bin]# ./sqlline -u
"jdbc:drill:schema=dfs.tmp;drillbit="
apache drill 1.14.0-SNAPSHOT
"just drill it"
0: jdbc:drill:schema=dfs.tmp> show tables;
+---+-+
| TABLE_SCHEMA | TABLE_NAME |
+---+-+
+---+-+
No rows selected (0.296 seconds)
{noformat}
> DESCRIBE command on Drill created parquet table does not return results
> ---
>
> Key: DRILL-6518
> URL: https://issues.apache.org/jira/browse/DRILL-6518
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Priority: Major
> Attachments: 0_0_0.parquet, item.drill.parquet_metadata
>
>
> Describe command on a Drill (1.14.0) created parquet table, does not return
> the table description.
> Parquet file and parquet metadata cache file for item table is attached here,
> it has the details of column types in the metadata cache file.
> {noformat}
> Apache Drill 1.14.0-SNAPSHOT
> commit : b447260e49dc4a8c906f5b310c037fe6dd77166f
> {noformat}
> {noformat}
> // DESCRIBE commands returns no information about the table.
> 0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> describe
> dfs.`/drill/testdata/tpcds_sf1/parquet/item`;
> +--++--+
> | COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
> +--++--+
> +--++--+
> No rows selected (0.221 seconds)
> 0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> refresh table metadata
> dfs.`/drill/testdata/tpcds_sf1/parquet/item`;
> +---+--+
> | ok | summary |
> +---+--+
> | true | Successfully updated metadata for table
> /drill/testdata/tpcds_sf1/parquet/item. |
> +---+--+
> 1 row selected (0.173 seconds)
> 0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> describe
> dfs.`/drill/testdata/tpcds_sf1/parquet/item`;
> +--++--+
> | COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
> +--++--+
> +--++--+
> No rows selected (0.229 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6518) DESCRIBE command on Drill created parquet table does not return results
Khurram Faraaz created DRILL-6518:
-
Summary: DESCRIBE command on Drill created parquet table does not
return results
Key: DRILL-6518
URL: https://issues.apache.org/jira/browse/DRILL-6518
Project: Apache Drill
Issue Type: Bug
Components: Storage - Parquet
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Attachments: 0_0_0.parquet, item.drill.parquet_metadata
Describe command on a Drill (1.14.0) created parquet table, does not return the
table description.
Parquet file and parquet metadata cache file for item table is attached here,
it has the details of column types in the metadata cache file.
{noformat}
Apache Drill 1.14.0-SNAPSHOT
commit : b447260e49dc4a8c906f5b310c037fe6dd77166f
{noformat}
{noformat}
// DESCRIBE commands returns no information about the table.
0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> describe
dfs.`/drill/testdata/tpcds_sf1/parquet/item`;
+--++--+
| COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
+--++--+
+--++--+
No rows selected (0.221 seconds)
0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> refresh table metadata
dfs.`/drill/testdata/tpcds_sf1/parquet/item`;
+---+--+
| ok | summary |
+---+--+
| true | Successfully updated metadata for table
/drill/testdata/tpcds_sf1/parquet/item. |
+---+--+
1 row selected (0.173 seconds)
0: jdbc:drill:schema=dfs.tpcds_sf1_parquet_vi> describe
dfs.`/drill/testdata/tpcds_sf1/parquet/item`;
+--++--+
| COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
+--++--+
+--++--+
No rows selected (0.229 seconds)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DRILL-6427) outputBatchSize is missing from the DEBUG output for HashJoinBatch operator
[
https://issues.apache.org/jira/browse/DRILL-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz closed DRILL-6427.
-
Verified on latest Apache master Drill 1.14.0.
> outputBatchSize is missing from the DEBUG output for HashJoinBatch operator
>
>
> Key: DRILL-6427
> URL: https://issues.apache.org/jira/browse/DRILL-6427
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Khurram Faraaz
>Assignee: Padma Penumarthy
>Priority: Minor
>
> Drill 1.14.0-SNAPSHOT commit : f99d1f1323c0a5bac99842d6283d3025f3cb527f
> outputBatchSize is missing from the DEBUG output for HashJoinBatch operator
> Query used in test,
> {noformat}
> select count(*) from `twovarchar_asc_128MB.parquet` t1,
> `twovarchar_asc_16MB.parquet` t2 WHERE t1.Varbinaryvalue = t2.Varbinaryvalue
> {noformat}
>
> Snippet from drillbit.log
> {noformat}
> 2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG
> o.a.d.e.p.impl.join.HashJoinBatch - left input: batch count : 1, avg batch
> bytes : 90145920, avg row bytes : 8166, record count : 11040
> 2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG
> o.a.d.e.p.impl.join.HashJoinBatch - right input: batch count : 1, avg batch
> bytes : 10567808, avg row bytes : 7506, record count : 1408
> 2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG
> o.a.d.e.p.impl.join.HashJoinBatch - output: batch count : 166, avg batch
> bytes : 15951453, avg row bytes : 15672, record count : 168960
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6517) IllegalStateException: Record count not set for this vector container
Khurram Faraaz created DRILL-6517:
-
Summary: IllegalStateException: Record count not set for this
vector container
Key: DRILL-6517
URL: https://issues.apache.org/jira/browse/DRILL-6517
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Assignee: Padma Penumarthy
Attachments: 24d7b377-7589-7928-f34f-57d02061acef.sys.drill
TPC-DS query is Canceled after 2 hrs and 47 mins and we see an
IllegalStateException: Record count not set for this vector container, in
drillbit.log
Steps to reproduce the problem, query profile
(24d7b377-7589-7928-f34f-57d02061acef) is attached here.
{noformat}
In drill-env.sh set max direct memory to 12G on all 4 nodes in cluster
export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-"12G"}
and set these options from sqlline,
alter system set `planner.memory.max_query_memory_per_node` = 10737418240;
alter system set `drill.exec.hashagg.fallback.enabled` = true;
To run the query (replace IP-ADDRESS with your foreman node's IP address)
cd /opt/mapr/drill/drill-1.14.0/bin
./sqlline -u
"jdbc:drill:schema=dfs.tpcds_sf1_parquet_views;drillbit=" -f
/root/query72.sql
{noformat}
Stack trace from drillbit.log
{noformat}
2018-06-18 20:08:51,912 [24d7b377-7589-7928-f34f-57d02061acef:frag:4:49] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException:
Record count not set for this vector container
Fragment 4:49
[Error Id: 73177a1c-f7aa-4c9e-99e1-d6e1280e3f27 on qa102-45.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Record count not set for this vector container
Fragment 4:49
[Error Id: 73177a1c-f7aa-4c9e-99e1-d6e1280e3f27 on qa102-45.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.lang.IllegalStateException: Record count not set for this
vector container
at com.google.common.base.Preconditions.checkState(Preconditions.java:173)
~[guava-18.0.jar:na]
at
org.apache.drill.exec.record.VectorContainer.getRecordCount(VectorContainer.java:394)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.getRecordCount(RemovingRecordBatch.java:49)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.RecordBatchSizer.(RecordBatchSizer.java:690)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.RecordBatchSizer.(RecordBatchSizer.java:662)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.JoinBatchMemoryManager.update(JoinBatchMemoryManager.java:73)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.JoinBatchMemoryManager.update(JoinBatchMemoryManager.java:79)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.prefetchFirstBatchFromBothSides(HashJoinBatch.java:242)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.buildSchema(HashJoinBatch.java:218)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:152)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.sniffNonEmptyBatch(HashJoinBatch.java:276)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.prefetchFirstBatchFromBothSides(HashJoinBatch.java:238)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
[jira] [Comment Edited] (DRILL-6453) TPC-DS query 72 has regressed
[
https://issues.apache.org/jira/browse/DRILL-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516497#comment-16516497
]
Khurram Faraaz edited comment on DRILL-6453 at 6/19/18 12:39 AM:
-
[~ben-zvi] [~priteshm] TPC-DS query 72 was executed on commit
b447260e49dc4a8c906f5b310c037fe6dd77166f
Here is a snippet from tail of drillbit.log (on foreman drillbit) after 17 mins
of query remaining in Running state
{noformat}
2018-06-18 17:21:58,163 [24d7b377-7589-7928-f34f-57d02061acef:frag:4:46] WARN
o.a.d.e.p.impl.join.HashJoinBatch - When using the minimum number of partitions
2 we require 76 MB memory but only have 40 MB available. Forcing legacy
behavoir of using unbounded memory in order to prevent regressions.
2018-06-18 17:21:58,163 [24d7b377-7589-7928-f34f-57d02061acef:frag:4:46] WARN
o.a.d.e.p.impl.join.HashJoinBatch - Spilling is disabled - not enough memory
available for internal partitioning. Falling back to use unbounded memory
2018-06-18 17:21:58,289 [24d7b377-7589-7928-f34f-57d02061acef:frag:4:61] WARN
o.a.d.e.p.i.j.HashJoinMemoryCalculator$BuildSidePartitioning - Build and Probe
phases: HashJoin needs to reserve 66381103 bytes of memory but there are only
41943040 bytes available. Using 2 num partitions with 32 initial partitions.
Additional info:
buildBatchSize = 720896
buildNumRecords = 6554
partitionBuildBatchSize = 155250
recordsPerPartitionBatchBuild = 1024
probeBatchSize = 21233664
probeNumRecords = 32767
partitionProbeBatchSize = 887999
recordsPerPartitionBatchProbe = 1024
2018-06-18 17:21:58,290 [24d7b377-7589-7928-f34f-57d02061acef:frag:4:61] WARN
o.a.d.e.p.impl.join.HashJoinBatch - When using the minimum number of partitions
2 we require 76 MB memory but only have 40 MB available. Forcing legacy
behavoir of using unbounded memory in order to prevent regressions.
2018-06-18 17:21:58,290 [24d7b377-7589-7928-f34f-57d02061acef:frag:4:61] WARN
o.a.d.e.p.impl.join.HashJoinBatch - Spilling is disabled - not enough memory
available for internal partitioning. Falling back to use unbounded memory
2018-06-18 17:21:58,494 [24d7b377-7589-7928-f34f-57d02061acef:frag:4:64] WARN
o.a.d.e.p.i.j.HashJoinMemoryCalculator$BuildSidePartitioning - Build and Probe
phases: HashJoin needs to reserve 66381103 bytes of memory but there are only
41943040 bytes available. Using 2 num partitions with 32 initial partitions.
Additional info:
buildBatchSize = 720896
buildNumRecords = 6554
partitionBuildBatchSize = 155250
recordsPerPartitionBatchBuild = 1024
probeBatchSize = 21233664
probeNumRecords = 32767
partitionProbeBatchSize = 887999
recordsPerPartitionBatchProbe = 1024
2018-06-18 17:21:58,494 [24d7b377-7589-7928-f34f-57d02061acef:frag:4:64] WARN
o.a.d.e.p.impl.join.HashJoinBatch - When using the minimum number of partitions
2 we require 76 MB memory but only have 40 MB available. Forcing legacy
behavoir of using unbounded memory in order to prevent regressions.
{noformat}
We do not see the parquet reader error. However, we still need to investigate
why the query takes so long (forever) to complete execution.
{noformat}
In drill-env.sh we set max direct memory to 12G on all 4nodes in cluster
export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-"12G"}
and also set these options from sqlline,
alter system set `planner.memory.max_query_memory_per_node` = 10737418240;
alter system set `drill.exec.hashagg.fallback.enabled` = true;{noformat}
was (Author: khfaraaz):
[~ben-zvi] [~priteshm] TPC-DS query 72 was executed on commit
b447260e49dc4a8c906f5b310c037fe6dd77166f
We do not see the parquet reader error. However, we still need to investigate
why the query takes so long (forever) to complete execution.
{noformat}
In drill-env.sh we set max direct memory to 12G on all 4nodes in cluster
export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-"12G"}
and also set these options from sqlline,
alter system set `planner.memory.max_query_memory_per_node` = 10737418240;
alter system set `drill.exec.hashagg.fallback.enabled` = true;{noformat}
> TPC-DS query 72 has regressed
> -
>
> Key: DRILL-6453
> URL: https://issues.apache.org/jira/browse/DRILL-6453
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Boaz Ben-Zvi
>Priority: Blocker
> Fix For: 1.14.0
>
> Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill
>
>
> TPC-DS query 72 seems to have regressed, query profile for the case where it
> Canceled after 2 hours on Drill 1.14.0 is attached here.
> {noformat}
> On, Drill 1.14.0-SNAPSHOT
> commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took
> around 55 seconds to execute)
> SF1 parquet data on 4 nodes;
>
[jira] [Commented] (DRILL-6453) TPC-DS query 72 has regressed
[
https://issues.apache.org/jira/browse/DRILL-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516497#comment-16516497
]
Khurram Faraaz commented on DRILL-6453:
---
[~ben-zvi] [~priteshm] TPC-DS query 72 was executed on commit
b447260e49dc4a8c906f5b310c037fe6dd77166f
We do not see the parquet reader error. However, we still need to investigate
why the query takes so long (forever) to complete execution.
{noformat}
In drill-env.sh we set max direct memory to 12G on all 4nodes in cluster
export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-"12G"}
and also set these options from sqlline,
alter system set `planner.memory.max_query_memory_per_node` = 10737418240;
alter system set `drill.exec.hashagg.fallback.enabled` = true;{noformat}
> TPC-DS query 72 has regressed
> -
>
> Key: DRILL-6453
> URL: https://issues.apache.org/jira/browse/DRILL-6453
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Boaz Ben-Zvi
>Priority: Blocker
> Fix For: 1.14.0
>
> Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill
>
>
> TPC-DS query 72 seems to have regressed, query profile for the case where it
> Canceled after 2 hours on Drill 1.14.0 is attached here.
> {noformat}
> On, Drill 1.14.0-SNAPSHOT
> commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took
> around 55 seconds to execute)
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> TPC-DS query 72 executed successfully & took 47 seconds to complete execution.
> {noformat}
> {noformat}
> TPC-DS data in the below run has date values stored as DATE datatype and not
> VARCHAR type
> On, Drill 1.14.0-SNAPSHOT
> commit : 82e1a12
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> and
> alter system set `exec.hashjoin.num_partitions` = 1;
> TPC-DS query 72 executed for 2 hrs and 11 mins and did not complete, I had to
> Cancel it by stopping the Foreman drillbit.
> As a result several minor fragments are reported to be in
> CANCELLATION_REQUESTED state on UI.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6453) TPC-DS query 72 has regressed
[
https://issues.apache.org/jira/browse/DRILL-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503946#comment-16503946
]
Khurram Faraaz commented on DRILL-6453:
---
[~ben-zvi] TPC-DS query 72 can not be executed on latest Apache master, due to
a known issue in the Parquet reader.
[~dgu-atmapr] can you please file a Jira for the below issue ?
{noformat}
Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Error in
parquet record reader.
Message:
Hadoop path:
/drill/testdata/tpcds_sf1/parquet/customer_demographics/0_0_0.parquet
Total records read: 0
Row group index: 0
Records in row group: 1920800
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {
optional int64 cd_demo_sk;
optional binary cd_gender (UTF8);
optional binary cd_marital_status (UTF8);
optional binary cd_education_status (UTF8);
optional int32 cd_purchase_estimate;
optional binary cd_credit_rating (UTF8);
optional int32 cd_dep_count;
optional int32 cd_dep_employed_count;
optional int32 cd_dep_college_count;
}
, metadata: {drill-writer.version=2, drill.version=1.13.0-SNAPSHOT}}, blocks:
[BlockMetaData{1920800, 112509832 [ColumnMetaData{UNCOMPRESSED [cd_demo_sk]
INT64 [BIT_PACKED, RLE, PLAIN], 4}, ColumnMetaData{UNCOMPRESSED [cd_gender]
BINARY [BIT_PACKED, RLE, PLAIN], 15367257}, ColumnMetaData{UNCOMPRESSED
[cd_marital_status] BINARY [BIT_PACKED, RLE, PLAIN], 24971685},
ColumnMetaData{UNCOMPRESSED [cd_education_status] BINARY [BIT_PACKED, RLE,
PLAIN], 34576113}, ColumnMetaData{UNCOMPRESSED [cd_purchase_estimate] INT32
[BIT_PACKED, RLE, PLAIN], 60645586}, ColumnMetaData{UNCOMPRESSED
[cd_credit_rating] BINARY [BIT_PACKED, RLE, PLAIN], 68329176},
ColumnMetaData{UNCOMPRESSED [cd_dep_count] INT32 [BIT_PACKED, RLE, PLAIN],
89459066}, ColumnMetaData{UNCOMPRESSED [cd_dep_employed_count] INT32
[BIT_PACKED, RLE, PLAIN], 97142656}, ColumnMetaData{UNCOMPRESSED
[cd_dep_college_count] INT32 [BIT_PACKED, RLE, PLAIN], 104826246}]}]}
at
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.handleException(ParquetRecordReader.java:275)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:302)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:172)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
... 21 common frames omitted
Caused by: java.lang.UnsupportedOperationException: Unsupoorted Operation
at
org.apache.drill.exec.store.parquet.columnreaders.PageReader.resetDefinitionLevelReader(PageReader.java:449)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput$VLColumnBulkInputCallback.resetDefinitionLevelReader(VarLenColumnBulkInput.java:422)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.VarLenBulkPageReader.getEntry(VarLenBulkPageReader.java:113)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput.next(VarLenColumnBulkInput.java:128)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput.next(VarLenColumnBulkInput.java:32)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe(VarCharVector.java:624)
~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(NullableVarCharVector.java:719)
~[vector-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.VarLengthColumnReaders$NullableVarCharColumn.setSafe(VarLengthColumnReaders.java:215)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.VarLengthValuesColumn.readRecordsInBulk(VarLengthValuesColumn.java:98)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.VarLenBinaryReader.readRecordsInBulk(VarLenBinaryReader.java:102)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.VarLenBinaryReader.readFields(VarLenBinaryReader.java:90)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.BatchReader$VariableWidthReader.readRecords(BatchReader.java:166)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.columnreaders.BatchReader.readBatch(BatchReader.java:42)
[jira] [Commented] (DRILL-6453) TPC-DS query 72 has regressed
[
https://issues.apache.org/jira/browse/DRILL-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503915#comment-16503915
]
Khurram Faraaz commented on DRILL-6453:
---
[~ben-zvi] the query shows same behavior with or without setting (does not
complete after 2 hours of execution)
`exec.hashjoin.num_partitions` = 1;
I will share the drillbit.log
> TPC-DS query 72 has regressed
> -
>
> Key: DRILL-6453
> URL: https://issues.apache.org/jira/browse/DRILL-6453
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Boaz Ben-Zvi
>Priority: Blocker
> Fix For: 1.14.0
>
> Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill
>
>
> TPC-DS query 72 seems to have regressed, query profile for the case where it
> Canceled after 2 hours on Drill 1.14.0 is attached here.
> {noformat}
> On, Drill 1.14.0-SNAPSHOT
> commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took
> around 55 seconds to execute)
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> TPC-DS query 72 executed successfully & took 47 seconds to complete execution.
> {noformat}
> {noformat}
> TPC-DS data in the below run has date values stored as DATE datatype and not
> VARCHAR type
> On, Drill 1.14.0-SNAPSHOT
> commit : 82e1a12
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> and
> alter system set `exec.hashjoin.num_partitions` = 1;
> TPC-DS query 72 executed for 2 hrs and 11 mins and did not complete, I had to
> Cancel it by stopping the Foreman drillbit.
> As a result several minor fragments are reported to be in
> CANCELLATION_REQUESTED state on UI.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6453) TPC-DS query 72 has regressed
[
https://issues.apache.org/jira/browse/DRILL-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502585#comment-16502585
]
Khurram Faraaz commented on DRILL-6453:
---
[~vvysotskyi] TPC-DS query 72 executed in close to 50 seconds, prior to this
commit
89e0fe6b34259a2f51a7c45070935a2a2400eca4
After above commit we see the increase in query execution time.
> TPC-DS query 72 has regressed
> -
>
> Key: DRILL-6453
> URL: https://issues.apache.org/jira/browse/DRILL-6453
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Volodymyr Vysotskyi
>Priority: Critical
> Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill
>
>
> TPC-DS query 72 seems to have regressed, query profile for the case where it
> Canceled after 2 hours on Drill 1.14.0 is attached here.
> {noformat}
> On, Drill 1.14.0-SNAPSHOT
> commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took
> around 55 seconds to execute)
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> TPC-DS query 72 executed successfully & took 47 seconds to complete execution.
> {noformat}
> {noformat}
> TPC-DS data in the below run has date values stored as DATE datatype and not
> VARCHAR type
> On, Drill 1.14.0-SNAPSHOT
> commit : 82e1a12
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> and
> alter system set `exec.hashjoin.num_partitions` = 1;
> TPC-DS query 72 executed for 2 hrs and 11 mins and did not complete, I had to
> Cancel it by stopping the Foreman drillbit.
> As a result several minor fragments are reported to be in
> CANCELLATION_REQUESTED state on UI.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6452) document steps to execute SQL queries from Postman (chrome extension) on Drill
[
https://issues.apache.org/jira/browse/DRILL-6452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495713#comment-16495713
]
Khurram Faraaz commented on DRILL-6452:
---
[~bbevens] thanks for your review and edits. I have accepted your changes.
Looks good!
> document steps to execute SQL queries from Postman (chrome extension) on Drill
> --
>
> Key: DRILL-6452
> URL: https://issues.apache.org/jira/browse/DRILL-6452
> Project: Apache Drill
> Issue Type: Task
> Components: Documentation
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Khurram Faraaz
>Priority: Minor
>
> We need documentation to list the steps with screen shots about executing SQL
> queries from Postman (chrome extension) on Drill.
> Here are the steps to execute SQL queries from Postman
> {noformat}
> 1. Install Postman extension for Chrome browser.
> To install Postman
> https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncdddomop?hl=en
> Then click on ADD TO CHROME button
> 2. On the top right of your Chrome browser window, click on the postman icon.
> In your Postman:
> 3. set the type to “POST” and enter the request URL as “http:// ip>:8047/query.json”
> 4. In the Header tab, add an entry for “Content-Type” as key and
> “application/json” as value.
> Add another entry for “User-Name” as key and “mapr” as value
> 5. In the Body tab, select “raw” and a new dropdown list should appear next
> to “raw" and on the dropdown select “JSON”
> 6. And in the Body box, enter your request body in JSON format. The file
> test.csv is expected to reside under /tmp folder (i.e. in dfs.tmp schema)
> {
> “queryType”: “SQL”,
> “query”: “select * from `dfs.tmp`.`test.csv`”
> }
> 5. Press send!
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6452) document steps to execute SQL queries from Postman (chrome extension) on Drill
[
https://issues.apache.org/jira/browse/DRILL-6452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-6452:
--
Description:
We need documentation to list the steps with screen shots about executing SQL
queries from Postman (chrome extension) on Drill.
Here are the steps to execute SQL queries from Postman
{noformat}
1. Install Postman extension for Chrome browser.
To install Postman
https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncdddomop?hl=en
Then click on ADD TO CHROME button
2. On the top right of your Chrome browser window, click on the postman icon.
In your Postman:
3. set the type to “POST” and enter the request URL as “http://:8047/query.json”
4. In the Header tab, add an entry for “Content-Type” as key and
“application/json” as value.
Add another entry for “User-Name” as key and “mapr” as value
5. In the Body tab, select “raw” and a new dropdown list should appear next to
“raw" and on the dropdown select “JSON”
6. And in the Body box, enter your request body in JSON format. The file
test.csv is expected to reside under /tmp folder (i.e. in dfs.tmp schema)
{
“queryType”: “SQL”,
“query”: “select * from `dfs.tmp`.`test.csv`”
}
5. Press send!
{noformat}
was:We need documentation to list the steps with screen shots about executing
SQL queries from Postman (chrome extension) on Drill.
> document steps to execute SQL queries from Postman (chrome extension) on Drill
> --
>
> Key: DRILL-6452
> URL: https://issues.apache.org/jira/browse/DRILL-6452
> Project: Apache Drill
> Issue Type: Task
> Components: Documentation
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>Assignee: Khurram Faraaz
>Priority: Minor
>
> We need documentation to list the steps with screen shots about executing SQL
> queries from Postman (chrome extension) on Drill.
> Here are the steps to execute SQL queries from Postman
> {noformat}
> 1. Install Postman extension for Chrome browser.
> To install Postman
> https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncdddomop?hl=en
> Then click on ADD TO CHROME button
> 2. On the top right of your Chrome browser window, click on the postman icon.
> In your Postman:
> 3. set the type to “POST” and enter the request URL as “http:// ip>:8047/query.json”
> 4. In the Header tab, add an entry for “Content-Type” as key and
> “application/json” as value.
> Add another entry for “User-Name” as key and “mapr” as value
> 5. In the Body tab, select “raw” and a new dropdown list should appear next
> to “raw" and on the dropdown select “JSON”
> 6. And in the Body box, enter your request body in JSON format. The file
> test.csv is expected to reside under /tmp folder (i.e. in dfs.tmp schema)
> {
> “queryType”: “SQL”,
> “query”: “select * from `dfs.tmp`.`test.csv`”
> }
> 5. Press send!
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6453) TPC-DS query 72 has regressed
Khurram Faraaz created DRILL-6453:
-
Summary: TPC-DS query 72 has regressed
Key: DRILL-6453
URL: https://issues.apache.org/jira/browse/DRILL-6453
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill
TPC-DS query 72 seems to have regressed, query profile for the case where it
Canceled after 2 hours on Drill 1.14.0 is attached here.
{noformat}
On, Drill 1.14.0-SNAPSHOT
commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took
around 55 seconds to execute)
SF1 parquet data on 4 nodes;
planner.memory.max_query_memory_per_node = 10737418240.
drill.exec.hashagg.fallback.enabled = true
TPC-DS query 72 executed successfully & took 47 seconds to complete execution.
{noformat}
{noformat}
TPC-DS data in the below run has date values stored as DATE datatype and not
VARCHAR type
On, Drill 1.14.0-SNAPSHOT
commit : 82e1a12
SF1 parquet data on 4 nodes;
planner.memory.max_query_memory_per_node = 10737418240.
drill.exec.hashagg.fallback.enabled = true
and
alter system set `exec.hashjoin.num_partitions` = 1;
TPC-DS query 72 executed for 2 hrs and 11 mins and did not complete, I had to
Cancel it by stopping the Foreman drillbit.
As a result several minor fragments are reported to be in
CANCELLATION_REQUESTED state on UI.
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6452) document steps to execute SQL queries from Postman (chrome extension) on Drill
Khurram Faraaz created DRILL-6452: - Summary: document steps to execute SQL queries from Postman (chrome extension) on Drill Key: DRILL-6452 URL: https://issues.apache.org/jira/browse/DRILL-6452 Project: Apache Drill Issue Type: Bug Components: Documentation Affects Versions: 1.14.0 Reporter: Khurram Faraaz We need documentation to list the steps with screen shots about executing SQL queries from Postman (chrome extension) on Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6441) IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.
[
https://issues.apache.org/jira/browse/DRILL-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-6441:
--
Description:
This is seen when the drillbit is shutdown using $DRILL_HOME/bin/drillbit.sh
stop, it happens when BaseAllocator tries to close.
Apache Drill 1.14.0 commit c6c5d27d91468a29656bee2acba55d3321978aab
Adding details from Tim's email here
{noformat}
This issue is not related to DRILL-5922, however it is yet another way we leak
memory. Looking at the code when you view or submit a query via the rest api as
an anonymous user, a session is created with an allocator. If you call
drillbit.sh stop during this time, the Drillbit can shutdown while the
anonymous session is active and while a child allocator is being used. In this
case you will see the error you reported below.{noformat}
{noformat}
2018-05-11 15:19:44,510 [2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:0:0:
State to report: FINISHED
Wed May 16 10:17:13 PDT 2018 Terminating drillbit pid 32076
2018-05-16 10:17:13,793 [Drillbit-ShutdownHook#0] INFO
o.apache.drill.exec.server.Drillbit - Received shutdown request.
2018-05-16 10:17:14,876 [Drillbit-ShutdownHook#0] INFO
o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 20, cache miss
count: 6, hit rate: 70%
2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] ERROR
o.a.d.exec.server.BootStrapContext - Error while closing
java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding child
allocators.
Allocator(ROOT) 0/0/9577600/34359738368 (res/actual/peak/limit)
child allocators: 8
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
ledgers: 0
reservations: 0
at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:496)
~[drill-memory-base-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:259)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:263)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:363)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] INFO
o.apache.drill.exec.server.Drillbit - Shutdown completed (1095 ms).
{noformat}
was:
This is seen when the drillbit is shutdown using $DRILL_HOME/bin/drillbit.sh
stop, it happens when BaseAllocator tries to close.
Apache Drill 1.14.0 commit c6c5d27d91468a29656bee2acba55d3321978aab
{noformat}
2018-05-11 15:19:44,510 [2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:0:0:
State to report: FINISHED
Wed May 16 10:17:13 PDT 2018 Terminating drillbit pid 32076
2018-05-16 10:17:13,793 [Drillbit-ShutdownHook#0] INFO
o.apache.drill.exec.server.Drillbit - Received shutdown request.
2018-05-16 10:17:14,876 [Drillbit-ShutdownHook#0] INFO
o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 20, cache miss
count: 6, hit rate: 70%
2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] ERROR
o.a.d.exec.server.BootStrapContext - Error while closing
java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding child
allocators.
[jira] [Created] (DRILL-6441) IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.
Khurram Faraaz created DRILL-6441:
-
Summary: IllegalStateException: Allocator[ROOT] closed with
outstanding child allocators.
Key: DRILL-6441
URL: https://issues.apache.org/jira/browse/DRILL-6441
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.14.0
Reporter: Khurram Faraaz
This is seen when the drillbit is shutdown using $DRILL_HOME/bin/drillbit.sh
stop, it happens when BaseAllocator tries to close.
Apache Drill 1.14.0 commit c6c5d27d91468a29656bee2acba55d3321978aab
{noformat}
2018-05-11 15:19:44,510 [2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:0:0:
State to report: FINISHED
Wed May 16 10:17:13 PDT 2018 Terminating drillbit pid 32076
2018-05-16 10:17:13,793 [Drillbit-ShutdownHook#0] INFO
o.apache.drill.exec.server.Drillbit - Received shutdown request.
2018-05-16 10:17:14,876 [Drillbit-ShutdownHook#0] INFO
o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 20, cache miss
count: 6, hit rate: 70%
2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] ERROR
o.a.d.exec.server.BootStrapContext - Error while closing
java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding child
allocators.
Allocator(ROOT) 0/0/9577600/34359738368 (res/actual/peak/limit)
child allocators: 8
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807
(res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
ledgers: 0
reservations: 0
at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:496)
~[drill-memory-base-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:259)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:263)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:363)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] INFO
o.apache.drill.exec.server.Drillbit - Shutdown completed (1095 ms).
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6427) outputBatchSize is missing from the DEBUG output for HashJoinBatch operator
Khurram Faraaz created DRILL-6427:
-
Summary: outputBatchSize is missing from the DEBUG output for
HashJoinBatch operator
Key: DRILL-6427
URL: https://issues.apache.org/jira/browse/DRILL-6427
Project: Apache Drill
Issue Type: Bug
Reporter: Khurram Faraaz
Assignee: Padma Penumarthy
Drill 1.14.0-SNAPSHOT commit : f99d1f1323c0a5bac99842d6283d3025f3cb527f
outputBatchSize is missing from the DEBUG output for HashJoinBatch operator
Query used in test,
{noformat}
select count(*) from `twovarchar_asc_128MB.parquet` t1,
`twovarchar_asc_16MB.parquet` t2 WHERE t1.Varbinaryvalue = t2.Varbinaryvalue
{noformat}
Snippet from drillbit.log
{noformat}
2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG
o.a.d.e.p.impl.join.HashJoinBatch - left input: batch count : 1, avg batch
bytes : 90145920, avg row bytes : 8166, record count : 11040
2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG
o.a.d.e.p.impl.join.HashJoinBatch - right input: batch count : 1, avg batch
bytes : 10567808, avg row bytes : 7506, record count : 1408
2018-05-18 11:23:59,655 [2500e5c3-8f54-1f92-6eeb-7a81499a8abd:frag:0:0] DEBUG
o.a.d.e.p.impl.join.HashJoinBatch - output: batch count : 166, avg batch bytes
: 15951453, avg row bytes : 15672, record count : 168960
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6395) Value Window Function - LEAD and LAG on VarChar result in "No applicable constructor/method found" error
[
https://issues.apache.org/jira/browse/DRILL-6395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-6395:
--
Labels: window_function (was: )
> Value Window Function - LEAD and LAG on VarChar result in "No applicable
> constructor/method found" error
> -
>
> Key: DRILL-6395
> URL: https://issues.apache.org/jira/browse/DRILL-6395
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.13.0
> Environment: windows 10, apache drill 1.13.0, 32GB Ram
>Reporter: Raymond Wong
>Priority: Major
> Labels: window_function
>
> {code:java}
> SELECT
> col2,
> LEAD(col1, 1) OVER (ORDER BY col2) AS nxtCol1
> FROM (
> SELECT 'A' AS col1, 1 AS col2
> UNION
> SELECT 'B' AS col1, 2 AS col2
> UNION
> SELECT 'C' AS col1, 3 AS col2
> ) AS A;
> {code}
> Causes error
> {code:java}
> SQL Error: SYSTEM ERROR: CompileException: Line 37, Column 40:
> No applicable constructor/method found for actual parameters "int, int, int,
> io.netty.buffer.DrillBuf";
> candidates are:
> "public void
> org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
> org.apache.drill.exec.expr.holders.VarCharHolder)",
> "public void
> org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
> org.apache.drill.exec.expr.holders.NullableVarCharHolder)",
> "public void
> org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
> byte[], int, int)",
> "public void
> org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
> java.nio.ByteBuffer, int, int)",
> "public void
> org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int, int,
> int, int, io.netty.buffer.DrillBuf)"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6395) Value Window Function - LEAD and LAG on VarChar result in "No applicable constructor/method found" error
[
https://issues.apache.org/jira/browse/DRILL-6395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469541#comment-16469541
]
Khurram Faraaz commented on DRILL-6395:
---
Here is the stack trace from drillbit.log,
seen on Drill 1.14.0-SNAPSHOT , commit :
c6c5d27d91468a29656bee2acba55d3321978aab
{noformat}
2018-05-09 14:21:57,085 [250c998a-5917-1cc5-9364-4d7cf441f209:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query id
250c998a-5917-1cc5-9364-4d7cf441f209: SELECT
col2,
LEAD(col1, 1) OVER (ORDER BY col2) AS nxtCol1
FROM (
SELECT 'A' AS col1, 1 AS col2
UNION
SELECT 'B' AS col1, 2 AS col2
UNION
SELECT 'C' AS col1, 3 AS col2
) AS A
...
2018-05-09 14:21:57,600 [250c998a-5917-1cc5-9364-4d7cf441f209:frag:0:0] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: CompileException: Line 37,
Column 40: No applicable constructor/method found for actual parameters "int,
int, int, io.netty.buffer.DrillBuf"; candidates are: "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int, byte[],
int, int)", "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
java.nio.ByteBuffer, int, int)", "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int, int,
int, int, io.netty.buffer.DrillBuf)", "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
org.apache.drill.exec.expr.holders.NullableVarCharHolder)", "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
org.apache.drill.exec.expr.holders.VarCharHolder)"
Fragment 0:0
[Error Id: 71fc40fc-0b52-4c9b-b3ae-1ddad76ea120 on qa102-45.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
CompileException: Line 37, Column 40: No applicable constructor/method found
for actual parameters "int, int, int, io.netty.buffer.DrillBuf"; candidates
are: "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int, byte[],
int, int)", "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
java.nio.ByteBuffer, int, int)", "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int, int,
int, int, io.netty.buffer.DrillBuf)", "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
org.apache.drill.exec.expr.holders.NullableVarCharHolder)", "public void
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(int,
org.apache.drill.exec.expr.holders.VarCharHolder)"
Fragment 0:0
[Error Id: 71fc40fc-0b52-4c9b-b3ae-1ddad76ea120 on qa102-45.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:359)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:214)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:325)
[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: org.apache.drill.common.exceptions.DrillRuntimeException:
org.apache.drill.exec.exception.SchemaChangeException: Exception when creating
the schema
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:167)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:118)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:108)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:137)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:118)
~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
