[jira] [Commented] (DRILL-5947) COUNT() on empty JSON results in schema change error

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245316#comment-16245316
 ] 

Paul Rogers commented on DRILL-5947:


Will fix this in conjunction with the revision to the JSON reader for the batch 
size control project.

> COUNT() on empty JSON results in schema change error
> -
>
> Key: DRILL-5947
> URL: https://issues.apache.org/jira/browse/DRILL-5947
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.11.0
>Reporter: Khurram Faraaz
>Assignee: Paul Rogers
>
> Count over a field, from a non-empty and another empty JSON file results in 
> schema change error, "Streaming aggregate does not support schema changes"
> {noformat}
> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support 
> was removed in 8.0
> apache drill 1.11.0-mapr 
> "drill baby drill"
> 0: jdbc:drill:schema=dfs.tmp> select count(id) from `diff_types`;
> Error: UNSUPPORTED_OPERATION ERROR: Streaming aggregate does not support 
> schema changes
> Prior schema : 
> BatchSchema [fields=[id(BIGINT:OPTIONAL)], selectionVector=NONE]
> New schema : 
> BatchSchema [fields=[id(INT:OPTIONAL)], selectionVector=NONE]
> Fragment 0:0
> [Error Id: c1c77a0e-c7cf-494e-b8d2-ef2cc7e816e0 on centos-01.qa.lab:31010] 
> (state=,code=0)
> 0: jdbc:drill:schema=dfs.tmp> select id from `dif`Display all 872 
> possibilities? (y or n)
> 0: jdbc:drill:schema=dfs.tmp> select id from `diff_types/m1.json`;
> +---+
> |  id   |
> +---+
> | 1234  |
> +---+
> 1 row selected (0.157 seconds)
> 0: jdbc:drill:schema=dfs.tmp> select id from `diff_types/m2.json`;
> +---+
> |  id   |
> +---+
> | null  |
> +---+
> 1 row selected (0.169 seconds)
> 0: jdbc:drill:schema=dfs.tmp> select * from `diff_types`;
> +---+
> |  id   |
> +---+
> | 1234  |
> | null  |
> +---+
> 2 rows selected (0.209 seconds)
> 0: jdbc:drill:schema=dfs.tmp> select count(*) from `diff_types`;
> +-+
> | EXPR$0  |
> +-+
> | 2   |
> +-+
> 1 row selected (0.208 seconds)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (DRILL-5947) COUNT() on empty JSON results in schema change error

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers reassigned DRILL-5947:
--

Assignee: Paul Rogers

> COUNT() on empty JSON results in schema change error
> -
>
> Key: DRILL-5947
> URL: https://issues.apache.org/jira/browse/DRILL-5947
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.11.0
>Reporter: Khurram Faraaz
>Assignee: Paul Rogers
>
> Count over a field, from a non-empty and another empty JSON file results in 
> schema change error, "Streaming aggregate does not support schema changes"
> {noformat}
> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support 
> was removed in 8.0
> apache drill 1.11.0-mapr 
> "drill baby drill"
> 0: jdbc:drill:schema=dfs.tmp> select count(id) from `diff_types`;
> Error: UNSUPPORTED_OPERATION ERROR: Streaming aggregate does not support 
> schema changes
> Prior schema : 
> BatchSchema [fields=[id(BIGINT:OPTIONAL)], selectionVector=NONE]
> New schema : 
> BatchSchema [fields=[id(INT:OPTIONAL)], selectionVector=NONE]
> Fragment 0:0
> [Error Id: c1c77a0e-c7cf-494e-b8d2-ef2cc7e816e0 on centos-01.qa.lab:31010] 
> (state=,code=0)
> 0: jdbc:drill:schema=dfs.tmp> select id from `dif`Display all 872 
> possibilities? (y or n)
> 0: jdbc:drill:schema=dfs.tmp> select id from `diff_types/m1.json`;
> +---+
> |  id   |
> +---+
> | 1234  |
> +---+
> 1 row selected (0.157 seconds)
> 0: jdbc:drill:schema=dfs.tmp> select id from `diff_types/m2.json`;
> +---+
> |  id   |
> +---+
> | null  |
> +---+
> 1 row selected (0.169 seconds)
> 0: jdbc:drill:schema=dfs.tmp> select * from `diff_types`;
> +---+
> |  id   |
> +---+
> | 1234  |
> | null  |
> +---+
> 2 rows selected (0.209 seconds)
> 0: jdbc:drill:schema=dfs.tmp> select count(*) from `diff_types`;
> +-+
> | EXPR$0  |
> +-+
> | 2   |
> +-+
> 1 row selected (0.208 seconds)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5923) State of a successfully completed query shown as "COMPLETED"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245302#comment-16245302
 ] 

ASF GitHub Bot commented on DRILL-5923:
---

Github user prasadns14 commented on the issue:

https://github.com/apache/drill/pull/1021
  
@arina-ielchiieva, please review


> State of a successfully completed query shown as "COMPLETED"
> 
>
> Key: DRILL-5923
> URL: https://issues.apache.org/jira/browse/DRILL-5923
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Prasad Nagaraj Subramanya
>Assignee: Prasad Nagaraj Subramanya
> Fix For: 1.12.0
>
>
> Drill UI currently lists a successfully completed query as "COMPLETED". 
> Successfully completed, failed and canceled queries are all grouped as 
> Completed queries. 
> It would be better to list the state of a successfully completed query as 
> "Succeeded" to avoid confusion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5921) Counters metrics should be listed in table

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245301#comment-16245301
 ] 

ASF GitHub Bot commented on DRILL-5921:
---

Github user prasadns14 commented on a diff in the pull request:

https://github.com/apache/drill/pull/1020#discussion_r149881450
  
--- Diff: exec/java-exec/src/main/resources/rest/metrics/metrics.ftl ---
@@ -138,21 +154,14 @@
   });
 };
 
-function updateOthers(metrics) {
-  $.each(["counters", "meters"], function(i, key) {
-if(! $.isEmptyObject(metrics[key])) {
-  $("#" + key + "Val").html(JSON.stringify(metrics[key], null, 2));
-}
-  });
-};
-
 var update = function() {
   $.get("/status/metrics", function(metrics) {
 updateGauges(metrics.gauges);
 updateBars(metrics.gauges);
 if(! $.isEmptyObject(metrics.timers)) createTable(metrics.timers, 
"timers");
 if(! $.isEmptyObject(metrics.histograms)) 
createTable(metrics.histograms, "histograms");
-updateOthers(metrics);
+if(! $.isEmptyObject(metrics.counters)) 
createCountersTable(metrics.counters);
+if(! $.isEmptyObject(metrics.meters)) 
$("#metersVal").html(JSON.stringify(metrics.meters, null, 2));
--- End diff --

@arina-ielchiieva, please review

1) Added three screenshots before (current behavior), snapshot1(reusing 
createTable function) and snapshot2 (new createCountersTable function)

2) The function "createTable" lists the keys for each metric class, which 
looks good for "timers" and "histograms" as they have many keys. But the 
"counters" metric has only one key "count" (same as guages metric which has 
only key value). So, I have a added a new function to just list the class and 
count value.

3) Currently we do not have any "meters" metric. So, I don't know how many 
keys "meters" metric will have. If we have multiple keys we can use createTable 
or else we may have to create a different function similar to 
createCountersTable. (depending on the key name)

I personally think if we have a single key, we shouldn't use the 
createTable function. We should just display the class and the only key value.

https://user-images.githubusercontent.com/4907022/32592926-b419532e-c4da-11e7-93f1-02514e51e28d.png;>
https://user-images.githubusercontent.com/4907022/32592927-b42e1b38-c4da-11e7-850e-32902b6eb6d0.png;>
https://user-images.githubusercontent.com/4907022/32592928-b4438b30-c4da-11e7-97c8-edc0482b1924.png;>






> Counters metrics should be listed in table
> --
>
> Key: DRILL-5921
> URL: https://issues.apache.org/jira/browse/DRILL-5921
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Prasad Nagaraj Subramanya
>Assignee: Prasad Nagaraj Subramanya
>Priority: Minor
> Fix For: 1.12.0
>
>
> Counter metrics are currently displayed as json string in the Drill UI. They 
> should be listed in a table similar to other metrics.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5932) HiveServer2 queries throw error with jdbc connection

[tooptoop4 (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

tooptoop4 updated DRILL-5932:
-
Fix Version/s: (was: 1.12.0)

> HiveServer2 queries throw error with jdbc connection
> 
>
> Key: DRILL-5932
> URL: https://issues.apache.org/jira/browse/DRILL-5932
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC, Storage - Hive
>Affects Versions: 1.11.0
> Environment: linux
> 2.3 hive version
>Reporter: tooptoop4
>Priority: Blocker
>
> Basic hive queries all throw error!
> {code:sql}
> copied 
> https://repo1.maven.org/maven2/org/apache/hive/hive-jdbc/2.3.0/hive-jdbc-2.3.0-standalone.jar
>  to /usr/lib/apache-drill-1.11.0/jars/3rdparty/hive-jdbc-2.3.0-standalone.jar
> added this storage plugin:
> {
>   "type": "jdbc",
>   "driver": "org.apache.hive.jdbc.HiveDriver",
>   "url": "jdbc:hive2://host:1/default",
>   "username": "hive",
>   "password": "hive1234",
>   "enabled": true
> }
> [ec2-user@host ~]$ cd /usr/lib/apache-drill-1.11.0
> [ec2-user@host apache-drill-1.11.0]$ ./bin/drill-embedded
> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support 
> was removed in 8.0
> Nov 01, 2017 7:53:53 AM org.glassfish.jersey.server.ApplicationHandler 
> initialize
> INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 
> 01:25:26...
> apache drill 1.11.0
> "this isn't your grandfather's sql"
> 0: jdbc:drill:zk=local> SELECT count(*) FROM hive2.`contact`;
> Error: DATA_READ ERROR: The JDBC storage plugin failed while trying setup the 
> SQL query.
> sql SELECT COUNT(*) AS EXPR$0
> FROM (SELECT 0 AS $f0
> FROM.default.contact) AS t
> plugin hive2
> Fragment 0:0
> [Error Id: 4b293e97-7547-49c5-91da-b9ee2f2184fc on 
> ip-myip.mydomain.orghere.com:31010] (state=,code=0)
> 0: jdbc:drill:zk=local> ALTER SESSION SET `exec.errors.verbose` = true;
> +---+---+
> |  ok   |summary|
> +---+---+
> | true  | exec.errors.verbose updated.  |
> +---+---+
> 1 row selected (0.351 seconds)
> 0: jdbc:drill:zk=local> select * from hive2.contact;
> Error: DATA_READ ERROR: The JDBC storage plugin failed while trying setup the 
> SQL query.
> sql SELECT *
> FROM.default.contact
> plugin hive2
> Fragment 0:0
> [Error Id: fe36b026-e8ff-4354-af6c-6073130680c9 on ip-ip.domain.org.com:31010]
>   (org.apache.hive.service.cli.HiveSQLException) Error while compiling 
> statement: FAILED: ParseException line 2:4 cannot recognize input near '.' 
> 'default' '.' in join source
> org.apache.hive.jdbc.Utils.verifySuccess():267
> org.apache.hive.jdbc.Utils.verifySuccessWithInfo():253
> org.apache.hive.jdbc.HiveStatement.runAsyncOnServer():313
> org.apache.hive.jdbc.HiveStatement.execute():253
> org.apache.hive.jdbc.HiveStatement.executeQuery():476
> org.apache.commons.dbcp.DelegatingStatement.executeQuery():208
> org.apache.commons.dbcp.DelegatingStatement.executeQuery():208
> org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup():177
> org.apache.drill.exec.physical.impl.ScanBatch.():104
> org.apache.drill.exec.physical.impl.ScanBatch.():126
> org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch():40
> org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch():33
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():156
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren():179
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():136
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren():179
> org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():109
> org.apache.drill.exec.physical.impl.ImplCreator.getExec():87
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():207
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1149
> java.util.concurrent.ThreadPoolExecutor$Worker.run():624
> java.lang.Thread.run():748
>   Caused By (org.apache.hive.service.cli.HiveSQLException) Error while 
> compiling statement: FAILED: ParseException line 2:4 cannot recognize input 
> near '.' 'default' '.' in join source
> org.apache.hive.service.cli.operation.Operation.toSQLException():380
> org.apache.hive.service.cli.operation.SQLOperation.prepare():206
> org.apache.hive.service.cli.operation.SQLOperation.runInternal():290
> org.apache.hive.service.cli.operation.Operation.run():320
> 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal():530
> 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync():517
> sun.reflect.GeneratedMethodAccessor86.invoke():-1
> 

[jira] [Created] (DRILL-5947) COUNT() on empty JSON results in schema change error

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5947:
-

 Summary: COUNT() on empty JSON results in schema 
change error
 Key: DRILL-5947
 URL: https://issues.apache.org/jira/browse/DRILL-5947
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Khurram Faraaz


Count over a field, from a non-empty and another empty JSON file results in 
schema change error, "Streaming aggregate does not support schema changes"

{noformat}
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was 
removed in 8.0
apache drill 1.11.0-mapr 
"drill baby drill"
0: jdbc:drill:schema=dfs.tmp> select count(id) from `diff_types`;
Error: UNSUPPORTED_OPERATION ERROR: Streaming aggregate does not support schema 
changes
Prior schema : 
BatchSchema [fields=[id(BIGINT:OPTIONAL)], selectionVector=NONE]
New schema : 
BatchSchema [fields=[id(INT:OPTIONAL)], selectionVector=NONE]

Fragment 0:0

[Error Id: c1c77a0e-c7cf-494e-b8d2-ef2cc7e816e0 on centos-01.qa.lab:31010] 
(state=,code=0)
0: jdbc:drill:schema=dfs.tmp> select id from `dif`Display all 872 
possibilities? (y or n)
0: jdbc:drill:schema=dfs.tmp> select id from `diff_types/m1.json`;
+---+
|  id   |
+---+
| 1234  |
+---+
1 row selected (0.157 seconds)
0: jdbc:drill:schema=dfs.tmp> select id from `diff_types/m2.json`;
+---+
|  id   |
+---+
| null  |
+---+
1 row selected (0.169 seconds)
0: jdbc:drill:schema=dfs.tmp> select * from `diff_types`;
+---+
|  id   |
+---+
| 1234  |
| null  |
+---+
2 rows selected (0.209 seconds)
0: jdbc:drill:schema=dfs.tmp> select count(*) from `diff_types`;
+-+
| EXPR$0  |
+-+
| 2   |
+-+
1 row selected (0.208 seconds)

{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5867) List profiles in pages rather than a long verbose listing

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua updated DRILL-5867:

Reviewer: Arina Ielchiieva

> List profiles in pages rather than a long verbose listing
> -
>
> Key: DRILL-5867
> URL: https://issues.apache.org/jira/browse/DRILL-5867
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Web Server
>Affects Versions: 1.11.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Minor
> Fix For: 1.12.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (DRILL-5867) List profiles in pages rather than a long verbose listing

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua resolved DRILL-5867.
-
Resolution: Fixed

> List profiles in pages rather than a long verbose listing
> -
>
> Key: DRILL-5867
> URL: https://issues.apache.org/jira/browse/DRILL-5867
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Web Server
>Affects Versions: 1.11.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Minor
> Fix For: 1.12.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5867) List profiles in pages rather than a long verbose listing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245129#comment-16245129
 ] 

ASF GitHub Bot commented on DRILL-5867:
---

Github user kkhatua commented on the issue:

https://github.com/apache/drill/pull/1029
  
@arina-ielchiieva / @paul-rogers Please review this.


> List profiles in pages rather than a long verbose listing
> -
>
> Key: DRILL-5867
> URL: https://issues.apache.org/jira/browse/DRILL-5867
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Web Server
>Affects Versions: 1.11.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Minor
> Fix For: 1.12.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5867) List profiles in pages rather than a long verbose listing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245128#comment-16245128
 ] 

ASF GitHub Bot commented on DRILL-5867:
---

GitHub user kkhatua opened a pull request:

https://github.com/apache/drill/pull/1029

DRILL-5867: List profiles in pages rather than a long verbose listing

Leverage existing DataTables libraries to paginate a long pre-fetched list 
of profiles for listing.
Added benefit of querying through that list (search field) is also 
available for a user
Minor change made to the display text for prefetching of profiles 
(DRILL-5259) so that it is not confused with what this commit adds to the UI.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kkhatua/drill DRILL-5867

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1029.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1029


commit 68d09a506d37134b660437260ef0fd4ca5ce3f9d
Author: Kunal Khatua 
Date:   2017-11-09T02:17:19Z

DRILL-5867: List profiles in pages rather than a long verbose listing

Leverage existing DataTables libraries to paginate a long pre-fetched list 
of profiles for listing.
Added benefit of querying through that list (search field) is also 
available for a user
Minor change made to the display text for prefetching of profiles 
(DRILL-5259) so that it is not confused with what this commit adds to the UI.




> List profiles in pages rather than a long verbose listing
> -
>
> Key: DRILL-5867
> URL: https://issues.apache.org/jira/browse/DRILL-5867
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Web Server
>Affects Versions: 1.11.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Minor
> Fix For: 1.12.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5867) List profiles in pages rather than a long verbose listing

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua updated DRILL-5867:

Fix Version/s: (was: 1.13.0)
   1.12.0

> List profiles in pages rather than a long verbose listing
> -
>
> Key: DRILL-5867
> URL: https://issues.apache.org/jira/browse/DRILL-5867
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Web Server
>Affects Versions: 1.11.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Minor
> Fix For: 1.12.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5923) State of a successfully completed query shown as "COMPLETED"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245069#comment-16245069
 ] 

ASF GitHub Bot commented on DRILL-5923:
---

Github user prasadns14 commented on a diff in the pull request:

https://github.com/apache/drill/pull/1021#discussion_r149846069
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/QueryStateDisplayName.java
 ---
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.server.rest.profile;
+
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+
+public class QueryStateDisplayName {
+  // Values should correspond to the QueryState enum in UserBitShared.proto
--- End diff --

QueryState is an enum.
I feel having an array is simple. We could use the QueryState ordinal as 
index to fetch the display name from the array. I can add a check to check for 
array index overflow, and return some default value like "Unknown state".


> State of a successfully completed query shown as "COMPLETED"
> 
>
> Key: DRILL-5923
> URL: https://issues.apache.org/jira/browse/DRILL-5923
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Prasad Nagaraj Subramanya
>Assignee: Prasad Nagaraj Subramanya
> Fix For: 1.12.0
>
>
> Drill UI currently lists a successfully completed query as "COMPLETED". 
> Successfully completed, failed and canceled queries are all grouped as 
> Completed queries. 
> It would be better to list the state of a successfully completed query as 
> "Succeeded" to avoid confusion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5923) State of a successfully completed query shown as "COMPLETED"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245062#comment-16245062
 ] 

ASF GitHub Bot commented on DRILL-5923:
---

Github user prasadns14 commented on a diff in the pull request:

https://github.com/apache/drill/pull/1021#discussion_r149844559
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/QueryStateDisplayName.java
 ---
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.server.rest.profile;
+
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+
+public class QueryStateDisplayName {
--- End diff --

sure will give a generic name


> State of a successfully completed query shown as "COMPLETED"
> 
>
> Key: DRILL-5923
> URL: https://issues.apache.org/jira/browse/DRILL-5923
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Prasad Nagaraj Subramanya
>Assignee: Prasad Nagaraj Subramanya
> Fix For: 1.12.0
>
>
> Drill UI currently lists a successfully completed query as "COMPLETED". 
> Successfully completed, failed and canceled queries are all grouped as 
> Completed queries. 
> It would be better to list the state of a successfully completed query as 
> "Succeeded" to avoid confusion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5923) State of a successfully completed query shown as "COMPLETED"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245060#comment-16245060
 ] 

ASF GitHub Bot commented on DRILL-5923:
---

Github user prasadns14 commented on a diff in the pull request:

https://github.com/apache/drill/pull/1021#discussion_r149844423
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/QueryStateDisplayName.java
 ---
@@ -0,0 +1,35 @@
+/**
--- End diff --

will fix it


> State of a successfully completed query shown as "COMPLETED"
> 
>
> Key: DRILL-5923
> URL: https://issues.apache.org/jira/browse/DRILL-5923
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Prasad Nagaraj Subramanya
>Assignee: Prasad Nagaraj Subramanya
> Fix For: 1.12.0
>
>
> Drill UI currently lists a successfully completed query as "COMPLETED". 
> Successfully completed, failed and canceled queries are all grouped as 
> Completed queries. 
> It would be better to list the state of a successfully completed query as 
> "Succeeded" to avoid confusion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5946) Document maprdb format plugin options

[Abhishek Girish (JIRA)]

Abhishek Girish created DRILL-5946:
--

 Summary: Document maprdb format plugin options
 Key: DRILL-5946
 URL: https://issues.apache.org/jira/browse/DRILL-5946
 Project: Apache Drill
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.12.0
Reporter: Abhishek Girish
Assignee: Bridget Bevens


The following options for maprdb format plugin need to be documented:
{code}
unionEnabled
readNumbersAsDouble
disablePushdown
allTextMode
ignoreSchemaChange
disableCountOptimization
nonExistentColumnsProjection
{code}
They are set as part of the maprdb format plugin configuration (within the DFS 
storage plugin configuration).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5945) Streaming aggregate does not support schema changes

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-5945:
-

 Summary: Streaming aggregate does not support schema changes
 Key: DRILL-5945
 URL: https://issues.apache.org/jira/browse/DRILL-5945
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.11.0
Reporter: Khurram Faraaz


There are two JSON files, one of the arrays "arr" in f2.json is empty, whereas 
array "arr" in f1.json is not empty, a count on arr results in, "Streaming 
aggregate does not support schema changes"

{noformat}
0: jdbc:drill:schema=dfs.tmp> select count(arr) from `f*.json`;
Error: UNSUPPORTED_OPERATION ERROR: Streaming aggregate does not support schema 
changes
Prior schema : 
BatchSchema [fields=[arr(VARCHAR:REPEATED) [$data$(VARCHAR:REQUIRED)]], 
selectionVector=NONE]
New schema : 
BatchSchema [fields=[arr(INT:REPEATED) [$data$(INT:REQUIRED)]], 
selectionVector=NONE]

Fragment 0:0

[Error Id: 9081b02f-0de8-486a-b58b-f28ec3b84d98 on centos-01.qa.lab:31010] 
(state=,code=0)

{noformat}

Stack trace from drillbit.log

{noformat}
2017-11-08 13:36:48,159 [25fc870f-e4d6-7107-c6cc-6e68ba92c4ba:frag:0:0] INFO  
o.a.d.e.p.i.a.StreamingAggBatch - User Error Occurred: Streaming aggregate does 
not support schema changes
Prior schema :
BatchSchema [fields=[arr(VARCHAR:REPEATED) [$data$(VARCHAR:REQUIRED)]], 
selectionVector=NONE]
New schema :
BatchSchema [fields=[arr(INT:REPEATED) [$data$(INT:REQUIRED)]], 
selectionVector=NONE]
org.apache.drill.common.exceptions.UserException: UNSUPPORTED_OPERATION ERROR: 
Streaming aggregate does not support schema changes
Prior schema :
BatchSchema [fields=[arr(VARCHAR:REPEATED) [$data$(VARCHAR:REQUIRED)]], 
selectionVector=NONE]
New schema :
BatchSchema [fields=[arr(INT:REPEATED) [$data$(INT:REQUIRED)]], 
selectionVector=NONE]


[Error Id: 9081b02f-0de8-486a-b58b-f28ec3b84d98 ]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
 ~[drill-common-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext(StreamingAggBatch.java:207)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105) 
[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95) 
[drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:234)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:227)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at java.security.AccessController.doPrivileged(Native Method) 
[na:1.8.0_144]
at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_144]
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
 [hadoop-common-2.7.0-mapr-1607.jar:na]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:227)
 [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.11.0-mapr.jar:1.11.0-mapr]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[na:1.8.0_144]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[na:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]

{noformat}

Data used in the query

{noformat}

[root@centos-01 json_data]# cat f1.json 
{
"id":1,
"arr":["str1","str2","str3","str4"]
}
[root@centos-01 json_data]# cat f2.json 
{
"id":2,
"arr":[]
}

>From sqlline

OpenJDK 64-Bit Server VM warning: ignoring option 

[jira] [Updated] (DRILL-5863) Sortable table incorrectly sorts minor fragments and time elements lexically instead of sorting by implicit value

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua updated DRILL-5863:

Affects Version/s: (was: 1.12.0)
   1.11.0

> Sortable table incorrectly sorts minor fragments and time elements lexically 
> instead of sorting by implicit value
> -
>
> Key: DRILL-5863
> URL: https://issues.apache.org/jira/browse/DRILL-5863
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Affects Versions: 1.11.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> The fix for this is to use dataTable library's {{data-order}} attribute for 
> the data elements that need to sort by an implicit value.
> ||Old order of Minor Fragment||New order of Minor Fragment||
> |...|...|
> |01-09-01  | 01-09-01|
> |01-10-01  | 01-10-01|
> |01-100-01 | 01-11-01|
> |01-101-01 | 01-12-01|
> |... | ... |
> ||Old order of Duration||New order of Duration|||
> |...|...|
> |1m15s  | 55.03s|
> |55s  | 1m15s|
> |...|...|



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5932) HiveServer2 queries throw error with jdbc connection

[tooptoop4 (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

tooptoop4 updated DRILL-5932:
-
Fix Version/s: 1.12.0

> HiveServer2 queries throw error with jdbc connection
> 
>
> Key: DRILL-5932
> URL: https://issues.apache.org/jira/browse/DRILL-5932
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC, Storage - Hive
>Affects Versions: 1.11.0
> Environment: linux
> 2.3 hive version
>Reporter: tooptoop4
>Priority: Blocker
> Fix For: 1.12.0
>
>
> Basic hive queries all throw error!
> {code:sql}
> copied 
> https://repo1.maven.org/maven2/org/apache/hive/hive-jdbc/2.3.0/hive-jdbc-2.3.0-standalone.jar
>  to /usr/lib/apache-drill-1.11.0/jars/3rdparty/hive-jdbc-2.3.0-standalone.jar
> added this storage plugin:
> {
>   "type": "jdbc",
>   "driver": "org.apache.hive.jdbc.HiveDriver",
>   "url": "jdbc:hive2://host:1/default",
>   "username": "hive",
>   "password": "hive1234",
>   "enabled": true
> }
> [ec2-user@host ~]$ cd /usr/lib/apache-drill-1.11.0
> [ec2-user@host apache-drill-1.11.0]$ ./bin/drill-embedded
> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support 
> was removed in 8.0
> Nov 01, 2017 7:53:53 AM org.glassfish.jersey.server.ApplicationHandler 
> initialize
> INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 
> 01:25:26...
> apache drill 1.11.0
> "this isn't your grandfather's sql"
> 0: jdbc:drill:zk=local> SELECT count(*) FROM hive2.`contact`;
> Error: DATA_READ ERROR: The JDBC storage plugin failed while trying setup the 
> SQL query.
> sql SELECT COUNT(*) AS EXPR$0
> FROM (SELECT 0 AS $f0
> FROM.default.contact) AS t
> plugin hive2
> Fragment 0:0
> [Error Id: 4b293e97-7547-49c5-91da-b9ee2f2184fc on 
> ip-myip.mydomain.orghere.com:31010] (state=,code=0)
> 0: jdbc:drill:zk=local> ALTER SESSION SET `exec.errors.verbose` = true;
> +---+---+
> |  ok   |summary|
> +---+---+
> | true  | exec.errors.verbose updated.  |
> +---+---+
> 1 row selected (0.351 seconds)
> 0: jdbc:drill:zk=local> select * from hive2.contact;
> Error: DATA_READ ERROR: The JDBC storage plugin failed while trying setup the 
> SQL query.
> sql SELECT *
> FROM.default.contact
> plugin hive2
> Fragment 0:0
> [Error Id: fe36b026-e8ff-4354-af6c-6073130680c9 on ip-ip.domain.org.com:31010]
>   (org.apache.hive.service.cli.HiveSQLException) Error while compiling 
> statement: FAILED: ParseException line 2:4 cannot recognize input near '.' 
> 'default' '.' in join source
> org.apache.hive.jdbc.Utils.verifySuccess():267
> org.apache.hive.jdbc.Utils.verifySuccessWithInfo():253
> org.apache.hive.jdbc.HiveStatement.runAsyncOnServer():313
> org.apache.hive.jdbc.HiveStatement.execute():253
> org.apache.hive.jdbc.HiveStatement.executeQuery():476
> org.apache.commons.dbcp.DelegatingStatement.executeQuery():208
> org.apache.commons.dbcp.DelegatingStatement.executeQuery():208
> org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup():177
> org.apache.drill.exec.physical.impl.ScanBatch.():104
> org.apache.drill.exec.physical.impl.ScanBatch.():126
> org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch():40
> org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch():33
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():156
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren():179
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():136
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren():179
> org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():109
> org.apache.drill.exec.physical.impl.ImplCreator.getExec():87
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():207
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1149
> java.util.concurrent.ThreadPoolExecutor$Worker.run():624
> java.lang.Thread.run():748
>   Caused By (org.apache.hive.service.cli.HiveSQLException) Error while 
> compiling statement: FAILED: ParseException line 2:4 cannot recognize input 
> near '.' 'default' '.' in join source
> org.apache.hive.service.cli.operation.Operation.toSQLException():380
> org.apache.hive.service.cli.operation.SQLOperation.prepare():206
> org.apache.hive.service.cli.operation.SQLOperation.runInternal():290
> org.apache.hive.service.cli.operation.Operation.run():320
> 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal():530
> 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync():517
> 

[jira] [Commented] (DRILL-5691) multiple count distinct query planning error at physical phase

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244608#comment-16244608
 ] 

ASF GitHub Bot commented on DRILL-5691:
---

Github user amansinha100 commented on the issue:

https://github.com/apache/drill/pull/889
  
@weijietong I am interested in getting your basic changes in.  It is 
unfortunate we are running into this issue with RelSubSet.  Let me see if I can 
make some changes on top of your changes  (I will keep your authorship) and 
have it pass all our functional regression tests. 


> multiple count distinct query planning error at physical phase 
> ---
>
> Key: DRILL-5691
> URL: https://issues.apache.org/jira/browse/DRILL-5691
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.9.0, 1.10.0
>Reporter: weijie.tong
>
> I materialized the count distinct query result in a cache , added a plugin 
> rule to translate the (Aggregate、Aggregate、Project、Scan) or 
> (Aggregate、Aggregate、Scan) to (Project、Scan) at the PARTITION_PRUNING phase. 
> Then ,once user issue count distinct queries , it will be translated to query 
> the cache to get the result.
> eg1: " select count(*),sum(a) ,count(distinct b)  from t where dt=xx " 
> eg2:"select count(*),sum(a) ,count(distinct b) ,count(distinct c) from t 
> where dt=xxx "
> eg3:"select count(distinct b), count(distinct c) from t where dt=xxx"
> eg1 will be right and have a query result as I expected , but eg2 will be 
> wrong at the physical phase.The error info is here: 
> https://gist.github.com/weijietong/1b8ed12db9490bf006e8b3fe0ee52269. 
> eg3 will also get the similar error.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4779) Kafka storage plugin support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244590#comment-16244590
 ] 

ASF GitHub Bot commented on DRILL-4779:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1027#discussion_r149763319
  
--- Diff: 
contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaGroupScan.java
 ---
@@ -0,0 +1,301 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.kafka;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.EndpointAffinity;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.ScanStats.GroupScanProperty;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.kafka.KafkaSubScan.KafkaSubScanSpec;
+import org.apache.drill.exec.store.kafka.decoders.MessageReader;
+import org.apache.drill.exec.store.kafka.decoders.MessageReaderFactory;
+import org.apache.drill.exec.store.schedule.AffinityCreator;
+import org.apache.drill.exec.store.schedule.AssignmentCreator;
+import org.apache.drill.exec.store.schedule.CompleteWork;
+import org.apache.drill.exec.store.schedule.EndpointByteMap;
+import org.apache.drill.exec.store.schedule.EndpointByteMapImpl;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.Node;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+@JsonTypeName("kafka-scan")
+public class KafkaGroupScan extends AbstractGroupScan {
+
+  static final Logger logger = 
LoggerFactory.getLogger(KafkaGroupScan.class);
+  private static final long MSG_SIZE = 1024;
+
+  private KafkaStoragePlugin kafkaStoragePlugin;
+  private KafkaStoragePluginConfig kafkaStoragePluginConfig;
+  private List columns;
+  private KafkaScanSpec kafkaScanSpec;
+
+  private List partitionWorkList;
+  private ListMultimap assignments;
+  private List affinities;
+
+  @JsonCreator
+  public KafkaGroupScan(@JsonProperty("userName") String userName,
+  @JsonProperty("kafkaStoragePluginConfig") KafkaStoragePluginConfig 
kafkaStoragePluginConfig,
+  @JsonProperty("columns") List columns, 
@JsonProperty("scanSpec") KafkaScanSpec scanSpec,
+  @JacksonInject StoragePluginRegistry pluginRegistry) {
+this(userName, kafkaStoragePluginConfig, columns, scanSpec, 
(KafkaStoragePlugin) pluginRegistry);
+  }
+
+  public KafkaGroupScan(KafkaStoragePlugin kafkaStoragePlugin, 
KafkaScanSpec kafkaScanSpec, List columns) {
+super(StringUtils.EMPTY);
+this.kafkaStoragePlugin = kafkaStoragePlugin;
+this.kafkaStoragePluginConfig = (KafkaStoragePluginConfig) 

[jira] [Commented] (DRILL-4779) Kafka storage plugin support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244587#comment-16244587
 ] 

ASF GitHub Bot commented on DRILL-4779:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1027#discussion_r149757320
  
--- Diff: 
contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/DrillKafkaConfig.java
 ---
@@ -0,0 +1,31 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.kafka;
+
+public class DrillKafkaConfig {
--- End diff --

Members are constants. Make this an interface?


> Kafka storage plugin support
> 
>
> Key: DRILL-4779
> URL: https://issues.apache.org/jira/browse/DRILL-4779
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Affects Versions: 1.11.0
>Reporter: B Anil Kumar
>Assignee: B Anil Kumar
>  Labels: doc-impacting
> Fix For: 1.12.0
>
>
> Implement Kafka storage plugin will enable the strong SQL support for Kafka.
> Initially implementation can target for supporting json and avro message types



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4779) Kafka storage plugin support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244592#comment-16244592
 ] 

ASF GitHub Bot commented on DRILL-4779:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1027#discussion_r149756110
  
--- Diff: contrib/storage-kafka/README.md ---
@@ -0,0 +1,230 @@
+# Drill Kafka Plugin
+
+Drill kafka storage plugin allows you to perform interactive analysis 
using SQL against Apache Kafka.
+
+Supported Kafka Version
+Kafka-0.10 and above 
+
+Message Formats
+Currently this plugin supports reading only Kafka messages of type 
JSON.
+
+
+Message Readers
+Message Readers are used for reading messages from Kafka. Type of the 
MessageReaders supported as of now are
+
+
+  
+MessageReader
+Description
+Key DeSerializer 
+Value DeSerializer
+  
+  
+JsonMessageReader
+To read Json messages
+org.apache.kafka.common.serialization.ByteArrayDeserializer 
+org.apache.kafka.common.serialization.ByteArrayDeserializer
+  
+
+
+
+Plugin Configurations
+Drill Kafka plugin supports following properties
+
+   kafkaConsumerProps: These are typical https://kafka.apache.org/documentation/#consumerconfigs;>Kafka consumer 
properties.
+drillKafkaProps: These are Drill Kafka plugin 
properties. As of now, it supports the following properties
+   
+drill.kafka.message.reader: Message Reader 
implementation to use while reading messages from Kafka. Message reader 
implementaion should be configured based on message format. Type of message 
readers
+ 
+ org.apache.drill.exec.store.kafka.decoders.JsonMessageReader
+ 
+
+drill.kafka.poll.timeout: Polling timeout used by 
Kafka client while fetching messages from Kafka cluster.
+
+
+
+
+Plugin Registration
+To register the kafka plugin, open the drill web interface. To open the 
drill web interface, enter http://drillbit:8047/storage in 
your browser.
+
+The following is an example plugin registration configuration
+
+{
+  "type": "kafka",
+  "kafkaConsumerProps": {
+"key.deserializer": 
"org.apache.kafka.common.serialization.ByteArrayDeserializer",
+"auto.offset.reset": "earliest",
+"bootstrap.servers": "localhost:9092",
+"enable.auto.commit": "true",
+"group.id": "drill-query-consumer-1",
+"value.deserializer": 
"org.apache.kafka.common.serialization.ByteArrayDeserializer",
+"session.timeout.ms": "3"
+  },
+  "drillKafkaProps": {
+"drill.kafka.message.reader": 
"org.apache.drill.exec.store.kafka.decoders.JsonMessageReader",
+"drill.kafka.poll.timeout": "2000"
+  },
+  "enabled": true
+}
+
+
+ Abstraction 
+In Drill, each Kafka topic is mapped to a SQL table and when a query is 
issued on a table, it scans all the messages from the earliest offset to the 
latest offset of that topic at that point of time. This plugin automatically 
discovers all the topics (tables), to allow you perform analysis without 
executing DDL statements.
--- End diff --

Does it make sense to provide a way to select a range of messages: a 
starting point or a count? Perhaps I want to run my query every five minutes, 
scanning only those messages since the previous scan. Or, I want to limit my 
take to, say, the next 1000 messages. Could we use a pseudo-column such as 
"kafkaMsgOffset" for that purpose? Maybe

```
SELECT * FROM  WHERE kafkaMsgOffset > 12345
```


> Kafka storage plugin support
> 
>
> Key: DRILL-4779
> URL: https://issues.apache.org/jira/browse/DRILL-4779
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Affects Versions: 1.11.0
>Reporter: B Anil Kumar
>Assignee: B Anil Kumar
>  Labels: doc-impacting
> Fix For: 1.12.0
>
>
> Implement Kafka storage plugin will enable the strong SQL support for Kafka.
> Initially implementation can target for supporting json and avro message types



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4779) Kafka storage plugin support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244589#comment-16244589
 ] 

ASF GitHub Bot commented on DRILL-4779:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1027#discussion_r149758022
  
--- Diff: 
contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaGroupScan.java
 ---
@@ -0,0 +1,301 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.kafka;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.EndpointAffinity;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.ScanStats.GroupScanProperty;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.kafka.KafkaSubScan.KafkaSubScanSpec;
+import org.apache.drill.exec.store.kafka.decoders.MessageReader;
+import org.apache.drill.exec.store.kafka.decoders.MessageReaderFactory;
+import org.apache.drill.exec.store.schedule.AffinityCreator;
+import org.apache.drill.exec.store.schedule.AssignmentCreator;
+import org.apache.drill.exec.store.schedule.CompleteWork;
+import org.apache.drill.exec.store.schedule.EndpointByteMap;
+import org.apache.drill.exec.store.schedule.EndpointByteMapImpl;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.Node;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+@JsonTypeName("kafka-scan")
+public class KafkaGroupScan extends AbstractGroupScan {
+
+  static final Logger logger = 
LoggerFactory.getLogger(KafkaGroupScan.class);
+  private static final long MSG_SIZE = 1024;
+
+  private KafkaStoragePlugin kafkaStoragePlugin;
+  private KafkaStoragePluginConfig kafkaStoragePluginConfig;
+  private List columns;
+  private KafkaScanSpec kafkaScanSpec;
+
+  private List partitionWorkList;
+  private ListMultimap assignments;
+  private List affinities;
+
+  @JsonCreator
+  public KafkaGroupScan(@JsonProperty("userName") String userName,
+  @JsonProperty("kafkaStoragePluginConfig") KafkaStoragePluginConfig 
kafkaStoragePluginConfig,
+  @JsonProperty("columns") List columns, 
@JsonProperty("scanSpec") KafkaScanSpec scanSpec,
+  @JacksonInject StoragePluginRegistry pluginRegistry) {
+this(userName, kafkaStoragePluginConfig, columns, scanSpec, 
(KafkaStoragePlugin) pluginRegistry);
+  }
+
+  public KafkaGroupScan(KafkaStoragePlugin kafkaStoragePlugin, 
KafkaScanSpec kafkaScanSpec, List columns) {
+super(StringUtils.EMPTY);
+this.kafkaStoragePlugin = kafkaStoragePlugin;
+this.kafkaStoragePluginConfig = (KafkaStoragePluginConfig) 

[jira] [Commented] (DRILL-4779) Kafka storage plugin support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244588#comment-16244588
 ] 

ASF GitHub Bot commented on DRILL-4779:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1027#discussion_r149756771
  
--- Diff: contrib/storage-kafka/pom.xml ---
@@ -0,0 +1,130 @@
+
+
+http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;
+   xmlns="http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
+   4.0.0
+
+   
+   drill-contrib-parent
+   org.apache.drill.contrib
+   1.12.0-SNAPSHOT
+   
+
+   drill-storage-kafka
+   contrib/kafka-storage-plugin
+
+   
+   
UTF-8
--- End diff --

Small nit: Drill prefers an indent of 2, even in the `pom.xml` file.


> Kafka storage plugin support
> 
>
> Key: DRILL-4779
> URL: https://issues.apache.org/jira/browse/DRILL-4779
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Affects Versions: 1.11.0
>Reporter: B Anil Kumar
>Assignee: B Anil Kumar
>  Labels: doc-impacting
> Fix For: 1.12.0
>
>
> Implement Kafka storage plugin will enable the strong SQL support for Kafka.
> Initially implementation can target for supporting json and avro message types



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4779) Kafka storage plugin support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244586#comment-16244586
 ] 

ASF GitHub Bot commented on DRILL-4779:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1027#discussion_r149770638
  
--- Diff: 
contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaRecordReader.java
 ---
@@ -0,0 +1,178 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.kafka;
+
+import static 
org.apache.drill.exec.store.kafka.DrillKafkaConfig.DRILL_KAFKA_POLL_TIMEOUT;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.kafka.KafkaSubScan.KafkaSubScanSpec;
+import org.apache.drill.exec.store.kafka.decoders.MessageReader;
+import org.apache.drill.exec.store.kafka.decoders.MessageReaderFactory;
+import org.apache.drill.exec.util.Utilities;
+import org.apache.drill.exec.vector.complex.impl.VectorContainerWriter;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.TopicPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Stopwatch;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+public class KafkaRecordReader extends AbstractRecordReader {
+  private static final Logger logger = 
LoggerFactory.getLogger(KafkaRecordReader.class);
+  public static final long DEFAULT_MESSAGES_PER_BATCH = 4000;
+
+  private VectorContainerWriter writer;
+  private MessageReader messageReader;
+
+  private boolean unionEnabled;
+  private KafkaConsumer kafkaConsumer;
+  private KafkaStoragePlugin plugin;
--- End diff --

`final`? And for others assigned in the constructor?


> Kafka storage plugin support
> 
>
> Key: DRILL-4779
> URL: https://issues.apache.org/jira/browse/DRILL-4779
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Affects Versions: 1.11.0
>Reporter: B Anil Kumar
>Assignee: B Anil Kumar
>  Labels: doc-impacting
> Fix For: 1.12.0
>
>
> Implement Kafka storage plugin will enable the strong SQL support for Kafka.
> Initially implementation can target for supporting json and avro message types



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4779) Kafka storage plugin support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244591#comment-16244591
 ] 

ASF GitHub Bot commented on DRILL-4779:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1027#discussion_r149760078
  
--- Diff: 
contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaGroupScan.java
 ---
@@ -0,0 +1,301 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.kafka;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.EndpointAffinity;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.ScanStats.GroupScanProperty;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.kafka.KafkaSubScan.KafkaSubScanSpec;
+import org.apache.drill.exec.store.kafka.decoders.MessageReader;
+import org.apache.drill.exec.store.kafka.decoders.MessageReaderFactory;
+import org.apache.drill.exec.store.schedule.AffinityCreator;
+import org.apache.drill.exec.store.schedule.AssignmentCreator;
+import org.apache.drill.exec.store.schedule.CompleteWork;
+import org.apache.drill.exec.store.schedule.EndpointByteMap;
+import org.apache.drill.exec.store.schedule.EndpointByteMapImpl;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.Node;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+@JsonTypeName("kafka-scan")
+public class KafkaGroupScan extends AbstractGroupScan {
+
+  static final Logger logger = 
LoggerFactory.getLogger(KafkaGroupScan.class);
+  private static final long MSG_SIZE = 1024;
--- End diff --

Please add a comment to explain that this is the assumed message size for 
purposes of computing data size. (At least, that's how it appears to be used 
below...)


> Kafka storage plugin support
> 
>
> Key: DRILL-4779
> URL: https://issues.apache.org/jira/browse/DRILL-4779
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Affects Versions: 1.11.0
>Reporter: B Anil Kumar
>Assignee: B Anil Kumar
>  Labels: doc-impacting
> Fix For: 1.12.0
>
>
> Implement Kafka storage plugin will enable the strong SQL support for Kafka.
> Initially implementation can target for supporting json and avro message types



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4779) Kafka storage plugin support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244584#comment-16244584
 ] 

ASF GitHub Bot commented on DRILL-4779:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1027#discussion_r149763028
  
--- Diff: 
contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaGroupScan.java
 ---
@@ -0,0 +1,301 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.kafka;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.EndpointAffinity;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.ScanStats.GroupScanProperty;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.kafka.KafkaSubScan.KafkaSubScanSpec;
+import org.apache.drill.exec.store.kafka.decoders.MessageReader;
+import org.apache.drill.exec.store.kafka.decoders.MessageReaderFactory;
+import org.apache.drill.exec.store.schedule.AffinityCreator;
+import org.apache.drill.exec.store.schedule.AssignmentCreator;
+import org.apache.drill.exec.store.schedule.CompleteWork;
+import org.apache.drill.exec.store.schedule.EndpointByteMap;
+import org.apache.drill.exec.store.schedule.EndpointByteMapImpl;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.Node;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+@JsonTypeName("kafka-scan")
+public class KafkaGroupScan extends AbstractGroupScan {
+
+  static final Logger logger = 
LoggerFactory.getLogger(KafkaGroupScan.class);
+  private static final long MSG_SIZE = 1024;
+
+  private KafkaStoragePlugin kafkaStoragePlugin;
+  private KafkaStoragePluginConfig kafkaStoragePluginConfig;
+  private List columns;
+  private KafkaScanSpec kafkaScanSpec;
+
+  private List partitionWorkList;
+  private ListMultimap assignments;
+  private List affinities;
+
+  @JsonCreator
+  public KafkaGroupScan(@JsonProperty("userName") String userName,
+  @JsonProperty("kafkaStoragePluginConfig") KafkaStoragePluginConfig 
kafkaStoragePluginConfig,
+  @JsonProperty("columns") List columns, 
@JsonProperty("scanSpec") KafkaScanSpec scanSpec,
+  @JacksonInject StoragePluginRegistry pluginRegistry) {
+this(userName, kafkaStoragePluginConfig, columns, scanSpec, 
(KafkaStoragePlugin) pluginRegistry);
+  }
+
+  public KafkaGroupScan(KafkaStoragePlugin kafkaStoragePlugin, 
KafkaScanSpec kafkaScanSpec, List columns) {
+super(StringUtils.EMPTY);
+this.kafkaStoragePlugin = kafkaStoragePlugin;
+this.kafkaStoragePluginConfig = (KafkaStoragePluginConfig) 

[jira] [Commented] (DRILL-4779) Kafka storage plugin support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244582#comment-16244582
 ] 

ASF GitHub Bot commented on DRILL-4779:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1027#discussion_r149757092
  
--- Diff: contrib/storage-kafka/pom.xml ---
@@ -0,0 +1,130 @@
+
--- End diff --

@vrozov, you are looking into Drill's Maven files. Can you review this new 
one as well?


> Kafka storage plugin support
> 
>
> Key: DRILL-4779
> URL: https://issues.apache.org/jira/browse/DRILL-4779
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Affects Versions: 1.11.0
>Reporter: B Anil Kumar
>Assignee: B Anil Kumar
>  Labels: doc-impacting
> Fix For: 1.12.0
>
>
> Implement Kafka storage plugin will enable the strong SQL support for Kafka.
> Initially implementation can target for supporting json and avro message types



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4779) Kafka storage plugin support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244585#comment-16244585
 ] 

ASF GitHub Bot commented on DRILL-4779:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1027#discussion_r149770390
  
--- Diff: 
contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaRecordReader.java
 ---
@@ -0,0 +1,178 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.kafka;
+
+import static 
org.apache.drill.exec.store.kafka.DrillKafkaConfig.DRILL_KAFKA_POLL_TIMEOUT;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.kafka.KafkaSubScan.KafkaSubScanSpec;
+import org.apache.drill.exec.store.kafka.decoders.MessageReader;
+import org.apache.drill.exec.store.kafka.decoders.MessageReaderFactory;
+import org.apache.drill.exec.util.Utilities;
+import org.apache.drill.exec.vector.complex.impl.VectorContainerWriter;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.TopicPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Stopwatch;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+public class KafkaRecordReader extends AbstractRecordReader {
--- End diff --

Minor point: maybe add a blank line or two between imports and class to 
improve readability.


> Kafka storage plugin support
> 
>
> Key: DRILL-4779
> URL: https://issues.apache.org/jira/browse/DRILL-4779
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Affects Versions: 1.11.0
>Reporter: B Anil Kumar
>Assignee: B Anil Kumar
>  Labels: doc-impacting
> Fix For: 1.12.0
>
>
> Implement Kafka storage plugin will enable the strong SQL support for Kafka.
> Initially implementation can target for supporting json and avro message types



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4779) Kafka storage plugin support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244583#comment-16244583
 ] 

ASF GitHub Bot commented on DRILL-4779:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1027#discussion_r149757481
  
--- Diff: 
contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/DrillKafkaConfig.java
 ---
@@ -0,0 +1,31 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.kafka;
+
+public class DrillKafkaConfig {
+
+  /**
+   * Timeout for fetching messages from Kafka
--- End diff --

Maybe clarify if this is a Drill config property, or a Drill system/session 
option.


> Kafka storage plugin support
> 
>
> Key: DRILL-4779
> URL: https://issues.apache.org/jira/browse/DRILL-4779
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Affects Versions: 1.11.0
>Reporter: B Anil Kumar
>Assignee: B Anil Kumar
>  Labels: doc-impacting
> Fix For: 1.12.0
>
>
> Implement Kafka storage plugin will enable the strong SQL support for Kafka.
> Initially implementation can target for supporting json and avro message types



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5691) multiple count distinct query planning error at physical phase

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244567#comment-16244567
 ] 

ASF GitHub Bot commented on DRILL-5691:
---

Github user amansinha100 commented on the issue:

https://github.com/apache/drill/pull/889
  
@weijietong I ran the functional tests with the PR and saw some failures.  
While debugging those failures, I found the simple scalar aggregate case works 
but anything with compound aggregate expressions  throws a CannotPlanException, 
for example:  

explain plan for select count(*) from cp.`tpch/lineitem.parquet` l where 
l_quantity < (select max(l2.l_quantity) + 1 from cp.`tpch/lineitem.parquet` l2)
 
Here, the right side of the inequality join is actually a scalar but the 
JoinUtils.isScalar() method does not detect it correctly.  The reason is the 
right side is a DrillProjectRel whose input is a RelSubSet.  The RelSubSet has 
a DrillAggregateRel but it looks like currently Calcite does not detect this is 
a scalar aggregate if it occurs as part of a RelSubSet.   I see Calcite has a 
comment in [1] which points to an open JIRA CALCITE-1048.  

[1] 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/metadata/RelMdMaxRowCount.java#L165


> multiple count distinct query planning error at physical phase 
> ---
>
> Key: DRILL-5691
> URL: https://issues.apache.org/jira/browse/DRILL-5691
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.9.0, 1.10.0
>Reporter: weijie.tong
>
> I materialized the count distinct query result in a cache , added a plugin 
> rule to translate the (Aggregate、Aggregate、Project、Scan) or 
> (Aggregate、Aggregate、Scan) to (Project、Scan) at the PARTITION_PRUNING phase. 
> Then ,once user issue count distinct queries , it will be translated to query 
> the cache to get the result.
> eg1: " select count(*),sum(a) ,count(distinct b)  from t where dt=xx " 
> eg2:"select count(*),sum(a) ,count(distinct b) ,count(distinct c) from t 
> where dt=xxx "
> eg3:"select count(distinct b), count(distinct c) from t where dt=xxx"
> eg1 will be right and have a query result as I expected , but eg2 will be 
> wrong at the physical phase.The error info is here: 
> https://gist.github.com/weijietong/1b8ed12db9490bf006e8b3fe0ee52269. 
> eg3 will also get the similar error.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5899) Simple pattern matchers can work with DrillBuf directly

[Padma Penumarthy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Padma Penumarthy updated DRILL-5899:

Labels: ready-to-commit  (was: )

> Simple pattern matchers can work with DrillBuf directly
> ---
>
> Key: DRILL-5899
> URL: https://issues.apache.org/jira/browse/DRILL-5899
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Padma Penumarthy
>Assignee: Padma Penumarthy
>Priority: Critical
>  Labels: ready-to-commit
>
> For the 4 simple patterns we have i.e. startsWith, endsWith, contains and 
> constant,, we do not need the overhead of charSequenceWrapper. We can work 
> with DrillBuf directly. This will save us from doing isAscii check and UTF8 
> decoding for each row.
> UTF-8 encoding ensures that no UTF-8 character is a prefix of any other valid 
> character. So, instead of decoding varChar from each row we are processing, 
> encode the patternString once during setup and do raw byte comparison. 
> Instead of bounds checking and reading one byte at a time, we get the whole 
> buffer in one shot and use that for comparison.
> This improved overall performance for filter operator by around 20%. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-4779) Kafka storage plugin support

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-4779:

Reviewer: Paul Rogers

> Kafka storage plugin support
> 
>
> Key: DRILL-4779
> URL: https://issues.apache.org/jira/browse/DRILL-4779
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Affects Versions: 1.11.0
>Reporter: B Anil Kumar
>Assignee: B Anil Kumar
>  Labels: doc-impacting
> Fix For: 1.12.0
>
>
> Implement Kafka storage plugin will enable the strong SQL support for Kafka.
> Initially implementation can target for supporting json and avro message types



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5944) Single corrupt compressed json file (in s3) causes query failure

[Paul Makkar (JIRA)]

Paul Makkar created DRILL-5944:
--

 Summary: Single corrupt compressed json file (in s3) causes query 
failure
 Key: DRILL-5944
 URL: https://issues.apache.org/jira/browse/DRILL-5944
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - JDBC
Affects Versions: 1.11.0
 Environment: 5 node x 8GB Drill cluster deployed on Kubernetes.
Reporter: Paul Makkar


I am running a CTAS query on an s3 bucket with 500 compressed json files - 
ouput as parquet.

Query ran from command line:
/opt/drill/apache-drill-1.11.0/bin/sqlline --verbose=true --showWarnings=true 
--showNestedErrs=true --force=true --run=therm.sql -u 
jdbc:drill:zk-k8s-drill:2181

therm.sql:
use `s3`.`drill-output`; *(s3 points to kairos bucket)
alter session set `store.format`='parquet';
ALTER SESSION SET `store.json.all_text_mode` = true;
create table temps_bucket0 as select t.id, t.`value` as temp, 
to_timestamp(cast(substr(t.`timestamp`,1,10) as int)) as ts, t.device_id from 
`s3`.`bucket=0/` as t where cast(t.`timestamp` as int) > 147528 and 
cast(t.`timestamp` as int) < 1491004799;

Drill ran for 17 min 50.246 sec and managed to write approx. 100M records then 
failed with the following message (see below). I tried to download and 
uncompress the file manually and it is corrupt. Ideally, Drill should log but 
skip the corrupt file.

Error: DATA_READ ERROR: Error parsing JSON - Unexpected character ('d' (code 
100)): was expecting comma to separate OBJECT entries

File  /bucket=0/190273.json.gz
Record  3654965
Column  37
Fragment 1:0

[Error Id: 9458cb2c-d0a4-4b66-9b65-4e8015e2ca97 on 10.75.186.7 :31010] 
(state=,code=0)
java.sql.SQLException: DATA_READ ERROR: Error parsing JSON - Unexpected 
character ('d' (code 100)): was expecting comma to separate OBJECT entries

File  /bucket=0/190273.json.gz
Record  3654965
Column  37
Fragment 1:0

[Error Id: 9458cb2c-d0a4-4b66-9b65-4e8015e2ca97 on 10.75.186.7 :31010]
at 
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489)
at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:593)
at 
org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:215)
at 
org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:140)
at sqlline.IncrementalRows.hasNext(IncrementalRows.java:62)
at 
sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
at sqlline.SqlLine.print(SqlLine.java:1593)
at sqlline.Commands.execute(Commands.java:852)
at sqlline.Commands.sql(Commands.java:751)
at sqlline.SqlLine.dispatch(SqlLine.java:746)
at sqlline.SqlLine.runCommands(SqlLine.java:1651)
at sqlline.Commands.run(Commands.java:1304)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:553)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.common.exceptions.UserRemoteException: DATA_READ 
ERROR: Error parsing JSON - Unexpected character ('d' (code 100)): was 
expecting comma to separate OBJECT entries

File  /bucket=0/190273.json.gz
Record  3654965
Column  37
Fragment 1:0

[Error Id: 9458cb2c-d0a4-4b66-9b65-4e8015e2ca97 on 10.75.186.7 :31010]
at 
org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:368)
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:90)
at 
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274)
at 
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:244)
at 
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at 
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at 

[jira] [Commented] (DRILL-5717) change some date time unit cases with specific timezone or Local

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243616#comment-16243616
 ] 

ASF GitHub Bot commented on DRILL-5717:
---

Github user vvysotskyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/904#discussion_r149613241
  
--- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/ExecTest.java 
---
@@ -120,4 +124,27 @@ protected LogicalExpression parseExpr(String expr) 
throws RecognitionException {
 return ret.e;
   }
 
+  /**
+   * mock current locale to US
+   */
+  public static void mockUSLocale() {
+new MockUp() {
+  @Mock
+  public DateFormatSymbols getDateFormatSymbols(Locale locale) {
+return new DateFormatSymbols(Locale.US);
+  }
+};
+  }
+
+  /**
+   * mock current timezone to UTC
+   */
+  public static void mockUTCTimezone() {
--- End diff --

Please rename this method to mockUtcDateTimeZone


> change some date time unit cases with specific timezone or Local
> 
>
> Key: DRILL-5717
> URL: https://issues.apache.org/jira/browse/DRILL-5717
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>
> Some date time test cases like  JodaDateValidatorTest  is not Local 
> independent .This will cause other Local's users's test phase to fail. We 
> should let these test cases to be Local env independent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5717) change some date time unit cases with specific timezone or Local

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243617#comment-16243617
 ] 

ASF GitHub Bot commented on DRILL-5717:
---

Github user vvysotskyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/904#discussion_r149616688
  
--- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/ExecTest.java 
---
@@ -120,4 +124,27 @@ protected LogicalExpression parseExpr(String expr) 
throws RecognitionException {
 return ret.e;
   }
 
+  /**
+   * mock current locale to US
+   */
+  public static void mockUSLocale() {
+new MockUp() {
+  @Mock
+  public DateFormatSymbols getDateFormatSymbols(Locale locale) {
+return new DateFormatSymbols(Locale.US);
+  }
+};
+  }
+
+  /**
+   * mock current timezone to UTC
--- End diff --

Please start the sentence with a capital letter and add a description about 
the method which is mocked.


> change some date time unit cases with specific timezone or Local
> 
>
> Key: DRILL-5717
> URL: https://issues.apache.org/jira/browse/DRILL-5717
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>
> Some date time test cases like  JodaDateValidatorTest  is not Local 
> independent .This will cause other Local's users's test phase to fail. We 
> should let these test cases to be Local env independent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5717) change some date time unit cases with specific timezone or Local

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243614#comment-16243614
 ] 

ASF GitHub Bot commented on DRILL-5717:
---

Github user vvysotskyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/904#discussion_r149612076
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestDateFunctions.java
 ---
@@ -33,128 +33,140 @@
 import org.apache.drill.exec.server.Drillbit;
 import org.apache.drill.exec.server.RemoteServiceSet;
 import org.apache.drill.exec.vector.ValueVector;
+import org.joda.time.DateTimeUtils;
 import org.joda.time.LocalDate;
-import org.joda.time.LocalTime;
 import org.joda.time.LocalDateTime;
+import org.joda.time.LocalTime;
 import org.junit.Ignore;
 import org.junit.Test;
-
-import com.google.common.base.Charsets;
-import com.google.common.io.Files;
 import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import java.text.DateFormatSymbols;
+import java.util.List;
+import java.util.Locale;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
 
+@RunWith(JMockit.class)
 @Category({UnlikelyTest.class, SqlFunctionTest.class})
 public class TestDateFunctions extends PopUnitTestBase {
-static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TestDateFunctions.class);
+  static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TestDateFunctions.class);
 
-public void testCommon(String[] expectedResults, String physicalPlan, 
String resourceFile) throws Exception {
-try (RemoteServiceSet serviceSet = 
RemoteServiceSet.getLocalServiceSet();
- Drillbit bit = new Drillbit(CONFIG, serviceSet);
- DrillClient client = new DrillClient(CONFIG, 
serviceSet.getCoordinator())) {
 
-// run query.
-bit.run();
-client.connect();
-List results = 
client.runQuery(org.apache.drill.exec.proto.UserBitShared.QueryType.PHYSICAL,
-
Files.toString(FileUtils.getResourceAsFile(physicalPlan), Charsets.UTF_8)
-.replace("#{TEST_FILE}", resourceFile));
+  public void testCommon(String[] expectedResults, String physicalPlan, 
String resourceFile) throws Exception {
+try (RemoteServiceSet serviceSet = 
RemoteServiceSet.getLocalServiceSet();
+ Drillbit bit = new Drillbit(CONFIG, serviceSet);
+ DrillClient client = new DrillClient(CONFIG, 
serviceSet.getCoordinator())) {
 
-RecordBatchLoader batchLoader = new 
RecordBatchLoader(bit.getContext().getAllocator());
+  // run query.
+  bit.run();
+  client.connect();
+  List results = 
client.runQuery(org.apache.drill.exec.proto.UserBitShared.QueryType.PHYSICAL,
+Files.toString(FileUtils.getResourceAsFile(physicalPlan), 
Charsets.UTF_8)
+  .replace("#{TEST_FILE}", resourceFile));
 
-QueryDataBatch batch = results.get(0);
-assertTrue(batchLoader.load(batch.getHeader().getDef(), 
batch.getData()));
+  RecordBatchLoader batchLoader = new 
RecordBatchLoader(bit.getContext().getAllocator());
 
+  QueryDataBatch batch = results.get(0);
+  assertTrue(batchLoader.load(batch.getHeader().getDef(), 
batch.getData()));
 
-int i = 0;
-for (VectorWrapper v : batchLoader) {
 
-ValueVector.Accessor accessor = 
v.getValueVector().getAccessor();
-System.out.println(accessor.getObject(0));
-assertEquals( expectedResults[i++], 
accessor.getObject(0).toString());
-}
-
-batchLoader.clear();
-for(QueryDataBatch b : results){
-b.release();
-}
-}
-}
-
-@Test
-@Ignore("relies on particular timezone")
-public void testDateIntervalArithmetic() throws Exception {
-String expectedResults[] = {"2009-02-23T00:00:00.000-08:00",
-"2008-02-24T00:00:00.000-08:00",
-"1970-01-01T13:20:33.000-08:00",
-"2008-02-24T12:00:00.000-08:00",
-"2009-04-23T12:00:00.000-07:00",
-"2008-02-24T12:00:00.000-08:00",
-"2009-04-23T12:00:00.000-07:00",
-"2009-02-23T00:00:00.000-08:00",
-"2008-02-24T00:00:00.000-08:00",
-  

[jira] [Commented] (DRILL-5717) change some date time unit cases with specific timezone or Local

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243613#comment-16243613
 ] 

ASF GitHub Bot commented on DRILL-5717:
---

Github user vvysotskyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/904#discussion_r149614134
  
--- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/ExecTest.java 
---
@@ -120,4 +124,27 @@ protected LogicalExpression parseExpr(String expr) 
throws RecognitionException {
 return ret.e;
   }
 
+  /**
+   * mock current locale to US
+   */
+  public static void mockUSLocale() {
--- End diff --

Please rename this method to mockUsDateFormatSymbols


> change some date time unit cases with specific timezone or Local
> 
>
> Key: DRILL-5717
> URL: https://issues.apache.org/jira/browse/DRILL-5717
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>
> Some date time test cases like  JodaDateValidatorTest  is not Local 
> independent .This will cause other Local's users's test phase to fail. We 
> should let these test cases to be Local env independent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5717) change some date time unit cases with specific timezone or Local

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243615#comment-16243615
 ] 

ASF GitHub Bot commented on DRILL-5717:
---

Github user vvysotskyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/904#discussion_r149615891
  
--- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/ExecTest.java 
---
@@ -120,4 +124,27 @@ protected LogicalExpression parseExpr(String expr) 
throws RecognitionException {
 return ret.e;
   }
 
+  /**
+   * mock current locale to US
--- End diff --

Please start the sentence with a capital letter and add a description about 
the method which is mocked.


> change some date time unit cases with specific timezone or Local
> 
>
> Key: DRILL-5717
> URL: https://issues.apache.org/jira/browse/DRILL-5717
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>
> Some date time test cases like  JodaDateValidatorTest  is not Local 
> independent .This will cause other Local's users's test phase to fail. We 
> should let these test cases to be Local env independent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5923) State of a successfully completed query shown as "COMPLETED"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243510#comment-16243510
 ] 

ASF GitHub Bot commented on DRILL-5923:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1021#discussion_r149596636
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/QueryStateDisplayName.java
 ---
@@ -0,0 +1,35 @@
+/**
--- End diff --

Please use comment, not Java doc for header.


> State of a successfully completed query shown as "COMPLETED"
> 
>
> Key: DRILL-5923
> URL: https://issues.apache.org/jira/browse/DRILL-5923
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Prasad Nagaraj Subramanya
>Assignee: Prasad Nagaraj Subramanya
> Fix For: 1.12.0
>
>
> Drill UI currently lists a successfully completed query as "COMPLETED". 
> Successfully completed, failed and canceled queries are all grouped as 
> Completed queries. 
> It would be better to list the state of a successfully completed query as 
> "Succeeded" to avoid confusion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5923) State of a successfully completed query shown as "COMPLETED"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243513#comment-16243513
 ] 

ASF GitHub Bot commented on DRILL-5923:
---

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/1021
  
@paul-rogers totally agree with your comments about public API (slipped 
from my mind when I was writing comment yesterday). I guess then it's fine to 
go with changes Prasad suggests, I just have left a couple of suggestions to 
make code clearer.


> State of a successfully completed query shown as "COMPLETED"
> 
>
> Key: DRILL-5923
> URL: https://issues.apache.org/jira/browse/DRILL-5923
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Prasad Nagaraj Subramanya
>Assignee: Prasad Nagaraj Subramanya
> Fix For: 1.12.0
>
>
> Drill UI currently lists a successfully completed query as "COMPLETED". 
> Successfully completed, failed and canceled queries are all grouped as 
> Completed queries. 
> It would be better to list the state of a successfully completed query as 
> "Succeeded" to avoid confusion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5923) State of a successfully completed query shown as "COMPLETED"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243508#comment-16243508
 ] 

ASF GitHub Bot commented on DRILL-5923:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1021#discussion_r149596833
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/QueryStateDisplayName.java
 ---
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.server.rest.profile;
+
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+
+public class QueryStateDisplayName {
--- End diff --

Let's name this class more generically since in future we might wan't add 
some other helper methods as well. 
For example, `ProfileUtil`.


> State of a successfully completed query shown as "COMPLETED"
> 
>
> Key: DRILL-5923
> URL: https://issues.apache.org/jira/browse/DRILL-5923
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Prasad Nagaraj Subramanya
>Assignee: Prasad Nagaraj Subramanya
> Fix For: 1.12.0
>
>
> Drill UI currently lists a successfully completed query as "COMPLETED". 
> Successfully completed, failed and canceled queries are all grouped as 
> Completed queries. 
> It would be better to list the state of a successfully completed query as 
> "Succeeded" to avoid confusion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5923) State of a successfully completed query shown as "COMPLETED"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243509#comment-16243509
 ] 

ASF GitHub Bot commented on DRILL-5923:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1021#discussion_r149596974
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/QueryStateDisplayName.java
 ---
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.server.rest.profile;
+
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+
+public class QueryStateDisplayName {
+  // Values should correspond to the QueryState enum in UserBitShared.proto
--- End diff --

Not quite right, value do not correspond to QueryState enum, it's rather a 
mapping to QueryState enum. To be more clear here I suggest you create a map 
with `QueryState` as a key and `String` as value:
```
STARTING -> Starting
COMPLETED -> Succeeded
```
Thus you'll be able to get desired representation value by `QueryState` and 
in case if value is null return some default, like `Unknown State`.


> State of a successfully completed query shown as "COMPLETED"
> 
>
> Key: DRILL-5923
> URL: https://issues.apache.org/jira/browse/DRILL-5923
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Prasad Nagaraj Subramanya
>Assignee: Prasad Nagaraj Subramanya
> Fix For: 1.12.0
>
>
> Drill UI currently lists a successfully completed query as "COMPLETED". 
> Successfully completed, failed and canceled queries are all grouped as 
> Completed queries. 
> It would be better to list the state of a successfully completed query as 
> "Succeeded" to avoid confusion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)