[jira] [Created] (DRILL-6383) View column types, modes are plan-time guesses, not actual types

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-6383:
--

 Summary: View column types, modes are plan-time guesses, not 
actual types
 Key: DRILL-6383
 URL: https://issues.apache.org/jira/browse/DRILL-6383
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.13.0
Reporter: Paul Rogers


Create a view views and look at the list of columns within the view. You'll see 
that they are often wrong in name, type and mode.

Consider a very simple CSV file with headers:

{noformat}
custId,name,balance,status
123,Fred,456.78
125,Betty,98.76,VIP
128,Barney,1.23,PAST DUE,30
{noformat}

Define the simplest possible view:

{noformat}
CREATE VIEW myView2 AS SELECT * FROM `csvh/cust.csvh`;
{noformat}

Then look at the view file:

{noformat}
{
  "name" : "myView2",
  "sql" : "SELECT *\nFROM `csvh/cust.csvh`",
  "fields" : [ {
"name" : "**",
"type" : "DYNAMIC_STAR",
"isNullable" : true
  } ],
  "workspaceSchemaPath" : [ "local", "data" ]
}
{noformat}

It is clear that the view simply captured the plan-time list of the new 
double-star for the wildcard. Since this is not a true type, it should not have 
an `isNullable` attribute.

OK, we have to spell out the columns:

{noformat}
CREATE VIEW myView3 AS SELECT custId  FROM `csvh/cust.csvh`;
{noformat}

Let's look at the view file:

{noformat}
{
  "name" : "myView3",
  "sql" : "SELECT `custId`\nFROM `csvh/cust.csvh`",
  "fields" : [ {
"name" : "custId",
"type" : "ANY",
"isNullable" : true
  } ],
  "workspaceSchemaPath" : [ "local", "data" ]
}
{noformat}

The name is correct. The type is `ANY`, which is wrong. Since this is a CSV 
file, the column type is `VARCHAR`. Further, because this is a CSV file which 
headers, the mode is REQUIRED, but is listed as nullable. To verify:

{noformat}
SELECT sqlTypeOf(custId), modeOf(custId) FROM myView3 LIMIT 1;
++---+
|   EXPR$0   |  EXPR$1   |
++---+
| CHARACTER VARYING  | NOT NULL  |
++---+
{noformat}

Now, let's try a CSV file without headers:

{noformat}
123,Fred,456.78
125,Betty,98.76,VIP
{noformat}

{noformat}
CREATE VIEW myView4 AS SELECT columns FROM `csv/cust.csv`;
SELECT * FROM myView4;
++
|columns |
++
| ["123","Fred","456.78"]|
| ["125","Betty","98.76","VIP"]  |
++
{noformat}

Let's look at the view file:

{noformat}
{
  "name" : "myView4",
  "sql" : "SELECT `columns`\nFROM `csv/cust.csv`",
  "fields" : [ {
"name" : "columns",
"type" : "ANY",
"isNullable" : true
  } ],
  "workspaceSchemaPath" : [ "local", "data" ]
}
{noformat}

This is almost non-sensical. `columns` is reported as type `ANY` and nullable. 
But, `columns` is Repeated `VARCHAR` and repeated types cannot be nullable.

The conclusion is that the type information is virtually worthless and the 
`isNullable` information is worse than worthless: it is plain wrong.

The type information is valid only if the planner can inver types:

{noformat}
CREATE VIEW myView5 AS
  SELECT CAST(custId AS INTEGER) AS custId FROM `csvh/cust.csvh`;
{noformat}

The view file:

{noformat}
{
  "name" : "myView5",
  "sql" : "SELECT CAST(`custId` AS INTEGER) AS `custId`\nFROM `csvh/cust.csvh`",
  "fields" : [ {
"name" : "custId",
"type" : "INTEGER",
"isNullable" : true
  } ],
  "workspaceSchemaPath" : [ "local", "data" ]
}
{noformat}

Note that the `type` is inferred from the cast, but `isNullable` is wrong 
because the underlying column is non-nullable:

{noformat}
SELECT modeOf(custId) FROM myView5 LIMIT 1;
+---+
|  EXPR$0   |
+---+
| NOT NULL  |
+---+
{noformat}

Expected that Drill would run the underlying query as a `LIMIT 0` query to 
extract the actual column types, and use that in the view.

Or, expected that Drill would simply omit the column list from the view if the 
data is meaningless.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6382) Views should "close" over options in effect at view creation time

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-6382:
--

 Summary: Views should "close" over options in effect at view 
creation time
 Key: DRILL-6382
 URL: https://issues.apache.org/jira/browse/DRILL-6382
 Project: Apache Drill
  Issue Type: Improvement
Affects Versions: 1.13.0
Reporter: Paul Rogers


Suppose I have the following JSON file:

{noformat}
{a: [ 123, "Fred", 123.45 ] }
{noformat}

Suppose I query the list with default options in place:

{noformat}
SELECT * FROM `json/scalar-list.json`;

Error: UNSUPPORTED_OPERATION ERROR: In a list of type BIGINT, 
  encountered a value of type VARCHAR. Drill does not support lists of 
different types.
{noformat}

Well, foo. The JSON contains a mixed scalar list. Luckily, I know about 
all-text mode:

{noformat}
ALTER SESSION SET `store.json.all_text_mode` = true;
SELECT * FROM `json/scalar-list.json`;
+--+
|a |
+--+
| ["123","Fred","123.45"]  |
+--+
{noformat}

No I can make a fancy query:

{noformat}
SELECT CAST(a[0] AS INT) AS custId,
 a[1] AS name,
 CAST(a[2] AS DOUBLE) AS balance
  FROM `json/scalar-list.json`;
+-+---+--+
| custId  | name  | balance  |
+-+---+--+
| 123 | Fred  | 123.45   |
+-+---+--+
{noformat}

And I can package up my query as a view:

{noformat}
CREATE VIEW myView AS 
  SELECT CAST(a[0] AS INT) AS custId,
   a[1] AS name,
   CAST(a[2] AS DOUBLE) AS balance
FROM `json/scalar-list.json`;
{noformat}

Let's test the view:

{noformat}
SELECT * FROM myView;
+-+---+--+
| custId  | name  | balance  |
+-+---+--+
| 123 | Fred  | 123.45   |
+-+---+--+
{noformat}

Next, let's try out the view the way that the user will: with default options:

{noformat}
ALTER SESSION RESET  `store.json.all_text_mode`;
SELECT * FROM myView;
Error: UNSUPPORTED_OPERATION ERROR: In a list of type BIGINT,
  encountered a value of type VARCHAR. Drill does not support lists of 
different types.
{noformat}

Oh, no! What happened? Let's check the view file:

{noformat}
{
  "name" : "myView",
  "sql" : "SELECT CAST(`a`[0] AS INTEGER) AS `custId`, `a`[1] AS `name`, 
CAST(`a`[2] AS DOUBLE) AS `balance`\nFROM `json/scalar-list.json`",
  "fields" : [ {
"name" : "custId",
"type" : "INTEGER",
"isNullable" : true
  }, {
"name" : "name",
"type" : "ANY",
"isNullable" : true
  }, {
"name" : "balance",
"type" : "DOUBLE",
"isNullable" : true
  } ],
  "workspaceSchemaPath" : [ "local", "data" ]
}
{noformat}

We can see from the file that the view captures the schema in effect at view 
creation, but it does *not* capture options in effect when the view was made. 
The user must remember to set the options.

Requested feature: capture the options in a new JSON tag in the view file. Pass 
those options along to operators created for this view. If any of the inputs to 
the view are views, then the inner view options override the outer view options.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6381) Add capability to do index based planning and execution

[Aman Sinha (JIRA)]

Aman Sinha created DRILL-6381:
-

 Summary: Add capability to do index based planning and execution
 Key: DRILL-6381
 URL: https://issues.apache.org/jira/browse/DRILL-6381
 Project: Apache Drill
  Issue Type: New Feature
  Components: Execution - Relational Operators, Query Planning  
Optimization
Reporter: Aman Sinha
Assignee: Aman Sinha


If the underlying data source supports indexes (primary and secondary indexes), 
Drill should leverage those during planning and execution in order to improve 
query performance.  

On the planning side, Drill planner should be enhanced to provide an 
abstraction layer which express the index metadata and statistics.  Further, a 
cost-based index selection is needed to decide which index(es) are suitable.  

On the execution side, appropriate operator enhancements would be needed to 
handle different categories of indexes such as covering, non-covering indexes, 
taking into consideration the index data may not be co-located with the primary 
table, i.e a global index.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: gitbox?

[Vlad Rozov]

All-

Drill was moved to gitbox. Please see update to 
https://issues.apache.org/jira/browse/INFRA-16411.


Committers and PMC, please note a one time action required on your side 
to enable write access on github.


Thank you,

Vlad

On 4/20/18 15:40, Parth Chandra wrote:

Created INFRA request: https://issues.apache.org/jira/browse/INFRA-16411

On Wed, Apr 18, 2018 at 10:48 AM, Parth Chandra  wrote:


Great.
Just as an additional clarification, once gitbox is enabled, we will not
need to have a remote set up to point to Apache hosted repository (you can
still point to the Apache github mirror) as it will no longer be
accessible.
All commits will be done via github.

On Wed, Apr 18, 2018 at 10:27 AM, Aman Sinha  wrote:


Yeah, let's go ahead with this.  I don't quite recall Vlad's explanations
in the hangout back in October, but I think we were all convinced, so I am
+1.

On Wed, Apr 18, 2018 at 3:42 AM, Arina Yelchiyeva <
arina.yelchiy...@gmail.com> wrote:


Thanks, Parth, that would be really helpful.

On Wed, Apr 18, 2018 at 4:38 AM, Parth Chandra 

wrote:

Hi Drill devs

   If no one has any objections I will open the Apache infra request to

move

to gitbox. Once this is setup, committers will be able to merge/commit
without ever leaving github.

Thanks

Parth

On Sun, Nov 12, 2017 at 8:00 AM, Kunal Khatua 

wrote:

My bad... I was trying to go deeper into the specifics of GitBox via
Google and mostly client related results came up.

Thanks!

-Original Message-
From: Uwe L. Korn [mailto:uw...@xhochy.com]
Sent: Sunday, November 12, 2017 3:20 AM
To: dev@drill.apache.org
Subject: Re: gitbox?

Note that this discussion is about the new Apache server-side Git

services

https://gitbox.apache.org/ and not about any specific client.

We are very happy with it in the Arrow and I can recommend

switching to

any other Apache project as soon as possible.

Uwe


Am 12.11.2017 um 09:08 schrieb Kunal Khatua :

Has anyone tried GitKraken? It's a cross platform client that's

proven

to be pretty reliable for me for close to a year.

My concern is that GitBox is exclusive to running on Mac.

-Original Message-
From: Parth Chandra [mailto:par...@apache.org]
Sent: Tuesday, October 31, 2017 2:52 PM
To: dev 
Subject: Re: gitbox?

Gitbox allows committers to streamline the review and merge

process.

It

provides a single button in github to merge pull requests to the

Apache

Drill mirror on github. This is then synchronized seamlessly with

the

Apache master.

The process would still require a committer to 1) review code, 2)

run

the functional tests if doing a batch commit.

Many other Apache projects have already moved to using gitbox.




On Tue, Oct 31, 2017 at 11:25 AM, Kunal Khatua 
Subject: Re: gitbox?

Bumping this thread up.

Vlad brought this up in the hangout today and it sounds like we

would

like to move to Gitbox. Thanks Vlad for the patient explanations!

Committers, let's use this thread to vote on the the suggestion.

I'm +1 on moving to gitbox.

Also, I can work with Vlad and Paul on updating the merge process

document.




On Wed, Aug 30, 2017 at 1:34 PM, Vlad Rozov 

wrote:

Hi,

As I am new to Drill, I don't know if migration from "Git WiP" (
https://git-wip-us.apache.org) to "Github Dual Master" (
https://gitbox.apache.org) was already discussed by the

community,

but from my Apache Apex experience I would recommend to consider
migrating Drill ASF repos to the gitbox. Such move will give
committers write access to the Drill repository on Github with

all

the perks that Github

provides.

Thank you,

Vlad

[jira] [Created] (DRILL-6380) Mongo db storage plugin tests can hang on jenkins.

[Timothy Farkas (JIRA)]

Timothy Farkas created DRILL-6380:
-

 Summary: Mongo db storage plugin tests can hang on jenkins.
 Key: DRILL-6380
 URL: https://issues.apache.org/jira/browse/DRILL-6380
 Project: Apache Drill
  Issue Type: Bug
Reporter: Timothy Farkas
Assignee: Timothy Farkas


When running on our Jenkins server the mongodb tests hang because the Config 
servers take up to 5 seconds to process each request (see *Error 2*). This 
causes the tests to never finish within a reasonable span of time. Searching 
online people run into this issue when mixing versions of mongo db, but that is 
not happening in our tests. A possible cause is *Error 1* which seems to 
indicate that the mongo db config servers are not completely initialized since 
the config servers should have a lockping document when starting up.

*Error 1*

{code}
[mongod output] 2018-05-01T23:38:47.468-0700 I COMMAND  [replSetDistLockPinger] 
command config.lockpings command: findAndModify { findAndModify: "lockpings", 
query: { _id: "ConfigServer" }, update: { $set: { ping: new Date(1525243123413) 
} }, upsert: true, writeConcern: { w: "majority", wtimeout: 15000 } } 
planSummary: IDHACK update: { $set: { ping: new Date(1525243123413) } } 
keysExamined:0 docsExamined:0 nMatched:0 nModified:0 upsert:1 keysInserted:2 
numYields:0 reslen:198 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, 
Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, 
Metadata: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 
protocol:op_query 4055ms
[mongod output] 2018-05-01T23:38:47.469-0700 W SHARDING [replSetDistLockPinger] 
pinging failed for distributed lock pinger :: caused by :: 
LockStateChangeFailed: findAndModify query predicate didn't match any lock 
document
[mongod output] 2018-05-01T23:38:47.498-0700 I SHARDING [Balancer] lock 
'balancer' successfully forced
[mongod output] 2018-05-01T23:38:47.498-0700 I SHARDING [Balancer] distributed 
lock 'balancer' acquired, ts : 5ae95cd5d1023488104e6282
[mongod output] 2018-05-01T23:38:47.498-0700 I SHARDING [Balancer] CSRS 
balancer thread is recovering
[mongod output] 2018-05-01T23:38:47.498-0700 I SHARDING [Balancer] CSRS 
balancer thread is recovered
[mongod output] 2018-05-01T23:38:48.056-0700 I NETWORK  [thread2] connection 
accepted from 127.0.0.1:50244 #10 (7 connections now open)
{code}

*Error 2*

{code}
[mongod output] 2018-05-01T23:39:37.690-0700 I COMMAND  [conn7] command 
config.settings command: find { find: "settings", filter: { _id: "chunksize" }, 
readConcern: { level: "majority", afterOpTime: { ts: Timestamp 1525243172000|1, 
t: 1 } }, limit: 1, maxTimeMS: 3 } planSummary: EOF keysExamined:0 
docsExamined:0 cursorExhausted:1 numYields:0 nreturned:0 reslen:354 locks:{ 
Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, 
Collection: { acquireCount: { r: 1 } } } protocol:op_command 4988ms
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] drill pull request #1225: DRILL-6272: Refactor dynamic UDFs and function ini...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1225#discussion_r185507108
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/coord/zk/TestZookeeperClient.java
 ---
@@ -125,7 +125,7 @@ public void testHasPathThrowsDrillRuntimeException() {
 
 Mockito
 .when(client.getCache().getCurrentData(absPath))
-.thenThrow(Exception.class);
+.thenThrow(RuntimeException.class);
--- End diff --

Let's do it in different PR.


---

[GitHub] drill issue #1201: DRILL-4091: Support for additional gis operations in gis ...

[cgivre]

Github user cgivre commented on the issue:

https://github.com/apache/drill/pull/1201
  
@arina-ielchiieva Should I do the commit or would you like to do that.


---

[GitHub] drill issue #1201: DRILL-4091: Support for additional gis operations in gis ...

[cgivre]

Github user cgivre commented on the issue:

https://github.com/apache/drill/pull/1201
  
In that case,  LGTM +1 
Thank you for your work on this!.  Are you going to take a look at the ESRI 
format plugin?  


---

[GitHub] drill pull request #1225: DRILL-6272: Refactor dynamic UDFs and function ini...

[vrozov]

Github user vrozov commented on a diff in the pull request:

https://github.com/apache/drill/pull/1225#discussion_r185497944
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/coord/zk/TestZookeeperClient.java
 ---
@@ -125,7 +125,7 @@ public void testHasPathThrowsDrillRuntimeException() {
 
 Mockito
 .when(client.getCache().getCurrentData(absPath))
-.thenThrow(Exception.class);
+.thenThrow(RuntimeException.class);
--- End diff --

IMO, this method needs to be changed to test 
`ZookeeperClient.hasPath(String path, boolean consistent)`. It is OK to do it 
in this PR or in a separate commit/JIRA/PR. If you decide to do it in a 
separate commit, please file JIRA.


---

[GitHub] drill issue #1201: DRILL-4091: Support for additional gis operations in gis ...

[ChrisSandison]

Github user ChrisSandison commented on the issue:

https://github.com/apache/drill/pull/1201
  
@cgivre I had some issues between this and a previous branch I was working 
on, but I fixed it by upgrade the base image that I was building it on. I ended 
up building it on maven:3.5.3-jdk-8


---

[GitHub] drill issue #1201: DRILL-4091: Support for additional gis operations in gis ...

[cgivre]

Github user cgivre commented on the issue:

https://github.com/apache/drill/pull/1201
  
LGTM +1

One thing, which I suspect may be a problem with my environment, not this 
PR, but when I try to build the module with the tests, I get the following 
errors:

```
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-assembly-plugin:2.6:single 
(source-release-assembly) on project drill-gis: Error reading assemblies: Error 
locating assembly descriptor: src/main/resources/assemblies/source-assembly.xml
[ERROR] 
[ERROR] [1] [INFO] Searching for file location: 
/Users/cgivre/github/drill-dev/drill-gis/drill/contrib/gis/src/main/resources/assemblies/source-assembly.xml
[ERROR] 
[ERROR] [2] [INFO] File: 
/Users/cgivre/github/drill-dev/drill-gis/drill/contrib/gis/src/main/resources/assemblies/source-assembly.xml
 does not exist.
[ERROR] 
[ERROR] [3] [INFO] File: 
/Users/cgivre/github/drill-dev/drill-gis/drill/contrib/gis/src/main/resources/assemblies/source-assembly.xml
 does not exist.
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
goal org.apache.maven.plugins:maven-assembly-plugin:2.6:single 
(source-release-assembly) on project drill-gis: Error reading assemblies: Error 
locating assembly descriptor: src/main/resources/assemblies/source-assembly.xml

[1] [INFO] Searching for file location: 
/Users/cgivre/github/drill-dev/drill-gis/drill/contrib/gis/src/main/resources/assemblies/source-assembly.xml

[2] [INFO] File: 
/Users/cgivre/github/drill-dev/drill-gis/drill/contrib/gis/src/main/resources/assemblies/source-assembly.xml
 does not exist.

[3] [INFO] File: 
/Users/cgivre/github/drill-dev/drill-gis/drill/contrib/gis/src/main/resources/assemblies/source-assembly.xml
 does not exist.
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:213)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:154)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:146)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:117)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:81)
at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
 (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke 
(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:497)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode 
(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main 
(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Error reading 
assemblies: Error locating assembly descriptor: 
src/main/resources/assemblies/source-assembly.xml

[1] [INFO] Searching for file location: 
/Users/cgivre/github/drill-dev/drill-gis/drill/contrib/gis/src/main/resources/assemblies/source-assembly.xml

[2] [INFO] File: 
/Users/cgivre/github/drill-dev/drill-gis/drill/contrib/gis/src/main/resources/assemblies/source-assembly.xml
 does not exist.

[3] [INFO] File: 
/Users/cgivre/github/drill-dev/drill-gis/drill/contrib/gis/src/main/resources/assemblies/source-assembly.xml
 does not exist.
at org.apache.maven.plugin.assembly.mojos.AbstractAssemblyMojo.execute 
(AbstractAssemblyMojo.java:453)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
(DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:208)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:154)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:146)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:117)
at 

[GitHub] drill pull request #1242: DRILL-6361: Revised typeOf() function versions

[vvysotskyi]

Github user vvysotskyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/1242#discussion_r185458392
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/expr/fn/impl/TestTypeFns.java
 ---
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn.impl;
+
+import static org.junit.Assert.assertEquals;
+
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class TestTypeFns extends ClusterTest {
+
+  @BeforeClass
+  public static void setup() throws Exception {
+// Use the following three lines if you add a function
+// to avoid the need for a full Drill build.
+ClusterFixtureBuilder builder = ClusterFixture.builder(dirTestWatcher)
+.configProperty("drill.classpath.scanning.cache.enabled", false);
+startCluster(builder);
+
+// Use the following line if a full Drill build has been
+// done since adding new functions.
+//
startCluster(ClusterFixture.builder(dirTestWatcher).maxParallelization(1));
+  }
+
+  @Test
+  public void testTypeOf() throws RpcException {
+// SMALLINT not supported in CAST
+//doTypeOfTest("SMALLINT");
+doTypeOfTest("INT");
+doTypeOfTest("BIGINT");
+doTypeOfTest("VARCHAR");
+doTypeOfTest("FLOAT", "FLOAT4");
+doTypeOfTest("DOUBLE", "FLOAT8");
+doTypeOfTestSpecial("a", "true", "BIT");
+doTypeOfTestSpecial("a", "CURRENT_DATE", "DATE");
+doTypeOfTestSpecial("a", "CURRENT_TIME", "TIME");
+doTypeOfTestSpecial("a", "CURRENT_TIMESTAMP", "TIMESTAMP");
+doTypeOfTestSpecial("a", "AGE(CURRENT_TIMESTAMP)", "INTERVAL");
+doTypeOfTestSpecial("BINARY_STRING(a)", "'\\xde\\xad\\xbe\\xef'", 
"VARBINARY");
+try {
+  client.alterSession("planner.enable_decimal_data_type", true);
--- End diff --

Please replace `planner.enable_decimal_data_type` with 
`PlannerSettings.ENABLE_DECIMAL_DATA_TYPE_KEY`


---

[jira] [Created] (DRILL-6379) Pushing additional filters for disjunction of expressions past JOIN

[Vitalii Diravka (JIRA)]

Vitalii Diravka created DRILL-6379:
--

 Summary: Pushing additional filters for disjunction of expressions 
past JOIN
 Key: DRILL-6379
 URL: https://issues.apache.org/jira/browse/DRILL-6379
 Project: Apache Drill
  Issue Type: Improvement
  Components: Query Planning  Optimization
Affects Versions: 1.13.0
Reporter: Vitalii Diravka
Assignee: Vitalii Diravka
 Fix For: Future


For queries with JOIN operator and filter as disjunction of expressions
the additional filters can be derived and pushed down to prevent of unnecessary 
scanning.
Query example:
{code}
SELECT * FROM t1 JOIN t2 ON T1.COLUMN = T2.COLUMN WHERE
(PC = X AND ) OR (PC = Y AND )
{code}

Unit test for _TestParquetFilterPushdownWithTransitivePredicates.java_:
{code}
  @Test
  public void testForOrOperatorTestOr() throws Exception {
String query = String.format("SELECT * FROM %s t1 " +
"JOIN %s t2 ON t1.`month` = t2.`month` " +
"WHERE ((t1.`period` = 4 AND t2.`year` = 1991) OR (t1.`period` = 3 
AND t1.`year` = 1991)) ",
FIRST_TABLE_NAME, SECOND_TABLE_NAME);

final String[] expectedPlan = {"first.*numRowGroups=2", 
"second.*numRowGroups=1"};
testPlanMatchingPatterns(query, expectedPlan);
  }
{code}

{code}
LogicalProject(**=[$0], **0=[$4])
  LogicalFilter(condition=[OR(AND(=($2, 4), =($6, 1991)), AND(=($2, 3), =($3, 
1991)))])
LogicalJoin(condition=[=($1, $5)], joinType=[inner])
  EnumerableTableScan(table=[[dfs, 
parquetFilterPush/transitiveClosure/first]])
  EnumerableTableScan(table=[[dfs, 
parquetFilterPush/transitiveClosure/second]])
{code}

This improvement can be solved by CALCITE-2296.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] drill issue #1225: DRILL-6272: Refactor dynamic UDFs and function initialize...

[arina-ielchiieva]

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/1225
  
@vrozov addressed code review comments in the new commit. Please review.


---

[GitHub] drill pull request #1225: DRILL-6272: Refactor dynamic UDFs and function ini...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1225#discussion_r185452673
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/sql/TestCTTAS.java ---
@@ -35,41 +39,46 @@
 import org.apache.hadoop.fs.RemoteIterator;
 import org.apache.hadoop.fs.permission.FsPermission;
 import org.junit.BeforeClass;
+import org.junit.Rule;
 import org.junit.Test;
 import org.junit.experimental.categories.Category;
+import org.junit.rules.ExpectedException;
 
 import java.io.File;
 import java.io.IOException;
+import java.nio.file.Files;
 import java.nio.file.Paths;
 import java.util.List;
+import java.util.Optional;
 import java.util.Properties;
-import java.util.UUID;
 
 import static 
org.apache.drill.exec.util.StoragePluginTestUtils.DFS_PLUGIN_NAME;
 import static 
org.apache.drill.exec.util.StoragePluginTestUtils.DFS_TMP_SCHEMA;
 import static org.hamcrest.CoreMatchers.containsString;
 import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertThat;
 import static org.junit.Assert.assertTrue;
 
 @Category(SqlTest.class)
 public class TestCTTAS extends BaseTestQuery {
 
-  private static final UUID session_id = 
UUID.nameUUIDFromBytes("sessionId".getBytes());
   private static final String temp2_wk = "tmp2";
   private static final String temp2_schema = String.format("%s.%s", 
DFS_PLUGIN_NAME, temp2_wk);
 
+  private static String sessionId;
   private static FileSystem fs;
   private static FsPermission expectedFolderPermission;
   private static FsPermission expectedFilePermission;
 
+  @Rule
+  public ExpectedException thrown = ExpectedException.none();
+
   @BeforeClass
   public static void init() throws Exception {
-MockUp uuidMockUp = mockRandomUUID(session_id);
--- End diff --

I had to re-write this part since `MockUp.tearDown()` is not present in 
newer jmockit version. Now we don't mock session id at all.


---

[GitHub] drill pull request #1225: DRILL-6272: Refactor dynamic UDFs and function ini...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1225#discussion_r185456183
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/sql/TestCTTAS.java ---
@@ -164,121 +166,113 @@ public void 
testResolveTemporaryTableWithPartialSchema() throws Exception {
   @Test
   public void testPartitionByWithTemporaryTables() throws Exception {
 String temporaryTableName = "temporary_table_with_partitions";
-mockRandomUUID(UUID.nameUUIDFromBytes(temporaryTableName.getBytes()));
+cleanSessionDirectory();
 test("create TEMPORARY table %s partition by (c1) as select * from (" +
 "select 'A' as c1 from (values(1)) union all select 'B' as c1 from 
(values(1))) t", temporaryTableName);
-checkPermission(temporaryTableName);
+checkPermission();
   }
 
-  @Test(expected = UserRemoteException.class)
+  @Test
   public void testCreationOutsideOfDefaultTemporaryWorkspace() throws 
Exception {
-try {
-  String temporaryTableName = 
"temporary_table_outside_of_default_workspace";
-  test("create TEMPORARY table %s.%s as select 'A' as c1 from 
(values(1))", temp2_schema, temporaryTableName);
-} catch (UserRemoteException e) {
-  assertThat(e.getMessage(), containsString(String.format(
-  "VALIDATION ERROR: Temporary tables are not allowed to be 
created / dropped " +
-  "outside of default temporary workspace [%s].", 
DFS_TMP_SCHEMA)));
-  throw e;
-}
+String temporaryTableName = 
"temporary_table_outside_of_default_workspace";
+
+thrown.expect(UserRemoteException.class);
+thrown.expectMessage(containsString(String.format(
+"VALIDATION ERROR: Temporary tables are not allowed to be created 
/ dropped " +
+"outside of default temporary workspace [%s].", 
DFS_TMP_SCHEMA)));
+
+test("create TEMPORARY table %s.%s as select 'A' as c1 from 
(values(1))", temp2_schema, temporaryTableName);
   }
 
-  @Test(expected = UserRemoteException.class)
+  @Test
   public void testCreateWhenTemporaryTableExistsWithoutSchema() throws 
Exception {
 String temporaryTableName = "temporary_table_exists_without_schema";
-try {
-  test("create TEMPORARY table %s as select 'A' as c1 from 
(values(1))", temporaryTableName);
-  test("create TEMPORARY table %s as select 'A' as c1 from 
(values(1))", temporaryTableName);
-} catch (UserRemoteException e) {
-  assertThat(e.getMessage(), containsString(String.format(
- "VALIDATION ERROR: A table or view with given name [%s]" +
- " already exists in schema [%s]", temporaryTableName, 
DFS_TMP_SCHEMA)));
-  throw e;
-}
+
+thrown.expect(UserRemoteException.class);
+thrown.expectMessage(containsString(String.format(
+"VALIDATION ERROR: A table or view with given name [%s]" +
+" already exists in schema [%s]", temporaryTableName, 
DFS_TMP_SCHEMA)));
+
+test("create TEMPORARY table %s as select 'A' as c1 from (values(1))", 
temporaryTableName);
+test("create TEMPORARY table %s as select 'A' as c1 from (values(1))", 
temporaryTableName);
   }
 
-  @Test(expected = UserRemoteException.class)
+  @Test
   public void testCreateWhenTemporaryTableExistsCaseInsensitive() throws 
Exception {
 String temporaryTableName = "temporary_table_exists_without_schema";
-try {
-  test("create TEMPORARY table %s as select 'A' as c1 from 
(values(1))", temporaryTableName);
-  test("create TEMPORARY table %s as select 'A' as c1 from 
(values(1))", temporaryTableName.toUpperCase());
-} catch (UserRemoteException e) {
-  assertThat(e.getMessage(), containsString(String.format(
-  "VALIDATION ERROR: A table or view with given name [%s]" +
-  " already exists in schema [%s]", 
temporaryTableName.toUpperCase(), DFS_TMP_SCHEMA)));
-  throw e;
-}
+
+thrown.expect(UserRemoteException.class);
+thrown.expectMessage(containsString(String.format(
+"VALIDATION ERROR: A table or view with given name [%s]" +
--- End diff --

Done.


---

[GitHub] drill pull request #1225: DRILL-6272: Refactor dynamic UDFs and function ini...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1225#discussion_r185449640
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/coord/zk/TestZookeeperClient.java
 ---
@@ -125,7 +125,7 @@ public void testHasPathThrowsDrillRuntimeException() {
 
 Mockito
 .when(client.getCache().getCurrentData(absPath))
-.thenThrow(Exception.class);
+.thenThrow(RuntimeException.class);
--- End diff --

I just fixed this test to work with new mockito lib version. I suspect this 
test checks if exception thrown from 
`client.getCache().getCurrentData(absPath)` method will be wrapped in 
DrillRuntimeException.


---

[GitHub] drill pull request #1225: DRILL-6272: Refactor dynamic UDFs and function ini...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1225#discussion_r185450679
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/udf/dynamic/JarBuilder.java 
---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.udf.dynamic;
+
+import ch.qos.logback.classic.Level;
+import ch.qos.logback.classic.Logger;
+import ch.qos.logback.classic.LoggerContext;
+import ch.qos.logback.classic.spi.ILoggingEvent;
+import ch.qos.logback.core.ConsoleAppender;
+import org.apache.maven.cli.MavenCli;
+import org.apache.maven.cli.logging.Slf4jLogger;
+import org.codehaus.plexus.DefaultPlexusContainer;
+import org.codehaus.plexus.PlexusContainer;
+import org.codehaus.plexus.logging.BaseLoggerManager;
+import org.slf4j.LoggerFactory;
+
+import java.util.LinkedList;
+import java.util.List;
+
+public class JarBuilder {
+
+  private final MavenCli cli;
+
+  public JarBuilder() {
+this.cli = new MavenCli() {
+  @Override
+  protected void customizeContainer(PlexusContainer container) {
+((DefaultPlexusContainer) container).setLoggerManager(new 
BaseLoggerManager() {
+  @Override
+  protected org.codehaus.plexus.logging.Logger createLogger(String 
s) {
+return new Slf4jLogger(setupLogger(JarBuilder.class.getName(), 
Level.INFO));
+  }
+});
+  }
+};
+  }
+
+  /**
+   * Builds jars using embedded maven. Includes files / resources based 
given pattern,
+   * otherwise using defaults provided in pom.xml.
+   *
+   * @param jarName jar name
+   * @param projectDir project dir
+   * @param includeFiles pattern indicating which files should be included
+   * @param includeResources pattern indicating which resources should be 
included
+   *
+   * @return build exit code, 0 if build was successful
+   */
+  public int build(String jarName, String projectDir, String includeFiles, 
String includeResources) {
+System.setProperty("maven.multiModuleProjectDirectory", projectDir);
+List params = new LinkedList<>();
+params.add("clean");
+params.add("package");
+params.add("-DskipTests");
+params.add("-Djar.finalName=" + jarName);
+if (includeFiles != null) {
+  params.add("-Dinclude.files=" + includeFiles);
+}
+if (includeResources != null) {
+  params.add("-Dinclude.resources=" + includeResources);
+}
+return cli.doMain(params.toArray(new String[params.size()]), 
projectDir, System.out, System.err);
+  }
+
+  private static Logger setupLogger(String string, Level logLevel) {
--- End diff --

Yes, since I want to output info level logging when creating jars. 
Otherwise, no logs on how jars were created will appear.


---

[GitHub] drill pull request #1225: DRILL-6272: Refactor dynamic UDFs and function ini...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1225#discussion_r185450442
  
--- Diff: exec/java-exec/src/test/resources/drill-udf/pom.xml ---
@@ -0,0 +1,90 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+  4.0.0
+
+  org.apache.drill.udf
+  drill-udf
+  1.0
+
+  
+${project.name}
+1.13.0
--- End diff --

My previous jars used 1.8.0. for example, since I created them when that 
only version was available. API for Drill UDFs is public so we should not 
change it. There is no enforcement though. 


---

[GitHub] drill pull request #1225: DRILL-6272: Refactor dynamic UDFs and function ini...

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1225#discussion_r185452897
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/sql/TestCTTAS.java ---
@@ -498,47 +489,50 @@ public void 
testDropTemporaryTableAsViewWithoutException() throws Exception {
 .go();
   }
 
-  @Test(expected = UserRemoteException.class)
+  @Test
   public void testDropTemporaryTableAsViewWithException() throws Exception 
{
 String temporaryTableName = 
"temporary_table_to_drop_like_view_with_exception";
 test("create TEMPORARY table %s as select 'A' as c1 from (values(1))", 
temporaryTableName);
 
-try {
-  test("drop view %s.%s", DFS_TMP_SCHEMA, temporaryTableName);
-} catch (UserRemoteException e) {
-  assertThat(e.getMessage(), containsString(String.format(
-  "VALIDATION ERROR: Unknown view [%s] in schema [%s]", 
temporaryTableName, DFS_TMP_SCHEMA)));
-  throw e;
+thrown.expect(UserRemoteException.class);
+thrown.expectMessage(containsString(String.format(
+"VALIDATION ERROR: Unknown view [%s] in schema [%s]", 
temporaryTableName, DFS_TMP_SCHEMA)));
+
+test("drop view %s.%s", DFS_TMP_SCHEMA, temporaryTableName);
+  }
+
+  private static String getSessionId() throws Exception {
--- End diff --

Re-written, it turned out I have access to `UserSession` through 
`DrillbitContext`, so no mock is required at all.


---

[GitHub] drill pull request #1242: DRILL-6361: Revised typeOf() function versions

[vvysotskyi]

Github user vvysotskyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/1242#discussion_r185435469
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/UnionFunctions.java
 ---
@@ -146,16 +139,131 @@ private static int getTypeValue(MinorType type) {
 @Inject
 DrillBuf buf;
 
+@Override
 public void setup() {}
 
+@Override
 public void eval() {
 
-  byte[] type;
+  String typeName;
   if (input.isSet()) {
- type = input.getType().getMinorType().name().getBytes();
+typeName = input.getType().getMinorType().name();
   } else {
-type = 
org.apache.drill.common.types.TypeProtos.MinorType.NULL.name().getBytes();
+typeName = 
org.apache.drill.common.types.TypeProtos.MinorType.NULL.name();
   }
+  byte[] type = typeName.getBytes();
+  buf = buf.reallocIfNeeded(type.length);
+  buf.setBytes(0, type);
+  out.buffer = buf;
+  out.start = 0;
+  out.end = type.length;
+}
+  }
+
+  @FunctionTemplate(name = "sqlTypeOf",
+  scope = FunctionTemplate.FunctionScope.SIMPLE,
+  nulls = NullHandling.INTERNAL)
+  public static class GetSqlType implements DrillSimpleFunc {
+
+@Param
+FieldReader input;
+@Output
+VarCharHolder out;
+@Inject
+DrillBuf buf;
+
+@Override
+public void setup() {}
+
+@Override
+public void eval() {
+
+  org.apache.drill.common.types.TypeProtos.MajorType type = 
input.getType();
+
+  // Note: extendType is a static function because the byte code magic
+  // for UDFS can't handle switch statements.
+
+  String typeName =
+  org.apache.drill.exec.expr.fn.impl.UnionFunctions.extendType(
+  type,
+  
org.apache.drill.common.types.Types.getBaseSqlTypeName(type));
+
+  
org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.varCharOutput(
--- End diff --

I think passing a holder into the method may prevent scalar replacement for 
this holder.


---

[GitHub] drill issue #1224: DRILL-6321: Customize Drill's conformance. Allow support ...

[vvysotskyi]

Github user vvysotskyi commented on the issue:

https://github.com/apache/drill/pull/1224
  
@parthchandra, thanks for the explanation. 
@chunhui-shi, thanks for making changes, +1


---

[GitHub] drill pull request #1241: DRILL-6364: Handle Cluster Info in WebUI when exis...

[sohami]

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185403964
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -252,33 +255,129 @@
   timeout = setTimeout(reloadStatus, refreshTime);
   }
 
-  function fillStatus(data,size) {
-  var status_map = (data.responseJSON);
-  for (i = 1; i <= size; i++) {
-var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue;
-address = address.trim();
-var port = $("#row-"+i).find("#port").html();
-var key = address+"-"+port;
+  function fillStatus(dataResponse,size) {
+  var status_map = (dataResponse.responseJSON);
+  //In case localhost has gone down (i.e. we don't know status 
from ZK)
+  if (typeof status_map == 'undefined') {
+//Query other nodes for state details
+for (j = 1; j <= size; j++) {
+  if ($("#row-"+j).find("#current").html() == "Current") {
+continue; //Skip LocalHost
+  }
+  var address = 
$("#row-"+j).find("#address").contents().get(0).nodeValue.trim();
+  var restPort = 
$("#row-"+j).find("#httpPort").contents().get(0).nodeValue.trim();
+  var altStateUrl = location.protocol + "//" + 
address+":"+restPort + "/state";
+  var goatResponse = $.getJSON(altStateUrl)
+.done(function(stateDataJson) {
+//Update Status & Buttons for alternate stateData
+if (typeof status_map == 'undefined') {
+  status_map = (stateDataJson); //Update
+  updateStatusAndShutdown(stateDataJson);
+}
+  });
+  //Don't loop any more
+  if (typeof status_map != 'undefined') {
+break;
+  }
+}
+  } else {
--- End diff --

Yes as discussed in which case there is race condition. Please see latest 
comment above.


---

[GitHub] drill pull request #1241: DRILL-6364: Handle Cluster Info in WebUI when exis...

[sohami]

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185406434
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -252,33 +255,129 @@
   timeout = setTimeout(reloadStatus, refreshTime);
   }
 
-  function fillStatus(data,size) {
-  var status_map = (data.responseJSON);
-  for (i = 1; i <= size; i++) {
-var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue;
-address = address.trim();
-var port = $("#row-"+i).find("#port").html();
-var key = address+"-"+port;
+  function fillStatus(dataResponse,size) {
+  var status_map = (dataResponse.responseJSON);
+  //In case localhost has gone down (i.e. we don't know status 
from ZK)
+  if (typeof status_map == 'undefined') {
+//Query other nodes for state details
+for (j = 1; j <= size; j++) {
+  if ($("#row-"+j).find("#current").html() == "Current") {
+continue; //Skip LocalHost
+  }
+  var address = 
$("#row-"+j).find("#address").contents().get(0).nodeValue.trim();
+  var restPort = 
$("#row-"+j).find("#httpPort").contents().get(0).nodeValue.trim();
+  var altStateUrl = location.protocol + "//" + 
address+":"+restPort + "/state";
+  var goatResponse = $.getJSON(altStateUrl)
+.done(function(stateDataJson) {
+//Update Status & Buttons for alternate stateData
+if (typeof status_map == 'undefined') {
+  status_map = (stateDataJson); //Update
+  updateStatusAndShutdown(stateDataJson);
+}
+  });
+  //Don't loop any more
+  if (typeof status_map != 'undefined') {
+break;
+  }
+}
+  } else {
+updateStatusAndShutdown(status_map);
+  }
+  }
+
+  function updateStatusAndShutdown(status_map) {
+let bitMap = {};
+if (typeof status_map != 'undefined') {
+for (var k in status_map) {
+  bitMap[k] = status_map[k];
+}
+}
+for (i = 1; i <= size; i++) {
+let key = "";
+if ($("#row-"+i).find("#stateKey").length > 0) { //Check if 
newBit that has no stateKey
+  key = $("#row-"+i).find("#stateKey").textContent;
+} else {
+  let address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
+  let port = $("#row-"+i).find("#httpPort").html();
+  key = address+"-"+port;
+}
 
-if (status_map[key] == null) {
+if (typeof status_map == 'undefined') {
+$("#row-"+i).find("#status").text(nAText);
+
$("#row-"+i).find("#shutdown").prop('disabled',true).css('opacity',0.5);
+$("#row-"+i).find("#queriesCount").text("");
+} else if (status_map[key] == null) {
 $("#row-"+i).find("#status").text("OFFLINE");
 
$("#row-"+i).find("#shutdown").prop('disabled',true).css('opacity',0.5);
 $("#row-"+i).find("#queriesCount").text("");
-}
-else {
+} else {
 if (status_map[key] == "ONLINE") {
 $("#row-"+i).find("#status").text(status_map[key]);
-
$("#row-"+i).find("#shutdown").prop('disabled',false).css('opacity',1.0);
-}
-else {
+<#if ( model.shouldShowAdminInfo() || 
!model.isAuthEnabled() ) >
+if ( location.protocol != "https" || 
($("#row-"+i).find("#current").html() == "Current") ) {
+  
$("#row-"+i).find("#shutdown").prop('disabled',false).css('opacity',1.0).css('cursor','pointer');
+}
+
+} else {
 if ($("#row-"+i).find("#current").html() == "Current") 
{
 fillQueryCount(i);
 }
 $("#row-"+i).find("#status").text(status_map[key]);
 }
+//Removing accounted key
+delete bitMap[key];
 }
-  }
+}
+//If bitMap is not empty, then new bits have been discovered!
+listNewDrillbits(bitMap, status_map);
+  }
+
+  //Add new Bits for listing
+  

[GitHub] drill pull request #1241: DRILL-6364: Handle Cluster Info in WebUI when exis...

[sohami]

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185406443
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -252,33 +255,129 @@
   timeout = setTimeout(reloadStatus, refreshTime);
   }
 
-  function fillStatus(data,size) {
-  var status_map = (data.responseJSON);
-  for (i = 1; i <= size; i++) {
-var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue;
-address = address.trim();
-var port = $("#row-"+i).find("#port").html();
-var key = address+"-"+port;
+  function fillStatus(dataResponse,size) {
+  var status_map = (dataResponse.responseJSON);
+  //In case localhost has gone down (i.e. we don't know status 
from ZK)
+  if (typeof status_map == 'undefined') {
+//Query other nodes for state details
+for (j = 1; j <= size; j++) {
+  if ($("#row-"+j).find("#current").html() == "Current") {
+continue; //Skip LocalHost
+  }
+  var address = 
$("#row-"+j).find("#address").contents().get(0).nodeValue.trim();
+  var restPort = 
$("#row-"+j).find("#httpPort").contents().get(0).nodeValue.trim();
+  var altStateUrl = location.protocol + "//" + 
address+":"+restPort + "/state";
+  var goatResponse = $.getJSON(altStateUrl)
+.done(function(stateDataJson) {
+//Update Status & Buttons for alternate stateData
+if (typeof status_map == 'undefined') {
+  status_map = (stateDataJson); //Update
+  updateStatusAndShutdown(stateDataJson);
+}
+  });
+  //Don't loop any more
+  if (typeof status_map != 'undefined') {
+break;
+  }
+}
+  } else {
+updateStatusAndShutdown(status_map);
+  }
+  }
+
+  function updateStatusAndShutdown(status_map) {
+let bitMap = {};
+if (typeof status_map != 'undefined') {
+for (var k in status_map) {
+  bitMap[k] = status_map[k];
+}
+}
+for (i = 1; i <= size; i++) {
+let key = "";
+if ($("#row-"+i).find("#stateKey").length > 0) { //Check if 
newBit that has no stateKey
+  key = $("#row-"+i).find("#stateKey").textContent;
+} else {
+  let address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
+  let port = $("#row-"+i).find("#httpPort").html();
+  key = address+"-"+port;
+}
 
-if (status_map[key] == null) {
+if (typeof status_map == 'undefined') {
+$("#row-"+i).find("#status").text(nAText);
+
$("#row-"+i).find("#shutdown").prop('disabled',true).css('opacity',0.5);
+$("#row-"+i).find("#queriesCount").text("");
+} else if (status_map[key] == null) {
 $("#row-"+i).find("#status").text("OFFLINE");
 
$("#row-"+i).find("#shutdown").prop('disabled',true).css('opacity',0.5);
 $("#row-"+i).find("#queriesCount").text("");
-}
-else {
+} else {
 if (status_map[key] == "ONLINE") {
 $("#row-"+i).find("#status").text(status_map[key]);
-
$("#row-"+i).find("#shutdown").prop('disabled',false).css('opacity',1.0);
-}
-else {
+<#if ( model.shouldShowAdminInfo() || 
!model.isAuthEnabled() ) >
+if ( location.protocol != "https" || 
($("#row-"+i).find("#current").html() == "Current") ) {
+  
$("#row-"+i).find("#shutdown").prop('disabled',false).css('opacity',1.0).css('cursor','pointer');
+}
+
+} else {
 if ($("#row-"+i).find("#current").html() == "Current") 
{
 fillQueryCount(i);
 }
 $("#row-"+i).find("#status").text(status_map[key]);
 }
+//Removing accounted key
+delete bitMap[key];
 }
-  }
+}
+//If bitMap is not empty, then new bits have been discovered!
+listNewDrillbits(bitMap, status_map);
+  }
+
+  //Add new Bits for listing
+  

[GitHub] drill pull request #1241: DRILL-6364: Handle Cluster Info in WebUI when exis...

[sohami]

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185403838
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -252,37 +257,139 @@
   timeout = setTimeout(reloadStatus, refreshTime);
   }
 
-  function fillStatus(data,size) {
-  var status_map = (data.responseJSON);
-  for (i = 1; i <= size; i++) {
-var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue;
-address = address.trim();
-var port = $("#row-"+i).find("#port").html();
-var key = address+"-"+port;
+  //Fill the Status table for all the listed drillbits
+  function fillStatus(dataResponse,size) {
+  let status_map = (dataResponse.responseJSON);
+  //In case localhost has gone down (i.e. we don't know status 
from ZK)
+  if (typeof status_map == 'undefined') {
+let rxUpdateCount = 0;
+let statusRespList = [];
+//Query other nodes for state details
+for (j = 1; j <= size; j++) {
+  let currentRow = $("#row-"+j);
+  if (currentRow.find("#current").html() == "Current") {
+continue; //Skip LocalHost
+  }
+  let address = 
currentRow.find("#address").contents().get(0).nodeValue.trim();
+  let restPort = 
currentRow.find("#httpPort").contents().get(0).nodeValue.trim();
+  let altStateUrl = location.protocol + "//" + 
address+":"+restPort + "/state";
+  let altResponse = $.getJSON(altStateUrl)
+.done(function(stateDataJson) {
+//Update Status & Buttons for alternate stateData
+if (rxUpdateCount == 0) {
--- End diff --

I am not an expert on javascript but little surprised to see how it's 
allowing reference of `rxUpdateCount` and `statusRespList` inside the callback 
function `done`, since these variables are defined local to `fillStatus` 
function.

However there is still a race condition here, you cannot avoid it without a 
lock or a sync call. Please see [.when in 
jQuery](http://api.jquery.com/jQuery.when/) and this 
[stackoverflow](https://stackoverflow.com/questions/3709597/wait-until-all-jquery-ajax-requests-are-done)
 post for example. Also when the size of Drillbit in cluster is in hundreds 
then you will end up making that many requests which is an overkill, why not 
try say 3 Drillbits at max ? If you still get undefined then treat them as 
status is not available.


---

[GitHub] drill pull request #1241: DRILL-6364: Handle Cluster Info in WebUI when exis...

[sohami]

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185406413
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -252,33 +255,129 @@
   timeout = setTimeout(reloadStatus, refreshTime);
   }
 
-  function fillStatus(data,size) {
-  var status_map = (data.responseJSON);
-  for (i = 1; i <= size; i++) {
-var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue;
-address = address.trim();
-var port = $("#row-"+i).find("#port").html();
-var key = address+"-"+port;
+  function fillStatus(dataResponse,size) {
+  var status_map = (dataResponse.responseJSON);
+  //In case localhost has gone down (i.e. we don't know status 
from ZK)
+  if (typeof status_map == 'undefined') {
+//Query other nodes for state details
+for (j = 1; j <= size; j++) {
+  if ($("#row-"+j).find("#current").html() == "Current") {
+continue; //Skip LocalHost
+  }
+  var address = 
$("#row-"+j).find("#address").contents().get(0).nodeValue.trim();
+  var restPort = 
$("#row-"+j).find("#httpPort").contents().get(0).nodeValue.trim();
+  var altStateUrl = location.protocol + "//" + 
address+":"+restPort + "/state";
+  var goatResponse = $.getJSON(altStateUrl)
+.done(function(stateDataJson) {
+//Update Status & Buttons for alternate stateData
+if (typeof status_map == 'undefined') {
+  status_map = (stateDataJson); //Update
+  updateStatusAndShutdown(stateDataJson);
+}
+  });
+  //Don't loop any more
+  if (typeof status_map != 'undefined') {
+break;
+  }
+}
+  } else {
+updateStatusAndShutdown(status_map);
+  }
+  }
+
+  function updateStatusAndShutdown(status_map) {
+let bitMap = {};
--- End diff --

Ok but status_map is used in listNewDrillbits and point was to use only 
one, in this case bit_map is fine.


---

[GitHub] drill pull request #1241: DRILL-6364: Handle Cluster Info in WebUI when exis...

[sohami]

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185392204
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -103,14 +105,14 @@
 
 ${drillbit.getState()}
 Not Available
-  
-<#if ( model.shouldShowAdminInfo() &&  ( 
drillbit.isCurrent() || ( !model.isAuthEnabled() && location.protocol != 
"https" ))) >
-  
-<#else>
+
+  <#if ( model.shouldShowAdminInfo() || 
!model.isAuthEnabled() || drillbit.isCurrent() ) >
--- End diff --

I think what is required here is to show shutdown button both for local and 
remote Drillbits only to admin users. But the button will be hidden and not 
enabled to click. Then later logic in [line 
317](https://github.com/kkhatua/drill/blob/ab3e8619c6259803eb362be290a3a3605839a194/exec/java-exec/src/main/resources/rest/index.ftl#L317)
 will enable the button.

But with current check even if user is non-Admin it will show the button. 
Check will be as below:

```
<#if (!model.isAuthEnabled() || model.shouldShowAdminInfo())>
<#if drillbit.isCurrent()>

<#else>



```

Please test with auth enabled and non-admin user too. Sharing example 
snapshot below.

https://user-images.githubusercontent.com/22159459/39505653-a37c1326-4d88-11e8-99c7-d8ab5495c7cb.png;>



---

[GitHub] drill pull request #1241: DRILL-6364: Handle Cluster Info in WebUI when exis...

[sohami]

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185406458
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -328,9 +432,11 @@
   //Iterates through all the nodes for update
   function reloadMetrics() {
   for (i = 1; i <= size; i++) {
-  var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
-  var httpPort = 
$("#row-"+i).find("#httpPort").contents().get(0).nodeValue.trim();
-  updateMetricsHtml(address, httpPort, i);
+  if ( $("#row-"+i).find("#stateKey").length == 0 ) {
+var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
+var httpPort = 
$("#row-"+i).find("#httpPort").contents().get(0).nodeValue.trim();
+updateMetricsHtml(address, httpPort, i);
--- End diff --

Not true please see the braces. They evaluate to different outcome.


---

[GitHub] drill issue #1242: DRILL-6361: Revised typeOf() function versions

[paul-rogers]

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/1242
  
@kfaraaz, the logic for all types is the same. But, still added 
`BIT`/`BOOLEAN`, `DATE`, `TIME`, `TIMESTAMP`, `INTERVAL`, `VARBINARY`, and 
`DECIMAL`. 

Found several bugs, but none in the type functions. Tried to extend 
`sqlTypeOf()` to include `DECIMAL` precision and scale, but found that that 
information is missing (another bug.)


---

[jira] [Created] (DRILL-6378) MajorType passed into UDF with FieldReader has unset prec, scale for DECIMAL

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-6378:
--

 Summary: MajorType passed into UDF with FieldReader has unset 
prec, scale for DECIMAL
 Key: DRILL-6378
 URL: https://issues.apache.org/jira/browse/DRILL-6378
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.13.0
Reporter: Paul Rogers


UDFs allow passing in a generic {{FieldReader}} such as for the {{typeof}} 
function. I tried to modify the new {{sqlTypeOf()}} function to include the 
DECIMAL precision and scale, something like: {{DECIMAL(9, 2)}}.

But, the {{MajorType}} associated with the {{FieldReader}} has these values as 
0. Not sure if it is because the DECIMAL is the result of a cast, or if there 
is a deeper bug.

Query:

{noformat}
ALTER SESSION SET `planner.enable_decimal_data_type` = true;

SELECT sqlTypeof(CAST(a AS DECIMAL)) FROM (VALUES (1)) AS T(a);
{noformat}

Debug information for the {{MajorType}}

{noformat}
typeTypeProtos$MajorType  (id=151)  
precision_  0   
scale_  0   
{noformat}

This bug is not simply a nuisance for formatting. The precision and scale are 
required to make sense of any {{DECIMAL}} values that the function wants to 
process: without these values, the function does not know the meaning of the 
{{DECIMAL}} data.

Once this bug is fixed, reenable the type suffix code in 
{{UnionFunctions.extendType()}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)