[jira] [Commented] (DRILL-5717) date time test cases is not Local independent

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123568#comment-16123568
 ] 

ASF GitHub Bot commented on DRILL-5717:
---

Github user vvysotskyi commented on the issue:

https://github.com/apache/drill/pull/904
  
Some tests from `TestDateFunctions`, `TestDateConversions` and 
`TestConstantFolding` classes also fails with some other locales (I checked it 
for `zh_TW` locale). 
I think they should also be fixed in the border of this PR.
Are there any other tests that depend on the locale?


> date time test cases is not Local independent
> -
>
> Key: DRILL-5717
> URL: https://issues.apache.org/jira/browse/DRILL-5717
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>
> Some date time test cases like  JodaDateValidatorTest  is not Local 
> independent .This will cause other Local's users's test phase to fail. We 
> should let these test cases to be Local env independent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5712) Update the pom files with dependency exclusions for commons-codec

[Sindhuri Ramanarayan Rayavaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sindhuri Ramanarayan Rayavaram updated DRILL-5712:
--
Component/s: Tools, Build & Test

> Update the pom files with dependency exclusions for commons-codec
> -
>
> Key: DRILL-5712
> URL: https://issues.apache.org/jira/browse/DRILL-5712
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Reporter: Sindhuri Ramanarayan Rayavaram
>Assignee: Sindhuri Ramanarayan Rayavaram
>
> In java-exec, we are adding a dependency for commons-codec of version 1.10. 
> Other dependencies like hadoop-common, parquet-column etc are trying to 
> download different versions for common codec. Exclusions should be added for 
> common-codec in these dependencies.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4264) Dots in identifier are not escaped correctly

[John Omernik (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123836#comment-16123836
 ] 

John Omernik commented on DRILL-4264:
-

Lots to process here...

Let me add a simple thing though... 

"As a user I have a new data set, I have no idea what's in it, from Drill 
(that's the key) I should be able to select * from directory, and if it's a 
known format (JSON, Parquet, CSV etc) I should get results back.  I know I can 
do select `field.one`, `field.two` from directory and get it, but say it's a 
parquet file created in Spark... there is no way for me explore that data in 
Drill ... I need select *

> Dots in identifier are not escaped correctly
> 
>
> Key: DRILL-4264
> URL: https://issues.apache.org/jira/browse/DRILL-4264
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Codegen
>Reporter: Alex
>Assignee: Volodymyr Vysotskyi
>  Labels: doc-impacting
> Fix For: 1.12.0
>
>
> If you have some json data like this...
> {code:javascript}
> {
>   "0.0.1":{
> "version":"0.0.1",
> "date_created":"2014-03-15"
>   },
>   "0.1.2":{
> "version":"0.1.2",
> "date_created":"2014-05-21"
>   }
> }
> {code}
> ... there is no way to select any of the rows since their identifiers contain 
> dots and when trying to select them, Drill throws the following error:
> Error: SYSTEM ERROR: UnsupportedOperationException: Unhandled field reference 
> "0.0.1"; a field reference identifier must not have the form of a qualified 
> name
> This must be fixed since there are many json data files containing dots in 
> some of the keys (e.g. when specifying version numbers etc)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5431) Support SSL

[Parth Chandra (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123633#comment-16123633
 ] 

Parth Chandra commented on DRILL-5431:
--

[~cshi] Please take a look at the design doc.

> Support SSL
> ---
>
> Key: DRILL-5431
> URL: https://issues.apache.org/jira/browse/DRILL-5431
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Client - Java, Client - ODBC
>Reporter: Sudheesh Katkam
>Assignee: Sudheesh Katkam
>
> Support SSL between Drillbit and JDBC/ODBC drivers. Drill already supports 
> HTTPS for web traffic.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5717) date time test cases is not Local independent

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123020#comment-16123020
 ] 

ASF GitHub Bot commented on DRILL-5717:
---

GitHub user weijietong opened a pull request:

https://github.com/apache/drill/pull/904

DRILL-5717: let some date time test cases be Local independent

Some date time test cases is US Local .This will cause developers from 
other areas to fail. Fix is to specify the explicit Local to the test cases. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/weijietong/drill DRILL-5717

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/904.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #904


commit b44f780a948c4a0898e7cee042c0590f0713f780
Author: weijietong 
Date:   2017-06-08T08:03:46Z

Merge pull request #1 from apache/master

sync

commit d045c757c80a759b435479cc89f33c749fc16ac2
Author: weijie.tong 
Date:   2017-08-11T08:01:36Z

Merge branch 'master' of github.com:weijietong/drill

commit 906eef175f6751663e58f36519b325f5a4e6dfea
Author: weijie.tong 
Date:   2017-08-11T08:14:54Z

let some test cases be Local independent




> date time test cases is not Local independent
> -
>
> Key: DRILL-5717
> URL: https://issues.apache.org/jira/browse/DRILL-5717
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>
> Some date time test cases like  JodaDateValidatorTest  is not Local 
> independent .This will cause other Local's users's test phase to fail. We 
> should let these test cases to be Local env independent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-3867) Store relative paths in metadata file

[Vitalii Diravka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vitalii Diravka updated DRILL-3867:
---
Description: 
git.commit.id.abbrev=cf4f745
git.commit.time=29.09.2015 @ 23\:19\:52 UTC

The below sequence of steps reproduces the issue

1. Create the cache file
{code}
0: jdbc:drill:zk=10.10.103.60:5181> refresh table metadata 
dfs.`/drill/testdata/metadata_caching/lineitem`;
+---+-+
|  ok   |   summary 
  |
+---+-+
| true  | Successfully updated metadata for table 
/drill/testdata/metadata_caching/lineitem.  |
+---+-+
1 row selected (1.558 seconds)
{code}

2. Move the directory
{code}
hadoop fs -mv /drill/testdata/metadata_caching/lineitem /drill/
{code}

3. Now run a query on top of it
{code}
0: jdbc:drill:zk=10.10.103.60:5181> select * from dfs.`/drill/lineitem` limit 1;
Error: SYSTEM ERROR: FileNotFoundException: Requested file 
maprfs:///drill/testdata/metadata_caching/lineitem/2006/1 does not exist.


[Error Id: b456d912-57a0-4690-a44b-140d4964903e on pssc-66.qa.lab:31010] 
(state=,code=0)
{code}
This is obvious given the fact that we are storing absolute file paths in the 
cache file.

*Summary description of the fix:*

In Drill 1.11 and later, Drill stores the paths to the Parquet files as 
relative paths instead of absolute paths. You can move partitioned Parquet 
directories from one location in the distributed files system to another 
without issuing the REFRESH TABLE METADATA command to rebuild the Parquet 
metadata files; the metadata remains valid in the new location.

Note

Reverting back to a previous version of Drill from 1.11 is not recommended 
because Drill will incorrectly interpret the Parquet metadata files created by 
Drill 1.11. Should this occur, remove the Parquet metadata files and run the 
refresh table metadata command to rebuild the files in the older format.


  was:
git.commit.id.abbrev=cf4f745
git.commit.time=29.09.2015 @ 23\:19\:52 UTC

The below sequence of steps reproduces the issue

1. Create the cache file
{code}
0: jdbc:drill:zk=10.10.103.60:5181> refresh table metadata 
dfs.`/drill/testdata/metadata_caching/lineitem`;
+---+-+
|  ok   |   summary 
  |
+---+-+
| true  | Successfully updated metadata for table 
/drill/testdata/metadata_caching/lineitem.  |
+---+-+
1 row selected (1.558 seconds)
{code}

2. Move the directory
{code}
hadoop fs -mv /drill/testdata/metadata_caching/lineitem /drill/
{code}

3. Now run a query on top of it
{code}
0: jdbc:drill:zk=10.10.103.60:5181> select * from dfs.`/drill/lineitem` limit 1;
Error: SYSTEM ERROR: FileNotFoundException: Requested file 
maprfs:///drill/testdata/metadata_caching/lineitem/2006/1 does not exist.


[Error Id: b456d912-57a0-4690-a44b-140d4964903e on pssc-66.qa.lab:31010] 
(state=,code=0)
{code}

This is obvious given the fact that we are storing absolute file paths in the 
cache file


> Store relative paths in metadata file
> -
>
> Key: DRILL-3867
> URL: https://issues.apache.org/jira/browse/DRILL-3867
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Metadata
>Affects Versions: 1.2.0
>Reporter: Rahul Challapalli
>Assignee: Vitalii Diravka
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.11.0
>
>
> git.commit.id.abbrev=cf4f745
> git.commit.time=29.09.2015 @ 23\:19\:52 UTC
> The below sequence of steps reproduces the issue
> 1. Create the cache file
> {code}
> 0: jdbc:drill:zk=10.10.103.60:5181> refresh table metadata 
> dfs.`/drill/testdata/metadata_caching/lineitem`;
> +---+-+
> |  ok   |   summary   
> |
> +---+-+
> | true  | Successfully updated metadata for table 
> /drill/testdata/metadata_caching/lineitem.  |
> +---+-+
> 1 row selected (1.558 seconds)
> {code}
> 2. Move the directory
> {code}
> hadoop fs -mv /drill/testdata/metadata_caching/lineitem /drill/
> 

[jira] [Created] (DRILL-5717) date time test cases is not Local independent

[weijie.tong (JIRA)]

weijie.tong created DRILL-5717:
--

 Summary: date time test cases is not Local independent
 Key: DRILL-5717
 URL: https://issues.apache.org/jira/browse/DRILL-5717
 Project: Apache Drill
  Issue Type: Bug
  Components: Tools, Build & Test
Affects Versions: 1.11.0, 1.9.0
Reporter: weijie.tong


Some date time test cases like  JodaDateValidatorTest  is not Local independent 
.This will cause other Local's users's test phase to fail. We should let these 
test cases to be Local env independent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5717) date time test cases is not Local independent

[weijie.tong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

weijie.tong updated DRILL-5717:
---
Reviewer: Arina Ielchiieva

> date time test cases is not Local independent
> -
>
> Key: DRILL-5717
> URL: https://issues.apache.org/jira/browse/DRILL-5717
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>
> Some date time test cases like  JodaDateValidatorTest  is not Local 
> independent .This will cause other Local's users's test phase to fail. We 
> should let these test cases to be Local env independent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-1162) 25 way join ended up with OOM

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123300#comment-16123300
 ] 

ASF GitHub Bot commented on DRILL-1162:
---

GitHub user vvysotskyi opened a pull request:

https://github.com/apache/drill/pull/905

DRILL-1162: Fix OOM for hash join operator when the right input has a much 
larger actual number of rows than the left one

Please see the problem description in 
[DRILL-1162](https://issues.apache.org/jira/browse/DRILL-1162).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vvysotskyi/drill DRILL-1162

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/905.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #905


commit de9a32997774acf34d2cb42f534decac5fd75cb5
Author: Volodymyr Vysotskyi 
Date:   2017-08-09T21:21:48Z

DRILL-1162: Fix OOM for hash join operator when the right input has a much 
larger actual number of rows than the left one




> 25 way join ended up with OOM
> -
>
> Key: DRILL-1162
> URL: https://issues.apache.org/jira/browse/DRILL-1162
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Volodymyr Vysotskyi
>Priority: Critical
> Fix For: Future
>
> Attachments: error.log, oom_error.log
>
>
> git.commit.id.abbrev=e5c2da0
> The below query results in 0 results being returned 
> {code:sql}
> select count(*) from `lineitem1.parquet` a 
> inner join `part.parquet` j on a.l_partkey = j.p_partkey 
> inner join `orders.parquet` k on a.l_orderkey = k.o_orderkey 
> inner join `supplier.parquet` l on a.l_suppkey = l.s_suppkey 
> inner join `partsupp.parquet` m on j.p_partkey = m.ps_partkey and l.s_suppkey 
> = m.ps_suppkey 
> inner join `customer.parquet` n on k.o_custkey = n.c_custkey 
> inner join `lineitem2.parquet` b on a.l_orderkey = b.l_orderkey 
> inner join `lineitem2.parquet` c on a.l_partkey = c.l_partkey 
> inner join `lineitem2.parquet` d on a.l_suppkey = d.l_suppkey 
> inner join `lineitem2.parquet` e on a.l_extendedprice = e.l_extendedprice 
> inner join `lineitem2.parquet` f on a.l_comment = f.l_comment 
> inner join `lineitem2.parquet` g on a.l_shipdate = g.l_shipdate 
> inner join `lineitem2.parquet` h on a.l_commitdate = h.l_commitdate 
> inner join `lineitem2.parquet` i on a.l_receiptdate = i.l_receiptdate 
> inner join `lineitem2.parquet` o on a.l_receiptdate = o.l_receiptdate 
> inner join `lineitem2.parquet` p on a.l_receiptdate = p.l_receiptdate 
> inner join `lineitem2.parquet` q on a.l_receiptdate = q.l_receiptdate 
> inner join `lineitem2.parquet` r on a.l_receiptdate = r.l_receiptdate 
> inner join `lineitem2.parquet` s on a.l_receiptdate = s.l_receiptdate 
> inner join `lineitem2.parquet` t on a.l_receiptdate = t.l_receiptdate 
> inner join `lineitem2.parquet` u on a.l_receiptdate = u.l_receiptdate 
> inner join `lineitem2.parquet` v on a.l_receiptdate = v.l_receiptdate 
> inner join `lineitem2.parquet` w on a.l_receiptdate = w.l_receiptdate 
> inner join `lineitem2.parquet` x on a.l_receiptdate = x.l_receiptdate;
> {code}
> However when we remove the last 'inner join' and run the query it returns 
> '716372534'. Since the last inner join is similar to the one's before it, it 
> should match some records and return the data appropriately.
> The logs indicated that it actually returned 0 results. Attached the log file.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5546) Schema change problems caused by empty batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124096#comment-16124096
 ] 

ASF GitHub Bot commented on DRILL-5546:
---

GitHub user jinfengni opened a pull request:

https://github.com/apache/drill/pull/906

DRILL-5546: Handle schema change exception failure caused by empty in…

…put or empty batche.

1. Modify ScanBatch's logic when it iterates list of RecordReader.
   1) Skip RecordReader if it returns 0 row && present same schema. A new 
schema (by calling Mutator.isNewSchema() ) means either a new top level field 
is added, or a field in a nested field is added, or an existing field type is 
changed.
   2) Implicit columns are added and populated only when the input is not 
empty, i.e. the batch contains > 0 row or rowCount == 0 && new schema.
   3) ScanBatch will return NONE directly (called as "fast NONE"), if all 
its RecordReaders haver empty input and thus are skipped, in stead of returing 
OK_NEW_SCHEMA first.

2. Modify IteratorValidatorBatchIterator to allow
   1) fast NONE ( before seeing a OK_NEW_SCHEMA)
   2) batch with empty list of columns.

2. Modify JsonRecordReader when it get 0 row. Do not insert a nullable-int 
column for 0 row input. Together with ScanBatch, Drill will skip empty json 
files.

3. Modify binary operators such as join, union to handle fast none for 
either one side or both sides. Abstract the logic in AbstractBinaryRecordBatch, 
except for MergeJoin as its implementation is quite different from others.

4. Fix and refactor union all operator.
  1) Correct union operator hanndling 0 input rows. Previously, it will 
ignore inputs with 0 row and put nullable-int into output schema, which causes 
various of schema change issue in down-stream operator. The new behavior is to 
take schema with 0 into account
  in determining the output schema, in the same way with > 0 input rows. By 
doing that, we ensure Union operator will not behave like a schema-lossy 
operator.
  2) Add a UnionInputIterator to simplify the logic to iterate the 
left/right inputs, removing significant chunk of duplicate codes in previous 
implementation.
  The new union all operator reduces the code size into half, comparing the 
old one.

5. Introduce UntypedNullVector to handle convertFromJson() function, when 
the input batch contains 0 row.
  Problem: The function convertFromJSon() is different from other regular 
functions in that it only knows the output schema after evaluation is 
performed. When input has 0 row, Drill essentially does not have
  a way to know the output type, and previously will assume Map type. That 
works under the assumption other operators like Union would ignore batch with 0 
row, which is no longer
  the case in the current implementation.
  Solution: Use MinorType.NULL at the output type for convertFromJSON() 
when input contains 0 row. The new UntypedNullVector is used to represent a 
column with MinorType.NULL.

6. HBaseGroupScan convert star column into list of row_key and column 
family. HBaseRecordReader should reject column star since it expectes star has 
been converted somewhere else.
  In HBase a column family always has map type, and a non-rowkey column 
always has nullable varbinary type, this ensures that HBaseRecordReader across 
different HBase regions will have the same top level schema, even if the region 
is
  empty or prune all the rows due to filter pushdown optimization. In other 
words, we will not see different top level schema from different 
HBaseRecordReader for the same table.
  However, such change will not be able to handle hard schema change : c1 
exists in cf1 in one region, but not in another region. Further work is 
required to handle hard schema change.

7. Modify scan cost estimation when the query involves * column. This is to 
remove the planning randomness since previously two different operators could 
have same cost.

8. Add a new flag 'outputProj' to Project operator, to indicate if Project 
is for the query's final output. Such Project is added by TopProjectVisitor, to 
handle fast NONE when all the inputs to the query are empty
and are skipped.
  1) column star is replaced with empty list
  2) regular column reference is replaced with nullable-int column
  3) An expression will go through ExpressionTreeMaterializer, and use the 
type of materialized expression as the output type
  4) Return an OK_NEW_SCHEMA with the schema using the above logic, then 
return a NONE to down-stream operator.

9. Add unit test to test operators handling empty input.

10. Add unit test to test query when inputs are all empty.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jinfengni/incubator-drill 

[jira] [Commented] (DRILL-5717) date time test cases is not Local independent

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123865#comment-16123865
 ] 

Paul Rogers commented on DRILL-5717:


Drill dates, by design, a defined to be in the server's time zone, even when 
sent to the client, and even when being converted from data without a time zone.

Given an input of 2017-08-11T23:00:00

Drill assumes that this is a date in the local time zone, say PST. But, it 
represents the time as if PST were UTC. That is, parse the date using Java Date 
and get a time relative to the epoch UTC but interpret it as relative to the 
epoch PST.

Then, the date is sent to the client where the PST-as-UTC is interpreted as 
GMT-as (say) EST. That is, again, the time is a number relative to the epoch 
UTC (set as if that were PST), but reinterpreted as seconds since the epoch EST.

The point is, the same number is sometimes PST, sometimes EST but (to Java) is 
UTC. (This whole area is terribly confusing, so the above may be slightly off; 
DRILL-5360, DRILL-5334 explored these issues in detail.)

This can push the date into a different day.

This is by design (though, it can be argued, not perhaps the best possible 
design).

Tests must somehow work around this creative interpretation of dates.

> date time test cases is not Local independent
> -
>
> Key: DRILL-5717
> URL: https://issues.apache.org/jira/browse/DRILL-5717
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>
> Some date time test cases like  JodaDateValidatorTest  is not Local 
> independent .This will cause other Local's users's test phase to fail. We 
> should let these test cases to be Local env independent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5431) Support SSL

[Laurent Goujon (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124141#comment-16124141
 ] 

Laurent Goujon commented on DRILL-5431:
---

Can we allow comments for the document? or shall we comment on the JIRA itself?

> Support SSL
> ---
>
> Key: DRILL-5431
> URL: https://issues.apache.org/jira/browse/DRILL-5431
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Client - Java, Client - ODBC
>Reporter: Sudheesh Katkam
>Assignee: Sudheesh Katkam
>
> Support SSL between Drillbit and JDBC/ODBC drivers. Drill already supports 
> HTTPS for web traffic.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5704) Improve error message on client side when queries fail with "Failed to create schema tree." when Impersonation is enabled and logins are anonymous

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124006#comment-16124006
 ] 

ASF GitHub Bot commented on DRILL-5704:
---

Github user parthchandra commented on the issue:

https://github.com/apache/drill/pull/895
  
+1


> Improve error message on client side when queries fail with "Failed to create 
> schema tree." when Impersonation is enabled and logins are anonymous
> --
>
> Key: DRILL-5704
> URL: https://issues.apache.org/jira/browse/DRILL-5704
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
> Fix For: 1.12.0
>
>
> Reported by [~agirish]
> When username is not specified then Drill set's the session user as anonymous 
> if impersonation is enabled. During query execution Drill tries to build 
> schema tree and as part of that it validates if the user has access to the 
> workspace or not by using FileClient Api liststatus which verifies the user 
> from the OS user. Since impersonation is only enabled here without 
> authentication and we don't specify any user in connection string, Drill will 
> use default user which is "anonymous" and pass that to check workspace 
> permission which will fail as node doesn't have any valid user with that name.
> {code:java}
> Caused by: java.io.IOException: Error getting user info for current user, 
> anonymous
>..
>..
> at 
> org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus(DrillFileSystem.java:523)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible(WorkspaceSchemaFactory.java:157)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.(FileSystemSchemaFactory.java:78)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas(FileSystemSchemaFactory.java:65)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas(FileSystemPlugin.java:150)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:365)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:72)
>  [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> ... 10 common frames omitted
> {code}
> # $DRILL_HOME/bin/sqlline -u "jdbc:drill:zk=localhost:5181" 
> sqlline> select * from sys.drillbits;
> User Error Occurred
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: Failed to 
> create schema tree.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5704) Improve error message on client side when queries fail with "Failed to create schema tree." when Impersonation is enabled and logins are anonymous

[Sorabh Hamirwasia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sorabh Hamirwasia updated DRILL-5704:
-
Labels: ready-to-commit  (was: )

> Improve error message on client side when queries fail with "Failed to create 
> schema tree." when Impersonation is enabled and logins are anonymous
> --
>
> Key: DRILL-5704
> URL: https://issues.apache.org/jira/browse/DRILL-5704
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Reported by [~agirish]
> When username is not specified then Drill set's the session user as anonymous 
> if impersonation is enabled. During query execution Drill tries to build 
> schema tree and as part of that it validates if the user has access to the 
> workspace or not by using FileClient Api liststatus which verifies the user 
> from the OS user. Since impersonation is only enabled here without 
> authentication and we don't specify any user in connection string, Drill will 
> use default user which is "anonymous" and pass that to check workspace 
> permission which will fail as node doesn't have any valid user with that name.
> {code:java}
> Caused by: java.io.IOException: Error getting user info for current user, 
> anonymous
>..
>..
> at 
> org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus(DrillFileSystem.java:523)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible(WorkspaceSchemaFactory.java:157)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.(FileSystemSchemaFactory.java:78)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas(FileSystemSchemaFactory.java:65)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas(FileSystemPlugin.java:150)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:365)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:72)
>  [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> ... 10 common frames omitted
> {code}
> # $DRILL_HOME/bin/sqlline -u "jdbc:drill:zk=localhost:5181" 
> sqlline> select * from sys.drillbits;
> User Error Occurred
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: Failed to 
> create schema tree.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5601) Rollup of External Sort memory management fixes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124217#comment-16124217
 ] 

ASF GitHub Bot commented on DRILL-5601:
---

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/860
  
Rebased onto master.


> Rollup of External Sort memory management fixes
> ---
>
> Key: DRILL-5601
> URL: https://issues.apache.org/jira/browse/DRILL-5601
> Project: Apache Drill
>  Issue Type: Task
>Affects Versions: 1.11.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Rollup of a set of specific JIRA entries that all relate to the very 
> difficult problem of managing memory within Drill in order for the external 
> sort to stay within a memory budget. In general, the fixes relate to better 
> estimating memory used by the three ways that Drill allocates vector memory 
> (see DRILL-5522) and to predicting the size of vectors that the sort will 
> create, to avoid repeated realloc-copy cycles (see DRILL-5594).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.

[Sindhuri Ramanarayan Rayavaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sindhuri Ramanarayan Rayavaram updated DRILL-5663:
--
Labels: ready-to-commit  (was: )

> Drillbit fails to start when only keystore path is provided without keystore 
> password.
> --
>
> Key: DRILL-5663
> URL: https://issues.apache.org/jira/browse/DRILL-5663
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Sorabh Hamirwasia
>Assignee: Sindhuri Ramanarayan Rayavaram
>  Labels: ready-to-commit
>
> When we configure keystore path without keystore password inside 
> drill-override.conf for WebServer, then Drillbit fails to start. We should 
> explicitly check for either both being present or both being absent. If any 
> one of them is only present then throw startup exception for Drill.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4735) Count(dir0) on parquet returns 0 result

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124339#comment-16124339
 ] 

ASF GitHub Bot commented on DRILL-4735:
---

Github user jinfengni commented on the issue:

https://github.com/apache/drill/pull/900
  
+1

LGTM. Thanks for the PR!



> Count(dir0) on parquet returns 0 result
> ---
>
> Key: DRILL-4735
> URL: https://issues.apache.org/jira/browse/DRILL-4735
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization, Storage - Parquet
>Affects Versions: 1.0.0, 1.4.0, 1.6.0, 1.7.0
>Reporter: Krystal
>Assignee: Arina Ielchiieva
>Priority: Critical
>
> Selecting a count of dir0, dir1, etc against a parquet directory returns 0 
> rows.
> select count(dir0) from `min_max_dir`;
> +-+
> | EXPR$0  |
> +-+
> | 0   |
> +-+
> select count(dir1) from `min_max_dir`;
> +-+
> | EXPR$0  |
> +-+
> | 0   |
> +-+
> If I put both dir0 and dir1 in the same select, it returns expected result:
> select count(dir0), count(dir1) from `min_max_dir`;
> +-+-+
> | EXPR$0  | EXPR$1  |
> +-+-+
> | 600 | 600 |
> +-+-+
> Here is the physical plan for count(dir0) query:
> {code}
> 00-00Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 20.0, 
> cumulative cost = {22.0 rows, 22.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id 
> = 1346
> 00-01  Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): 
> rowcount = 20.0, cumulative cost = {20.0 rows, 20.0 cpu, 0.0 io, 0.0 network, 
> 0.0 memory}, id = 1345
> 00-02Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): 
> rowcount = 20.0, cumulative cost = {20.0 rows, 20.0 cpu, 0.0 io, 0.0 network, 
> 0.0 memory}, id = 1344
> 00-03  
> Scan(groupscan=[org.apache.drill.exec.store.pojo.PojoRecordReader@3da85d3b[columns
>  = null, isStarQuery = false, isSkipQuery = false]]) : rowType = 
> RecordType(BIGINT count): rowcount = 20.0, cumulative cost = {20.0 rows, 20.0 
> cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1343
> {code}
> Here is part of the explain plan for the count(dir0) and count(dir1) in the 
> same select:
> {code}
> 00-00Screen : rowType = RecordType(BIGINT EXPR$0, BIGINT EXPR$1): 
> rowcount = 60.0, cumulative cost = {1206.0 rows, 15606.0 cpu, 0.0 io, 0.0 
> network, 0.0 memory}, id = 1623
> 00-01  Project(EXPR$0=[$0], EXPR$1=[$1]) : rowType = RecordType(BIGINT 
> EXPR$0, BIGINT EXPR$1): rowcount = 60.0, cumulative cost = {1200.0 rows, 
> 15600.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1622
> 00-02StreamAgg(group=[{}], EXPR$0=[COUNT($0)], EXPR$1=[COUNT($1)]) : 
> rowType = RecordType(BIGINT EXPR$0, BIGINT EXPR$1): rowcount = 60.0, 
> cumulative cost = {1200.0 rows, 15600.0 cpu, 0.0 io, 0.0 network, 0.0 
> memory}, id = 1621
> 00-03  Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath 
> [path=maprfs:/drill/testdata/min_max_dir/1999/Apr/voter20.parquet/0_0_0.parquet],
>  ReadEntryWithPath 
> [path=maprfs:/drill/testdata/min_max_dir/1999/MAR/voter15.parquet/0_0_0.parquet],
>  ReadEntryWithPath 
> [path=maprfs:/drill/testdata/min_max_dir/1985/jan/voter5.parquet/0_0_0.parquet],
>  ReadEntryWithPath 
> [path=maprfs:/drill/testdata/min_max_dir/1985/apr/voter60.parquet/0_0_0.parquet],...,
>  ReadEntryWithPath 
> [path=maprfs:/drill/testdata/min_max_dir/2014/jul/voter35.parquet/0_0_0.parquet]],
>  selectionRoot=maprfs:/drill/testdata/min_max_dir, numFiles=16, 
> usedMetadataFile=false, columns=[`dir0`, `dir1`]]]) : rowType = 
> RecordType(ANY dir0, ANY dir1): rowcount = 600.0, cumulative cost = {600.0 
> rows, 1200.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1620
> {code}
> Notice that in the first case, 
> "org.apache.drill.exec.store.pojo.PojoRecordReader" is used.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-4735) Count(dir0) on parquet returns 0 result

[Jinfeng Ni (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni updated DRILL-4735:
--
Labels: ready-to-commit  (was: )

> Count(dir0) on parquet returns 0 result
> ---
>
> Key: DRILL-4735
> URL: https://issues.apache.org/jira/browse/DRILL-4735
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization, Storage - Parquet
>Affects Versions: 1.0.0, 1.4.0, 1.6.0, 1.7.0
>Reporter: Krystal
>Assignee: Arina Ielchiieva
>Priority: Critical
>  Labels: ready-to-commit
>
> Selecting a count of dir0, dir1, etc against a parquet directory returns 0 
> rows.
> select count(dir0) from `min_max_dir`;
> +-+
> | EXPR$0  |
> +-+
> | 0   |
> +-+
> select count(dir1) from `min_max_dir`;
> +-+
> | EXPR$0  |
> +-+
> | 0   |
> +-+
> If I put both dir0 and dir1 in the same select, it returns expected result:
> select count(dir0), count(dir1) from `min_max_dir`;
> +-+-+
> | EXPR$0  | EXPR$1  |
> +-+-+
> | 600 | 600 |
> +-+-+
> Here is the physical plan for count(dir0) query:
> {code}
> 00-00Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 20.0, 
> cumulative cost = {22.0 rows, 22.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id 
> = 1346
> 00-01  Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): 
> rowcount = 20.0, cumulative cost = {20.0 rows, 20.0 cpu, 0.0 io, 0.0 network, 
> 0.0 memory}, id = 1345
> 00-02Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): 
> rowcount = 20.0, cumulative cost = {20.0 rows, 20.0 cpu, 0.0 io, 0.0 network, 
> 0.0 memory}, id = 1344
> 00-03  
> Scan(groupscan=[org.apache.drill.exec.store.pojo.PojoRecordReader@3da85d3b[columns
>  = null, isStarQuery = false, isSkipQuery = false]]) : rowType = 
> RecordType(BIGINT count): rowcount = 20.0, cumulative cost = {20.0 rows, 20.0 
> cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1343
> {code}
> Here is part of the explain plan for the count(dir0) and count(dir1) in the 
> same select:
> {code}
> 00-00Screen : rowType = RecordType(BIGINT EXPR$0, BIGINT EXPR$1): 
> rowcount = 60.0, cumulative cost = {1206.0 rows, 15606.0 cpu, 0.0 io, 0.0 
> network, 0.0 memory}, id = 1623
> 00-01  Project(EXPR$0=[$0], EXPR$1=[$1]) : rowType = RecordType(BIGINT 
> EXPR$0, BIGINT EXPR$1): rowcount = 60.0, cumulative cost = {1200.0 rows, 
> 15600.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1622
> 00-02StreamAgg(group=[{}], EXPR$0=[COUNT($0)], EXPR$1=[COUNT($1)]) : 
> rowType = RecordType(BIGINT EXPR$0, BIGINT EXPR$1): rowcount = 60.0, 
> cumulative cost = {1200.0 rows, 15600.0 cpu, 0.0 io, 0.0 network, 0.0 
> memory}, id = 1621
> 00-03  Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath 
> [path=maprfs:/drill/testdata/min_max_dir/1999/Apr/voter20.parquet/0_0_0.parquet],
>  ReadEntryWithPath 
> [path=maprfs:/drill/testdata/min_max_dir/1999/MAR/voter15.parquet/0_0_0.parquet],
>  ReadEntryWithPath 
> [path=maprfs:/drill/testdata/min_max_dir/1985/jan/voter5.parquet/0_0_0.parquet],
>  ReadEntryWithPath 
> [path=maprfs:/drill/testdata/min_max_dir/1985/apr/voter60.parquet/0_0_0.parquet],...,
>  ReadEntryWithPath 
> [path=maprfs:/drill/testdata/min_max_dir/2014/jul/voter35.parquet/0_0_0.parquet]],
>  selectionRoot=maprfs:/drill/testdata/min_max_dir, numFiles=16, 
> usedMetadataFile=false, columns=[`dir0`, `dir1`]]]) : rowType = 
> RecordType(ANY dir0, ANY dir1): rowcount = 600.0, cumulative cost = {600.0 
> rows, 1200.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 1620
> {code}
> Notice that in the first case, 
> "org.apache.drill.exec.store.pojo.PojoRecordReader" is used.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (DRILL-4211) Column aliases not pushed down to JDBC stores in some cases when Drill expects aliased columns to be returned.

[Timothy Farkas (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122525#comment-16122525
 ] 

Timothy Farkas edited comment on DRILL-4211 at 8/12/17 12:39 AM:
-

After discussing with Paul and Jinfeng, this issue can be fixed by pushing the 
whole select portion of the query down into the JDBC operator. And required the 
select portion of the query to alias potentially conflicting columns.


was (Author: timothyfarkas):
After discussing with Paul and Juinfeng, this issue can be fixed by pushing the 
whole select portion of the query down into the JDBC operator. And required the 
select portion of the query to alias potentially conflicting columns.

> Column aliases not pushed down to JDBC stores in some cases when Drill 
> expects aliased columns to be returned.
> --
>
> Key: DRILL-4211
> URL: https://issues.apache.org/jira/browse/DRILL-4211
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.3.0, 1.11.0
> Environment: Postgres db storage
>Reporter: Robert Hamilton-Smith
>Assignee: Timothy Farkas
>  Labels: newbie
> Fix For: 1.12.0
>
>
> When making an sql statement that incorporates a join to a table and then a 
> self join to that table to get a parent value , Drill brings back 
> inconsistent results. 
> Here is the sql in postgres with correct output:
> {code:sql}
> select trx.categoryguid,
> cat.categoryname, w1.categoryname as parentcat
> from transactions trx
> join categories cat on (cat.CATEGORYGUID = trx.CATEGORYGUID)
> join categories w1 on (cat.categoryparentguid = w1.categoryguid)
> where cat.categoryparentguid IS NOT NULL;
> {code}
> Output:
> ||categoryid||categoryname||parentcategory||
> |id1|restaurants|food|
> |id1|restaurants|food|
> |id2|Coffee Shops|food|
> |id2|Coffee Shops|food|
> When run in Drill with correct storage prefix:
> {code:sql}
> select trx.categoryguid,
> cat.categoryname, w1.categoryname as parentcat
> from db.schema.transactions trx
> join db.schema.categories cat on (cat.CATEGORYGUID = trx.CATEGORYGUID)
> join db.schema.wpfm_categories w1 on (cat.categoryparentguid = 
> w1.categoryguid)
> where cat.categoryparentguid IS NOT NULL
> {code}
> Results are:
> ||categoryid||categoryname||parentcategory||
> |id1|restaurants|null|
> |id1|restaurants|null|
> |id2|Coffee Shops|null|
> |id2|Coffee Shops|null|
> Physical plan is:
> {code:sql}
> 00-00Screen : rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) 
> categoryname, VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = 
> {110.0 rows, 110.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64293
> 00-01  Project(categoryguid=[$0], categoryname=[$1], parentcat=[$2]) : 
> rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) categoryname, 
> VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = {100.0 rows, 
> 100.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64292
> 00-02Project(categoryguid=[$9], categoryname=[$41], parentcat=[$47]) 
> : rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) categoryname, 
> VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = {100.0 rows, 
> 100.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64291
> 00-03  Jdbc(sql=[SELECT *
> FROM "public"."transactions"
> INNER JOIN (SELECT *
> FROM "public"."categories"
> WHERE "categoryparentguid" IS NOT NULL) AS "t" ON 
> "transactions"."categoryguid" = "t"."categoryguid"
> INNER JOIN "public"."categories" AS "categories0" ON "t"."categoryparentguid" 
> = "categories0"."categoryguid"]) : rowType = RecordType(VARCHAR(255) 
> transactionguid, VARCHAR(255) relatedtransactionguid, VARCHAR(255) 
> transactioncode, DECIMAL(1, 0) transactionpending, VARCHAR(50) 
> transactionrefobjecttype, VARCHAR(255) transactionrefobjectguid, 
> VARCHAR(1024) transactionrefobjectvalue, TIMESTAMP(6) transactiondate, 
> VARCHAR(256) transactiondescription, VARCHAR(50) categoryguid, VARCHAR(3) 
> transactioncurrency, DECIMAL(15, 3) transactionoldbalance, DECIMAL(13, 3) 
> transactionamount, DECIMAL(15, 3) transactionnewbalance, VARCHAR(512) 
> transactionnotes, DECIMAL(2, 0) transactioninstrumenttype, VARCHAR(20) 
> transactioninstrumentsubtype, VARCHAR(20) transactioninstrumentcode, 
> VARCHAR(50) transactionorigpartyguid, VARCHAR(255) 
> transactionorigaccountguid, VARCHAR(50) transactionrecpartyguid, VARCHAR(255) 
> transactionrecaccountguid, VARCHAR(256) transactionstatementdesc, DECIMAL(1, 
> 0) transactionsplit, DECIMAL(1, 0) transactionduplicated, DECIMAL(1, 0) 
> transactionrecategorized, TIMESTAMP(6) transactioncreatedat, TIMESTAMP(6) 
> transactionupdatedat, VARCHAR(50) transactionmatrulerefobjtype, 

[jira] [Commented] (DRILL-4398) SYSTEM ERROR: IllegalStateException: Memory was leaked by query

[Muhammad Gelbana (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124376#comment-16124376
 ] 

Muhammad Gelbana commented on DRILL-4398:
-

Thanks [~zfong] for the response. I'll open a new issue shortly.
{noformat}
Fragment 0:0

[Error Id: 0403a63e-86cc-428e-929b-e8dcd561a6bf on iWebStitchFixDev:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.drill.exec.rpc.ChannelClosedException: Channel closed 
/72.55.136.6:31010 <--> /72.55.136.6:40834.
at 
org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:166)
at 
org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:146)
at 
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
at 
io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
at 
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
at 
io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
at 
io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
at 
io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
at 
io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
at 
io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
at 
io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1099)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:615)
at 
io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:600)
at 
io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:615)
at 
io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:600)
at 
io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:466)
at 
io.netty.handler.timeout.ReadTimeoutHandler.readTimedOut(ReadTimeoutHandler.java:187)
at 
org.apache.drill.exec.rpc.BasicServer$LogggingReadTimeoutHandler.readTimedOut(BasicServer.java:121)
at 
io.netty.handler.timeout.ReadTimeoutHandler$ReadTimeoutTask.run(ReadTimeoutHandler.java:212)
at 
io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at 
io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
at 
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
... 1 more
Suppressed: org.apache.drill.exec.rpc.RpcException: Failure sending 
message.
at org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:126)
at 
org.apache.drill.exec.rpc.user.UserServer$UserClientConnectionImpl.sendData(UserServer.java:285)
at 
org.apache.drill.exec.ops.AccountingUserConnection.sendData(AccountingUserConnection.java:42)
at 
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:118)
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:232)
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:226)
at 

[jira] [Commented] (DRILL-5714) Fix NPE when mapr-db plugin is used in table function

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124235#comment-16124235
 ] 

ASF GitHub Bot commented on DRILL-5714:
---

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/902
  
Thanks for the explanation. Looks good.
+1 (again)


> Fix NPE when mapr-db plugin is used in table function
> -
>
> Key: DRILL-5714
> URL: https://issues.apache.org/jira/browse/DRILL-5714
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - MapRDB
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Query:
> {noformat}
> select * from table(mapr.`/tmp/test`(type=>'maprdb', allTextMode=>true))
> {noformat}
> Error:
> {noformat}
> 2017-08-03 13:13:56,527 [267cde6b-6327-11b4-a4f3-f59a8d82ae17:frag:0:0] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException
> Fragment 0:0
> [Error Id: 07937ad2-90ce-43e3-b9c1-2540e1df05fa on master:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> NullPointerException
> Fragment 0:0
> [Error Id: 07937ad2-90ce-43e3-b9c1-2540e1df05fa on master:31010]
>   at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
>  ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:295)
>  [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264)
>  [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
>   at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_95]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_95]
>   at java.lang.Thread.run(Thread.java:745) [na:1.7.0_95]
> Caused by: org.apache.drill.common.exceptions.ExecutionSetupException: 
> java.lang.NullPointerException
>   at 
> org.apache.drill.exec.store.mapr.db.MapRDBScanBatchCreator.getBatch(MapRDBScanBatchCreator.java:52)
>  ~[drill-format-mapr-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.store.mapr.db.MapRDBScanBatchCreator.getBatch(MapRDBScanBatchCreator.java:36)
>  ~[drill-format-mapr-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:156)
>  ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:179)
>  ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:136)
>  ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:179)
>  ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:109)
>  ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:87) 
> ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:207)
>  [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
>   ... 4 common frames omitted
> Caused by: java.lang.NullPointerException: null
>   at 
> org.apache.drill.exec.store.mapr.db.MapRDBScanBatchCreator.getBatch(MapRDBScanBatchCreator.java:46)
>  ~[drill-format-mapr-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
>   ... 12 common frames omitted
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5717) date time test cases is not Local independent

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124371#comment-16124371
 ] 

ASF GitHub Bot commented on DRILL-5717:
---

Github user weijietong commented on the issue:

https://github.com/apache/drill/pull/904
  
Yes, thanks for the notice ,have noticed those cases before ,the work is in 
process.some of that will be part of this PR. Others like sql_to_timestamp 
function need to discuss whether it should have a Local parameter or not set 
the specific UTC timezone when get the date.


> date time test cases is not Local independent
> -
>
> Key: DRILL-5717
> URL: https://issues.apache.org/jira/browse/DRILL-5717
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Affects Versions: 1.9.0, 1.11.0
>Reporter: weijie.tong
>
> Some date time test cases like  JodaDateValidatorTest  is not Local 
> independent .This will cause other Local's users's test phase to fail. We 
> should let these test cases to be Local env independent.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5709) Provide a value vector method to convert a vector to nullable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124396#comment-16124396
 ] 

ASF GitHub Bot commented on DRILL-5709:
---

Github user Ben-Zvi commented on a diff in the pull request:

https://github.com/apache/drill/pull/901#discussion_r132806738
  
--- Diff: 
exec/vector/src/main/java/org/apache/drill/exec/vector/BaseValueVector.java ---
@@ -133,5 +134,21 @@ public static boolean checkBufRefs(final ValueVector 
vv) {
   public BufferAllocator getAllocator() {
 return allocator;
   }
+
+  public static void fillBitsVector(UInt1Vector bits, int valueCount) {
+
+// Create a new bits vector, all values non-null
+
+bits.allocateNew(valueCount);
+UInt1Vector.Mutator bitsMutator = bits.getMutator();
+for (int i = 0; i < valueCount; i++) {
+  bitsMutator.set(i, 1);
+}
+  }
--- End diff --

And need to add:

bitsMutator.setValueCount(valueCount);




> Provide a value vector method to convert a vector to nullable
> -
>
> Key: DRILL-5709
> URL: https://issues.apache.org/jira/browse/DRILL-5709
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.12.0
>
>
> The hash agg spill work has need to convert a non-null scalar vector to the 
> nullable equivalent. For efficiency, the code wishes to simply transfer the 
> underlying data buffer(s), and create the required "bits" vector, rather than 
> generating code that does the transfer row-by-row.
> The solution is to add a {{toNullable(ValueVector nullableVector)}} method to 
> the {{ValueVector}} class, then implement it where needed.
> Since the target code only works with scalars (that is, no arrays, no maps, 
> no lists), the code only handles these cases, throwing an 
> {{UnsupportedOperationException}} in other cases.
> Usage:
> {code}
> ValueVector nonNullableVector = // your non-nullable vector
> MajorType type = MajorType.newBuilder(nonNullableVector.getType)
> .setMode(DataMode.OPTIONAL)
> .build();
> MaterializedField field = MaterializedField.create(name, type);
> ValueVector nullableVector = TypeHelper.getNewVector(field, 
> oContext.getAllocator());
> nonNullableVector.toNullable(nullableVector);
> // Data is now in nullableVector
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5709) Provide a value vector method to convert a vector to nullable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124397#comment-16124397
 ] 

ASF GitHub Bot commented on DRILL-5709:
---

Github user Ben-Zvi commented on a diff in the pull request:

https://github.com/apache/drill/pull/901#discussion_r132806694
  
--- Diff: 
exec/vector/src/main/java/org/apache/drill/exec/vector/BaseValueVector.java ---
@@ -133,5 +134,21 @@ public static boolean checkBufRefs(final ValueVector 
vv) {
   public BufferAllocator getAllocator() {
 return allocator;
   }
+
+  public static void fillBitsVector(UInt1Vector bits, int valueCount) {
+
+// Create a new bits vector, all values non-null
+
+bits.allocateNew(valueCount);
+UInt1Vector.Mutator bitsMutator = bits.getMutator();
+for (int i = 0; i < valueCount; i++) {
+  bitsMutator.set(i, 1);
+}
--- End diff --

This loop may be a bit expensive; Is there a way to mark the whole bitmap 
with '1's "at once" ?  A la "memset()".  
For example, keep a constant bitmap (of max length - 64K) someplace, and 
here just dup/copy it whole.



> Provide a value vector method to convert a vector to nullable
> -
>
> Key: DRILL-5709
> URL: https://issues.apache.org/jira/browse/DRILL-5709
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.12.0
>
>
> The hash agg spill work has need to convert a non-null scalar vector to the 
> nullable equivalent. For efficiency, the code wishes to simply transfer the 
> underlying data buffer(s), and create the required "bits" vector, rather than 
> generating code that does the transfer row-by-row.
> The solution is to add a {{toNullable(ValueVector nullableVector)}} method to 
> the {{ValueVector}} class, then implement it where needed.
> Since the target code only works with scalars (that is, no arrays, no maps, 
> no lists), the code only handles these cases, throwing an 
> {{UnsupportedOperationException}} in other cases.
> Usage:
> {code}
> ValueVector nonNullableVector = // your non-nullable vector
> MajorType type = MajorType.newBuilder(nonNullableVector.getType)
> .setMode(DataMode.OPTIONAL)
> .build();
> MaterializedField field = MaterializedField.create(name, type);
> ValueVector nullableVector = TypeHelper.getNewVector(field, 
> oContext.getAllocator());
> nonNullableVector.toNullable(nullableVector);
> // Data is now in nullableVector
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124260#comment-16124260
 ] 

ASF GitHub Bot commented on DRILL-5663:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/874#discussion_r132800051
  
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/SSLConfig.java 
---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec;
+
+import com.typesafe.config.Config;
+import org.apache.drill.common.exceptions.DrillException;
+
+public class SSLConfig {
+
+  private final String keystorePath;
+
+  private final String keystorePassword;
+
+  private final String truststorePath;
+
+  private final String truststorePassword;
+
+
+  public SSLConfig(Config config) throws DrillException {
+
+keystorePath = 
config.getString(ExecConstants.HTTP_KEYSTORE_PATH).trim();
+
+keystorePassword = 
config.getString(ExecConstants.HTTP_KEYSTORE_PASSWORD).trim();
+
+truststorePath = 
config.getString(ExecConstants.HTTP_TRUSTSTORE_PATH).trim();
+
+truststorePassword = 
config.getString(ExecConstants.HTTP_TRUSTSTORE_PASSWORD).trim();
+
+/*If keystorePath or keystorePassword is provided in the configuration 
file use that*/
+if (!keystorePath.isEmpty() || !keystorePassword.isEmpty()) {
+  if (keystorePath.isEmpty()) {
+throw new DrillException(" *.ssl.keyStorePath in the configuration 
file is empty, but *.ssl.keyStorePassword is set");
+  }
+  else if (keystorePassword.isEmpty()){
+throw new DrillException(" *.ssl.keyStorePassword in the 
configuration file is empty, but *.ssl.keyStorePath is set ");
+  }
+
+}
+  }
+
+  public boolean isSslValid() { return !keystorePath.isEmpty() && 
!keystorePassword.isEmpty(); }
+
+  public String getKeyStorePath() {
+return keystorePath;
+  }
+
+  public String getKeyStorePassword() {
+return keystorePassword;
+  }
+
+  public boolean hasTrustStorePath() { return !truststorePath.isEmpty(); }
+
+  public boolean hasTrustStorePassword() {return 
!truststorePassword.isEmpty(); }
+
+  public String getTrustStorePath() {
+return truststorePath;
+  }
+
+  public String getTrustStorePassword() {
+return truststorePassword;
+  }
--- End diff --

Very minor suggestion: these simple returns can also be one-liners like the 
"has" methods.


> Drillbit fails to start when only keystore path is provided without keystore 
> password.
> --
>
> Key: DRILL-5663
> URL: https://issues.apache.org/jira/browse/DRILL-5663
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Sorabh Hamirwasia
>Assignee: Sindhuri Ramanarayan Rayavaram
>
> When we configure keystore path without keystore password inside 
> drill-override.conf for WebServer, then Drillbit fails to start. We should 
> explicitly check for either both being present or both being absent. If any 
> one of them is only present then throw startup exception for Drill.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124256#comment-16124256
 ] 

ASF GitHub Bot commented on DRILL-5663:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/874#discussion_r132800512
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/TestSSLConfig.java ---
@@ -0,0 +1,108 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec;
+
+
+import org.apache.drill.common.exceptions.DrillException;
+import org.apache.drill.test.ConfigBuilder;
+import org.junit.Test;
+import static junit.framework.TestCase.fail;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class TestSSLConfig {
+
+  @Test
+  public void testMissingKeystorePath() throws Exception {
+
+ConfigBuilder config = new ConfigBuilder();
+config.put(ExecConstants.HTTP_KEYSTORE_PATH, "");
+config.put(ExecConstants.HTTP_KEYSTORE_PASSWORD, "root");
+try {
+  SSLConfig sslv = new SSLConfig(config.build());
+  fail();
+  //Expected
+} catch (Exception e) {
+  assertTrue(e instanceof DrillException);
+}
+  }
+
+  @Test
+  public void testMissingKeystorePassword() throws Exception {
+
+ConfigBuilder config = new ConfigBuilder();
+config.put(ExecConstants.HTTP_KEYSTORE_PATH, "/root");
+config.put(ExecConstants.HTTP_KEYSTORE_PASSWORD, "");
+try {
+  SSLConfig sslv = new SSLConfig(config.build());
+  fail();
+  //Expected
+} catch (Exception e) {
+  assertTrue(e instanceof DrillException);
+}
+  }
+
+  @Test
+  public void testForKeystoreConfig() throws Exception {
+
+ConfigBuilder config = new ConfigBuilder();
+config.put(ExecConstants.HTTP_KEYSTORE_PATH, "/root");
+config.put(ExecConstants.HTTP_KEYSTORE_PASSWORD, "root");
+try {
+  SSLConfig sslv = new SSLConfig(config.build());
+  assertEquals("/root", sslv.getKeyStorePath());
+  assertEquals("root", sslv.getKeyStorePassword());
+} catch (Exception e) {
+  fail();
+
+}
+  }
+
+  @Test
+  public void testForBackwardCompatability() throws Exception {
+
+ConfigBuilder config = new ConfigBuilder();
+config.put("javax.net.ssl.keyStore", "/root");
+config.put("javax.net.ssl.keyStorePassword", "root");
+SSLConfig sslv = new SSLConfig(config.build());
+assertEquals("/root",sslv.getKeyStorePath());
+assertEquals("root", sslv.getKeyStorePassword());
+  }
+
+  @Test
+  public void testForTrustStore() throws Exception {
+
+ConfigBuilder config = new ConfigBuilder();
+config.put(ExecConstants.HTTP_TRUSTSTORE_PATH, "/root");
+config.put(ExecConstants.HTTP_TRUSTSTORE_PASSWORD, "root");
+SSLConfig sslv = new SSLConfig(config.build());
+assertEquals(true, sslv.hasTrustStorePath());
+assertEquals(true,sslv.hasTrustStorePassword());
+assertEquals("/root",sslv.getTrustStorePath());
+assertEquals("root",sslv.getTrustStorePassword());
+  }
+}
--- End diff --

Thanks for the tests!

Maybe delete the extra lines at the end of the file...


> Drillbit fails to start when only keystore path is provided without keystore 
> password.
> --
>
> Key: DRILL-5663
> URL: https://issues.apache.org/jira/browse/DRILL-5663
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Sorabh Hamirwasia
>Assignee: Sindhuri Ramanarayan Rayavaram
>
> When we configure keystore path without keystore password inside 
> drill-override.conf for WebServer, then Drillbit fails to 

[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124258#comment-16124258
 ] 

ASF GitHub Bot commented on DRILL-5663:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/874#discussion_r132800128
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/WebServer.java 
---
@@ -263,18 +272,17 @@ private ServerConnector createHttpsConnector() throws 
Exception {
 logger.info("Setting up HTTPS connector for web server");
 
 final SslContextFactory sslContextFactory = new SslContextFactory();
+SSLConfig ssl = new SSLConfig(config);
 
-if (config.hasPath(ExecConstants.HTTP_KEYSTORE_PATH) &&
-
!Strings.isNullOrEmpty(config.getString(ExecConstants.HTTP_KEYSTORE_PATH))) {
+if(ssl.isSslValid()){
--- End diff --

{code}
if (...) {
{code}


> Drillbit fails to start when only keystore path is provided without keystore 
> password.
> --
>
> Key: DRILL-5663
> URL: https://issues.apache.org/jira/browse/DRILL-5663
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Sorabh Hamirwasia
>Assignee: Sindhuri Ramanarayan Rayavaram
>
> When we configure keystore path without keystore password inside 
> drill-override.conf for WebServer, then Drillbit fails to start. We should 
> explicitly check for either both being present or both being absent. If any 
> one of them is only present then throw startup exception for Drill.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124254#comment-16124254
 ] 

ASF GitHub Bot commented on DRILL-5663:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/874#discussion_r132799898
  
--- Diff: distribution/src/resources/drill-override-example.conf ---
@@ -93,6 +93,13 @@ drill.exec: {
   credentials: true
 }
   },
+  # Below SSL parameters need to be set for custom transport layer 
settings.
+  ssl{
+keyStorePath: "/keystore.file",
+keyStorePassword: "ks_passwd",
+trustStorePath: "/truststore.file",
+trustStorePassword: "ts_passwd"
--- End diff --

ssl{ --> ssl: {


> Drillbit fails to start when only keystore path is provided without keystore 
> password.
> --
>
> Key: DRILL-5663
> URL: https://issues.apache.org/jira/browse/DRILL-5663
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Sorabh Hamirwasia
>Assignee: Sindhuri Ramanarayan Rayavaram
>
> When we configure keystore path without keystore password inside 
> drill-override.conf for WebServer, then Drillbit fails to start. We should 
> explicitly check for either both being present or both being absent. If any 
> one of them is only present then throw startup exception for Drill.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124257#comment-16124257
 ] 

ASF GitHub Bot commented on DRILL-5663:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/874#discussion_r132800461
  
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -52,6 +52,14 @@ drill.client: {
 drill.tmp-dir: "/tmp"
 drill.tmp-dir: ${?DRILL_TMP_DIR}
 
+javax.net.ssl :
+ {
+ keyStore = "",
+ keyStorePassword = "",
+ trustStore = "",
+ trustStorePassword = ""
+ },
--- End diff --

{code}
javax.net.ssl: {
  keyStore = "",
{code}

That is, adjust formatting of the header line and indent the values.


> Drillbit fails to start when only keystore path is provided without keystore 
> password.
> --
>
> Key: DRILL-5663
> URL: https://issues.apache.org/jira/browse/DRILL-5663
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Sorabh Hamirwasia
>Assignee: Sindhuri Ramanarayan Rayavaram
>
> When we configure keystore path without keystore password inside 
> drill-override.conf for WebServer, then Drillbit fails to start. We should 
> explicitly check for either both being present or both being absent. If any 
> one of them is only present then throw startup exception for Drill.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124255#comment-16124255
 ] 

ASF GitHub Bot commented on DRILL-5663:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/874#discussion_r132800101
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/WebServer.java 
---
@@ -139,7 +142,13 @@ public void start() throws Exception {
 
 final ServerConnector serverConnector;
 if (config.getBoolean(ExecConstants.HTTP_ENABLE_SSL)) {
-  serverConnector = createHttpsConnector();
+  try {
+serverConnector = createHttpsConnector();
+  }
+  catch(DrillException e){
+throw new DrillbitStartupException(e.getMessage(),e);
--- End diff --

Space after comma.


> Drillbit fails to start when only keystore path is provided without keystore 
> password.
> --
>
> Key: DRILL-5663
> URL: https://issues.apache.org/jira/browse/DRILL-5663
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Sorabh Hamirwasia
>Assignee: Sindhuri Ramanarayan Rayavaram
>
> When we configure keystore path without keystore password inside 
> drill-override.conf for WebServer, then Drillbit fails to start. We should 
> explicitly check for either both being present or both being absent. If any 
> one of them is only present then throw startup exception for Drill.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124259#comment-16124259
 ] 

ASF GitHub Bot commented on DRILL-5663:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/874#discussion_r132800092
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/WebServer.java 
---
@@ -139,7 +142,13 @@ public void start() throws Exception {
 
 final ServerConnector serverConnector;
 if (config.getBoolean(ExecConstants.HTTP_ENABLE_SSL)) {
-  serverConnector = createHttpsConnector();
+  try {
+serverConnector = createHttpsConnector();
+  }
+  catch(DrillException e){
--- End diff --

{code}
catch (DrillbitException e) {
{code}
(note spaces)


> Drillbit fails to start when only keystore path is provided without keystore 
> password.
> --
>
> Key: DRILL-5663
> URL: https://issues.apache.org/jira/browse/DRILL-5663
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Sorabh Hamirwasia
>Assignee: Sindhuri Ramanarayan Rayavaram
>
> When we configure keystore path without keystore password inside 
> drill-override.conf for WebServer, then Drillbit fails to start. We should 
> explicitly check for either both being present or both being absent. If any 
> one of them is only present then throw startup exception for Drill.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5712) Update the pom files with dependency exclusions for commons-codec

[Sindhuri Ramanarayan Rayavaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sindhuri Ramanarayan Rayavaram updated DRILL-5712:
--
Labels: ready-to-commit  (was: )

> Update the pom files with dependency exclusions for commons-codec
> -
>
> Key: DRILL-5712
> URL: https://issues.apache.org/jira/browse/DRILL-5712
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Reporter: Sindhuri Ramanarayan Rayavaram
>Assignee: Sindhuri Ramanarayan Rayavaram
>  Labels: ready-to-commit
>
> In java-exec, we are adding a dependency for commons-codec of version 1.10. 
> Other dependencies like hadoop-common, parquet-column etc are trying to 
> download different versions for common codec. Exclusions should be added for 
> common-codec in these dependencies.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5477) String functions (lower, upper, initcap) should work for UTF-8

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124313#comment-16124313
 ] 

Kunal Khatua commented on DRILL-5477:
-

[~smopsy], we should scope work for this, along with documentation tasks. There 
is no reference of the need for setting system property 
{{saffron.default.charset}}.

> String functions (lower, upper, initcap) should work for UTF-8
> --
>
> Key: DRILL-5477
> URL: https://issues.apache.org/jira/browse/DRILL-5477
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>
> Drill string functions lower / upper / initcap work only for ASCII, but not 
> for UTF-8. UTF-8 is a multi-byte code that requires special encoding/decoding 
> to convert to Unicode characters. Without that encoding, these functions 
> won't work for Cyrillic, Greek or any other character set with upper/lower 
> distinctions.
> Currently, when user applies these functions for UTF-8, Drill returns the 
> same value as was given.
> Example:
> {noformat}
> select upper('привет') from (values(1)) -> привет
> {noformat}
> There is disabled unit test in 
> https://github.com/arina-ielchiieva/drill/blob/master/exec/java-exec/src/test/java/org/apache/drill/exec/expr/fn/impl/TestStringFunctions.java#L33
>  which should be enabled once issue is fixed.
> Please note, by default Calcite does not allow to use UTF-8. Update system 
> property *saffron.default.charset* to *UTF-16LE* if you encounter the 
> following error:
> {noformat}
> org.apache.drill.exec.rpc.RpcException: 
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: 
> CalciteException: Failed to encode 'привет' in character set 'ISO-8859-1'
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-3640) Drill JDBC driver support Statement.setQueryTimeout(int)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124352#comment-16124352
 ] 

ASF GitHub Bot commented on DRILL-3640:
---

Github user parthchandra commented on the issue:

https://github.com/apache/drill/pull/858
  
Been looking at this and the first thing that occurs to me is that we are 
not too clear about what the timeout means in the context of ResultSet. The API 
specification is rather silent on that topic. 
The only reference I could find to this question is this one: 
http://mail-archives.apache.org/mod_mbox/db-derby-dev/200504.mbox/%3c426909ba.1060...@sun.com%3E
We have the same choices:
 

> 1. setQueryTimeout() only affects Statement.execute()
> 2.  setQueryTimeout() affects Statement.execute() and 
ResultSet.next(),starting from zero for each invocation
> 3.  setQueryTimeout() affects Statement.execute() and 
ResultSet.next(),accumulating time spent in each invocation

My own inclination was to select #2 as the appropriate behavior. In fact 
that is what I assumed before I looked at the code. Laurent's suggestion to 
implement the timeout in DrillCursor provides this behavior and is a little bit 
easier to implement.

OTOH, Kunal has chosen #3 as the right behavior. MySQL implements this 
behavior, BTW, so it is not going to be a surprise to end users. And he has 
already done the work. 

I'm +0 on this so far. Let me see if I can get a quick prototype to test 
things out.  



> Drill JDBC driver support Statement.setQueryTimeout(int)
> 
>
> Key: DRILL-3640
> URL: https://issues.apache.org/jira/browse/DRILL-3640
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Client - JDBC
>Affects Versions: 1.2.0
>Reporter: Chun Chang
>Assignee: Kunal Khatua
> Fix For: 1.12.0
>
>
> It would be nice if we have this implemented. Run away queries can be 
> automatically canceled by setting the timeout. 
> java.sql.SQLFeatureNotSupportedException: Setting network timeout is not 
> supported.
>   at 
> org.apache.drill.jdbc.impl.DrillStatementImpl.setQueryTimeout(DrillStatementImpl.java:152)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-4211) Column aliases not pushed down to JDBC stores in some cases when Drill expects aliased columns to be returned.

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-4211:
-
Fix Version/s: 1.12.0

> Column aliases not pushed down to JDBC stores in some cases when Drill 
> expects aliased columns to be returned.
> --
>
> Key: DRILL-4211
> URL: https://issues.apache.org/jira/browse/DRILL-4211
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.3.0, 1.11.0
> Environment: Postgres db storage
>Reporter: Robert Hamilton-Smith
>Assignee: Timothy Farkas
>  Labels: newbie
> Fix For: 1.12.0
>
>
> When making an sql statement that incorporates a join to a table and then a 
> self join to that table to get a parent value , Drill brings back 
> inconsistent results. 
> Here is the sql in postgres with correct output:
> {code:sql}
> select trx.categoryguid,
> cat.categoryname, w1.categoryname as parentcat
> from transactions trx
> join categories cat on (cat.CATEGORYGUID = trx.CATEGORYGUID)
> join categories w1 on (cat.categoryparentguid = w1.categoryguid)
> where cat.categoryparentguid IS NOT NULL;
> {code}
> Output:
> ||categoryid||categoryname||parentcategory||
> |id1|restaurants|food|
> |id1|restaurants|food|
> |id2|Coffee Shops|food|
> |id2|Coffee Shops|food|
> When run in Drill with correct storage prefix:
> {code:sql}
> select trx.categoryguid,
> cat.categoryname, w1.categoryname as parentcat
> from db.schema.transactions trx
> join db.schema.categories cat on (cat.CATEGORYGUID = trx.CATEGORYGUID)
> join db.schema.wpfm_categories w1 on (cat.categoryparentguid = 
> w1.categoryguid)
> where cat.categoryparentguid IS NOT NULL
> {code}
> Results are:
> ||categoryid||categoryname||parentcategory||
> |id1|restaurants|null|
> |id1|restaurants|null|
> |id2|Coffee Shops|null|
> |id2|Coffee Shops|null|
> Physical plan is:
> {code:sql}
> 00-00Screen : rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) 
> categoryname, VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = 
> {110.0 rows, 110.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64293
> 00-01  Project(categoryguid=[$0], categoryname=[$1], parentcat=[$2]) : 
> rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) categoryname, 
> VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = {100.0 rows, 
> 100.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64292
> 00-02Project(categoryguid=[$9], categoryname=[$41], parentcat=[$47]) 
> : rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) categoryname, 
> VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = {100.0 rows, 
> 100.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64291
> 00-03  Jdbc(sql=[SELECT *
> FROM "public"."transactions"
> INNER JOIN (SELECT *
> FROM "public"."categories"
> WHERE "categoryparentguid" IS NOT NULL) AS "t" ON 
> "transactions"."categoryguid" = "t"."categoryguid"
> INNER JOIN "public"."categories" AS "categories0" ON "t"."categoryparentguid" 
> = "categories0"."categoryguid"]) : rowType = RecordType(VARCHAR(255) 
> transactionguid, VARCHAR(255) relatedtransactionguid, VARCHAR(255) 
> transactioncode, DECIMAL(1, 0) transactionpending, VARCHAR(50) 
> transactionrefobjecttype, VARCHAR(255) transactionrefobjectguid, 
> VARCHAR(1024) transactionrefobjectvalue, TIMESTAMP(6) transactiondate, 
> VARCHAR(256) transactiondescription, VARCHAR(50) categoryguid, VARCHAR(3) 
> transactioncurrency, DECIMAL(15, 3) transactionoldbalance, DECIMAL(13, 3) 
> transactionamount, DECIMAL(15, 3) transactionnewbalance, VARCHAR(512) 
> transactionnotes, DECIMAL(2, 0) transactioninstrumenttype, VARCHAR(20) 
> transactioninstrumentsubtype, VARCHAR(20) transactioninstrumentcode, 
> VARCHAR(50) transactionorigpartyguid, VARCHAR(255) 
> transactionorigaccountguid, VARCHAR(50) transactionrecpartyguid, VARCHAR(255) 
> transactionrecaccountguid, VARCHAR(256) transactionstatementdesc, DECIMAL(1, 
> 0) transactionsplit, DECIMAL(1, 0) transactionduplicated, DECIMAL(1, 0) 
> transactionrecategorized, TIMESTAMP(6) transactioncreatedat, TIMESTAMP(6) 
> transactionupdatedat, VARCHAR(50) transactionmatrulerefobjtype, VARCHAR(50) 
> transactionmatrulerefobjguid, VARCHAR(50) transactionmatrulerefobjvalue, 
> VARCHAR(50) transactionuserruleguid, DECIMAL(2, 0) transactionsplitorder, 
> TIMESTAMP(6) transactionprocessedat, TIMESTAMP(6) 
> transactioncategoryassignat, VARCHAR(50) transactionsystemcategoryguid, 
> VARCHAR(50) transactionorigmandateid, VARCHAR(100) fingerprint, VARCHAR(50) 
> categoryguid0, VARCHAR(50) categoryparentguid, DECIMAL(3, 0) categorytype, 
> VARCHAR(50) categoryname, VARCHAR(50) categorydescription, VARCHAR(50) 
> partyguid, VARCHAR(50) categoryguid1,