[jira] [Commented] (DRILL-5842) Refactor and simplify the fragment, operator contexts for testing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223261#comment-16223261
 ] 

ASF GitHub Bot commented on DRILL-5842:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/978#discussion_r147546965
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/unit/MiniPlanUnitTestBase.java
 ---
@@ -360,7 +366,7 @@ public T columnsToRead(String ... columnsToRead) {
*/
   public class JsonScanBuilder extends ScanPopBuider {
 List jsonBatches = null;
-List inputPaths = Collections.EMPTY_LIST;
+List inputPaths = Collections.emptyList();
--- End diff --

why replace a field with a function call, which eventually return that 
static field? 


> Refactor and simplify the fragment, operator contexts for testing
> -
>
> Key: DRILL-5842
> URL: https://issues.apache.org/jira/browse/DRILL-5842
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Fix For: 1.12.0
>
>
> Drill's execution engine has a "fragment context" that provides state for a 
> fragment as a whole, and an "operator context" which provides state for a 
> single operator. Historically, these have both been concrete classes that 
> make generous references to the Drillbit context, and hence need a full Drill 
> server in order to operate.
> Drill has historically made extensive use of system-level testing: build the 
> entire server and fire queries at it to test each component. Over time, we 
> are augmenting that approach with unit tests: the ability to test each 
> operator (or parts of an operator) in isolation.
> Since each operator requires access to both the operator and fragment 
> context, the fact that the contexts depend on the overall server creates a 
> large barrier to unit testing. An earlier checkin started down the path of 
> defining the contexts as interfaces that can have different run-time and 
> test-time implementations to enable testing.
> This ticket asks to refactor those interfaces: simplifying the operator 
> context and introducing an interface for the fragment context. New code will 
> use these new interfaces, while older code continues to use the concrete 
> implementations. Over time, as operators are enhanced, they can be modified 
> to allow unit-level testing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5842) Refactor and simplify the fragment, operator contexts for testing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223259#comment-16223259
 ] 

ASF GitHub Bot commented on DRILL-5842:
---

Github user jinfengni commented on the issue:

https://github.com/apache/drill/pull/978
  
I understand the motivation of this PR is to refactor the various of 
"context" used in drill, so that we could get rid of JMock in test. I'm a bit 
confused after reading the statement that "we are not yet in a a position to 
replace the mocks with the test-time alternatives". I thought that's the 
purpose of this PR, yet it does not demonstrate the benefit, after going 
through big chunk of code change ?




> Refactor and simplify the fragment, operator contexts for testing
> -
>
> Key: DRILL-5842
> URL: https://issues.apache.org/jira/browse/DRILL-5842
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Fix For: 1.12.0
>
>
> Drill's execution engine has a "fragment context" that provides state for a 
> fragment as a whole, and an "operator context" which provides state for a 
> single operator. Historically, these have both been concrete classes that 
> make generous references to the Drillbit context, and hence need a full Drill 
> server in order to operate.
> Drill has historically made extensive use of system-level testing: build the 
> entire server and fire queries at it to test each component. Over time, we 
> are augmenting that approach with unit tests: the ability to test each 
> operator (or parts of an operator) in isolation.
> Since each operator requires access to both the operator and fragment 
> context, the fact that the contexts depend on the overall server creates a 
> large barrier to unit testing. An earlier checkin started down the path of 
> defining the contexts as interfaces that can have different run-time and 
> test-time implementations to enable testing.
> This ticket asks to refactor those interfaces: simplifying the operator 
> context and introducing an interface for the fragment context. New code will 
> use these new interfaces, while older code continues to use the concrete 
> implementations. Over time, as operators are enhanced, they can be modified 
> to allow unit-level testing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4990) Use new HDFS API access instead of listStatus to check if users have permissions to access workspace.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223220#comment-16223220
 ] 

ASF GitHub Bot commented on DRILL-4990:
---

Github user ppadma commented on the issue:

https://github.com/apache/drill/pull/652
  
This pull request was never merged because of a problem with windows test 
setup we have.  As a workaround, I added code to fall back to using old API if 
new API fails for some reason. All tests are passing fine with this change. 
This is nice to include in 1.12 as it provides performance improvement for 
all DFS based queries especially when there are large number of files. 
Can we review the new diffs please ?


> Use new HDFS API access instead of listStatus to check if users have 
> permissions to access workspace.
> -
>
> Key: DRILL-4990
> URL: https://issues.apache.org/jira/browse/DRILL-4990
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.8.0
>Reporter: Padma Penumarthy
>Assignee: Padma Penumarthy
>
> For every query, we build the schema tree 
> (runSQL->getPlan->getNewDefaultSchema->getRootSchema). All workspaces in all 
> storage plugins are checked and are added to the schema tree if they are 
> accessible by the user who initiated the query.  For file system plugin, 
> listStatus API is used to check if  the workspace is accessible or not 
> (WorkspaceSchemaFactory.accessible) by the user.  The idea seem to be if the 
> user does not have access to file(s) in the workspace, listStatus will 
> generate an exception and we return false. But, listStatus (which lists all 
> the entries of a directory) is an expensive operation when there are large 
> number of files in the directory. A new API is added in Hadoop 2.6 called 
> access (HDFS-6570) which provides the ability to check if the user has 
> permissions on a file/directory.  Use this new API instead of listStatus. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5889) sqlline loses RPC connection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223215#comment-16223215
 ] 

ASF GitHub Bot commented on DRILL-5889:
---

GitHub user ppadma opened a pull request:

https://github.com/apache/drill/pull/1015

DRILL-5889: Simple pattern matchers can work with DrillBuf directly

For the 4 simple patterns we have i.e. startsWith, endsWith, contains and 
constant,, we do not need the overhead of charSequenceWrapper. We can work with 
DrillBuf directly. This will save us from doing isAscii check and UTF8 decoding 
for each row.
UTF-8 encoding ensures that no UTF-8 character is a prefix of any other 
valid character. So, instead of decoding varChar from each row we are 
processing, encode the patternString once during setup and do raw byte 
comparison. 
This improved overall performance for filter operator by around 20%.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ppadma/drill DRILL-5899

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1015.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1015


commit ebe2c6f9110f0501a85cb5ae6e119e05254b9f3e
Author: Padma Penumarthy 
Date:   2017-10-25T20:37:25Z

DRILL-5889: Simple pattern matchers can work with DrillBuf directly




> sqlline loses RPC connection
> 
>
> Key: DRILL-5889
> URL: https://issues.apache.org/jira/browse/DRILL-5889
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.11.0
>Reporter: Robert Hou
>Assignee: Pritesh Maker
> Attachments: 26183ef9-44b2-ef32-adf8-cc2b5ba9f9c0.sys.drill, 
> drillbit.log
>
>
> Query is:
> {noformat}
> alter session set `planner.memory.max_query_memory_per_node` = 10737418240;
> select count(*), max(`filename`) from dfs.`/drill/testdata/hash-agg/data1` 
> group by no_nulls_col, nulls_col;
> {noformat}
> Error is:
> {noformat}
> 0: jdbc:drill:drillbit=10.10.100.190> select count(*), max(`filename`) from 
> dfs.`/drill/testdata/hash-agg/data1` group by no_nulls_col, nulls_col;
> Error: CONNECTION ERROR: Connection /10.10.100.190:45776 <--> 
> /10.10.100.190:31010 (user client) closed unexpectedly. Drillbit down?
> [Error Id: db4aea70-11e6-4e63-b0cc-13cdba0ee87a ] (state=,code=0)
> {noformat}
> From drillbit.log:
> 2017-10-18 14:04:23,044 [UserServer-1] INFO  
> o.a.drill.exec.rpc.user.UserServer - RPC connection /10.10.100.190:31010 <--> 
> /10.10.100.190:45776 (user server) timed out.  Timeout was set to 30 seconds. 
> Closing connection.
> Plan is:
> {noformat}
> | 00-00Screen
> 00-01  Project(EXPR$0=[$0], EXPR$1=[$1])
> 00-02UnionExchange
> 01-01  Project(EXPR$0=[$2], EXPR$1=[$3])
> 01-02HashAgg(group=[{0, 1}], EXPR$0=[$SUM0($2)], EXPR$1=[MAX($3)])
> 01-03  Project(no_nulls_col=[$0], nulls_col=[$1], EXPR$0=[$2], 
> EXPR$1=[$3])
> 01-04HashToRandomExchange(dist0=[[$0]], dist1=[[$1]])
> 02-01  UnorderedMuxExchange
> 03-01Project(no_nulls_col=[$0], nulls_col=[$1], 
> EXPR$0=[$2], EXPR$1=[$3], E_X_P_R_H_A_S_H_F_I_E_L_D=[hash32AsDouble($1, 
> hash32AsDouble($0, 1301011))])
> 03-02  HashAgg(group=[{0, 1}], EXPR$0=[COUNT()], 
> EXPR$1=[MAX($2)])
> 03-03Scan(groupscan=[ParquetGroupScan 
> [entries=[ReadEntryWithPath [path=maprfs:///drill/testdata/hash-agg/data1]], 
> selectionRoot=maprfs:/drill/testdata/hash-agg/data1, numFiles=1, 
> usedMetadataFile=false, columns=[`no_nulls_col`, `nulls_col`, `filename`]]])
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5899) Simple pattern matchers can work with DrillBuf directly

[Padma Penumarthy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Padma Penumarthy updated DRILL-5899:

Description: 
For the 4 simple patterns we have i.e. startsWith, endsWith, contains and 
constant,, we do not need the overhead of charSequenceWrapper. We can work with 
DrillBuf directly. This will save us from doing isAscii check and UTF8 decoding 
for each row.
UTF-8 encoding ensures that no UTF-8 character is a prefix of any other valid 
character. So, instead of decoding varChar from each row we are processing, 
encode the patternString once during setup and do raw byte comparison. Instead 
of bounds checking and reading one byte at a time, we get the whole buffer in 
one shot and use that for comparison.
This improved overall performance for filter operator by around 20%. 


  was:
For simple pattern matcher, we do not have to do isAscii check. 
UTF-8 encoding ensures that no UTF-8 character is a prefix of any other valid 
character. So, for the 4 simple patterns we have i.e. startsWith, endsWith, 
contains and constant, we can get rid of this check. This will help improve 
performance. 



> Simple pattern matchers can work with DrillBuf directly
> ---
>
> Key: DRILL-5899
> URL: https://issues.apache.org/jira/browse/DRILL-5899
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Padma Penumarthy
>Assignee: Padma Penumarthy
>Priority: Critical
>
> For the 4 simple patterns we have i.e. startsWith, endsWith, contains and 
> constant,, we do not need the overhead of charSequenceWrapper. We can work 
> with DrillBuf directly. This will save us from doing isAscii check and UTF8 
> decoding for each row.
> UTF-8 encoding ensures that no UTF-8 character is a prefix of any other valid 
> character. So, instead of decoding varChar from each row we are processing, 
> encode the patternString once during setup and do raw byte comparison. 
> Instead of bounds checking and reading one byte at a time, we get the whole 
> buffer in one shot and use that for comparison.
> This improved overall performance for filter operator by around 20%. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5899) Simple pattern matchers can work with DrillBuf directly

[Padma Penumarthy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Padma Penumarthy updated DRILL-5899:

Summary: Simple pattern matchers can work with DrillBuf directly  (was: 
Simple pattern matchers can work with DrillBuf directly instead of CharSequence 
Wrapper)

> Simple pattern matchers can work with DrillBuf directly
> ---
>
> Key: DRILL-5899
> URL: https://issues.apache.org/jira/browse/DRILL-5899
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Padma Penumarthy
>Assignee: Padma Penumarthy
>Priority: Critical
>
> For simple pattern matcher, we do not have to do isAscii check. 
> UTF-8 encoding ensures that no UTF-8 character is a prefix of any other valid 
> character. So, for the 4 simple patterns we have i.e. startsWith, endsWith, 
> contains and constant, we can get rid of this check. This will help improve 
> performance. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5899) Simple pattern matchers can work with DrillBuf directly instead of CharSequence Wrapper

[Padma Penumarthy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Padma Penumarthy updated DRILL-5899:

Summary: Simple pattern matchers can work with DrillBuf directly instead of 
CharSequence Wrapper  (was: No need to do isAscii check for simple pattern 
matcher)

> Simple pattern matchers can work with DrillBuf directly instead of 
> CharSequence Wrapper
> ---
>
> Key: DRILL-5899
> URL: https://issues.apache.org/jira/browse/DRILL-5899
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Padma Penumarthy
>Assignee: Padma Penumarthy
>Priority: Critical
>
> For simple pattern matcher, we do not have to do isAscii check. 
> UTF-8 encoding ensures that no UTF-8 character is a prefix of any other valid 
> character. So, for the 4 simple patterns we have i.e. startsWith, endsWith, 
> contains and constant, we can get rid of this check. This will help improve 
> performance. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5910) ClassNotFoundException message enhancement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223114#comment-16223114
 ] 

ASF GitHub Bot commented on DRILL-5910:
---

Github user sohami commented on the issue:

https://github.com/apache/drill/pull/1013
  
I tried with the repro step and based on that doesn't see 
`DrillRuntimeException` message in stack trace but since the PR is adding the 
factory name in same exception message, we will not see that in stack trace as 
well.


> ClassNotFoundException message enhancement 
> ---
>
> Key: DRILL-5910
> URL: https://issues.apache.org/jira/browse/DRILL-5910
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> We need to add factory name in exception message  when ClassNotFoundException 
> is caught and DrillRuntimeException is than re-thrown in 
> ClientAuthenticatorProvider constructor.
> Steps to repoduce:
> 1) Configure plain authentication
> 2) Add 
> -Ddrill.customAuthFactories=org.apache.drill.exec.rpc.security.maprsasl.MapRSaslFactory
>  to SQLLINE_JAVA_OPTS or another class that is not present in classpath.
> 3) Run sqlline and connect to drillbit



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5910) ClassNotFoundException message enhancement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223111#comment-16223111
 ] 

ASF GitHub Bot commented on DRILL-5910:
---

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/1013#discussion_r147541442
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/security/ClientAuthenticatorProvider.java
 ---
@@ -57,17 +57,17 @@ private ClientAuthenticatorProvider() {
 
 // then, custom factories
 if (customFactories != null) {
-  try {
-final String[] factories = customFactories.split(",");
-for (final String factory : factories) {
+  final String[] factories = customFactories.split(",");
+  for (final String factory : factories) {
+try {
   final Class clazz = Class.forName(factory);
   if (AuthenticatorFactory.class.isAssignableFrom(clazz)) {
 final AuthenticatorFactory instance = (AuthenticatorFactory) 
clazz.newInstance();
 authFactories.put(instance.getSimpleName(), instance);
   }
+} catch (final ClassNotFoundException | IllegalAccessException | 
InstantiationException e) {
+  throw new DrillRuntimeException(String.format("Failed to create 
auth factory '%s'", factory), e);
--- End diff --

@vrozov  +1 - It makes sense to log the error and continue rather than 
failing because a custom factory cannot be instantiated. This is because client 
environment might be set to configure these custom factories but then for 
authentication it might be using other mechanism like Plain/Kerberos in which 
case connection should succeed.


> ClassNotFoundException message enhancement 
> ---
>
> Key: DRILL-5910
> URL: https://issues.apache.org/jira/browse/DRILL-5910
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> We need to add factory name in exception message  when ClassNotFoundException 
> is caught and DrillRuntimeException is than re-thrown in 
> ClientAuthenticatorProvider constructor.
> Steps to repoduce:
> 1) Configure plain authentication
> 2) Add 
> -Ddrill.customAuthFactories=org.apache.drill.exec.rpc.security.maprsasl.MapRSaslFactory
>  to SQLLINE_JAVA_OPTS or another class that is not present in classpath.
> 3) Run sqlline and connect to drillbit



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5910) ClassNotFoundException message enhancement

[Sorabh Hamirwasia (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223096#comment-16223096
 ] 

Sorabh Hamirwasia commented on DRILL-5910:
--

I tried this and am seeing below exception. In the stack trace I don't see a 
message for the DrillRuntimeException and this PR is adding factory info only 
in the message of DrillRuntimeException which will again get lost. If it's not 
true can you please share the trace after your fix ?


{code:java}
[mapr@qa-node163 bin]$ ./sqlline -u 
"jdbc:drill:drillbit=10.10.100.163;auth=plain"
java.lang.ExceptionInInitializerError
apache drill 1.12.0-SNAPSHOT
"a drill in the hand is better than two in the bush"
0: jdbc:drill:drillbit=10.10.100.163> !quit
java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.drill.exec.rpc.security.ClientAuthenticatorProvider$Holder
at 
org.apache.drill.exec.rpc.security.ClientAuthenticatorProvider.getInstance(ClientAuthenticatorProvider.java:45)
at 
org.apache.drill.exec.rpc.user.UserClient.getAuthenticatorFactory(UserClient.java:284)
at 
org.apache.drill.exec.rpc.user.UserClient.authenticate(UserClient.java:240)
at 
org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:171)
at 
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432)
at 
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:379)
at 
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:158)
at 
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
at 
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at 
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.close(Commands.java:925)
at sqlline.Commands.quit(Commands.java:889)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.begin(SqlLine.java:621)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize 
class org.apache.drill.exec.rpc.security.ClientAuthenticatorProvider$Holder
at 
org.apache.drill.exec.rpc.security.ClientAuthenticatorProvider.getInstance(ClientAuthenticatorProvider.java:45)
at 
org.apache.drill.exec.rpc.user.UserClient.getAuthenticatorFactory(UserClient.java:284)
at 
org.apache.drill.exec.rpc.user.UserClient.authenticate(UserClient.java:240)
at 
org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:171)
at 
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432)
at 
org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:379)
at 
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:158)
at 
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
at 
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at 
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
at sqlline.Commands.close(Commands.java:925)
at sqlline.Commands.closeall(Commands.java:899)
at sqlline.SqlLine.begin(SqlLine.java:649)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
{code}


> ClassNotFoundException message enhancement 
> ---
>
> Key: DRILL-5910
> URL: https://issues.apache.org/jira/browse/DRILL-5910
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> We need to add factory name in exception message  when ClassNotFoundException 
> is caught and DrillRuntimeException is than 

[jira] [Assigned] (DRILL-4708) connection closed unexpectedly

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker reassigned DRILL-4708:


Assignee: Karthikeyan Manivannan  (was: Vlad Rozov)

> connection closed unexpectedly
> --
>
> Key: DRILL-4708
> URL: https://issues.apache.org/jira/browse/DRILL-4708
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - RPC
>Affects Versions: 1.7.0
>Reporter: Chun Chang
>Assignee: Karthikeyan Manivannan
>Priority: Critical
> Attachments: data.tgz
>
>
> Running DRILL functional automation, we often see query failed randomly due 
> to the following unexpected connection close error.
> {noformat}
> Execution Failures:
> /root/drillAutomation/framework/framework/resources/Functional/ctas/ctas_flatten/10rows/filter5.q
> Query: 
> select * from dfs.ctas_flatten.`filter5_10rows_ctas`
> Failed with exception
> java.sql.SQLException: CONNECTION ERROR: Connection /10.10.100.171:36185 <--> 
> drillats4.qa.lab/10.10.100.174:31010 (user client) closed unexpectedly. 
> Drillbit down?
> [Error Id: 3d5dad8e-80d0-4c7f-9012-013bf01ce2b7 ]
>   at 
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:247)
>   at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
>   at 
> oadd.net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
>   at 
> org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
>   at 
> org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:210)
>   at 
> org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:99)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)
> Caused by: oadd.org.apache.drill.common.exceptions.UserException: CONNECTION 
> ERROR: Connection /10.10.100.171:36185 <--> 
> drillats4.qa.lab/10.10.100.174:31010 (user client) closed unexpectedly. 
> Drillbit down?
> [Error Id: 3d5dad8e-80d0-4c7f-9012-013bf01ce2b7 ]
>   at 
> oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>   at 
> oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$ChannelClosedHandler$1.operationComplete(QueryResultHandler.java:373)
>   at 
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
>   at 
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
>   at 
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
>   at 
> oadd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
>   at 
> oadd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
>   at 
> oadd.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
>   at 
> oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
>   at 
> oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
>   at 
> oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
>   at 
> oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:89)
>   at 
> oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162)
>   at 
> oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
>   at 
> oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>   at 
> oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>   at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>   at 
> oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>   ... 1 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-4286) Have an ability to put server in quiescent mode of operation

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-4286:
-
Reviewer: Karthikeyan Manivannan

> Have an ability to put server in quiescent mode of operation
> 
>
> Key: DRILL-4286
> URL: https://issues.apache.org/jira/browse/DRILL-4286
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Execution - Flow
>Reporter: Victoria Markman
>Assignee: Venkata Jyothsna Donapati
>
> I think drill will benefit from mode of operation that is called "quiescent" 
> in some databases. 
> From IBM Informix server documentation:
> {code}
> Change gracefully from online to quiescent mode
> Take the database server gracefully from online mode to quiescent mode to 
> restrict access to the database server without interrupting current 
> processing. After you perform this task, the database server sets a flag that 
> prevents new sessions from gaining access to the database server. The current 
> sessions are allowed to finish processing. After you initiate the mode 
> change, it cannot be canceled. During the mode change from online to 
> quiescent, the database server is considered to be in Shutdown mode.
> {code}
> This is different from shutdown, when processes are terminated. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5842) Refactor and simplify the fragment, operator contexts for testing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223052#comment-16223052
 ] 

ASF GitHub Bot commented on DRILL-5842:
---

Github user priteshm commented on the issue:

https://github.com/apache/drill/pull/978
  
@sohami  can you please review this?


> Refactor and simplify the fragment, operator contexts for testing
> -
>
> Key: DRILL-5842
> URL: https://issues.apache.org/jira/browse/DRILL-5842
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Fix For: 1.12.0
>
>
> Drill's execution engine has a "fragment context" that provides state for a 
> fragment as a whole, and an "operator context" which provides state for a 
> single operator. Historically, these have both been concrete classes that 
> make generous references to the Drillbit context, and hence need a full Drill 
> server in order to operate.
> Drill has historically made extensive use of system-level testing: build the 
> entire server and fire queries at it to test each component. Over time, we 
> are augmenting that approach with unit tests: the ability to test each 
> operator (or parts of an operator) in isolation.
> Since each operator requires access to both the operator and fragment 
> context, the fact that the contexts depend on the overall server creates a 
> large barrier to unit testing. An earlier checkin started down the path of 
> defining the contexts as interfaces that can have different run-time and 
> test-time implementations to enable testing.
> This ticket asks to refactor those interfaces: simplifying the operator 
> context and introducing an interface for the fragment context. New code will 
> use these new interfaces, while older code continues to use the concrete 
> implementations. Over time, as operators are enhanced, they can be modified 
> to allow unit-level testing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5425) Support HTTP Kerberos auth using SPNEGO

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-5425:
-
Reviewer: Sorabh Hamirwasia

> Support HTTP Kerberos auth using SPNEGO
> ---
>
> Key: DRILL-5425
> URL: https://issues.apache.org/jira/browse/DRILL-5425
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Web Server
>Reporter: Sudheesh Katkam
>Assignee: Sindhuri Ramanarayan Rayavaram
>
> DRILL-4280 supports Kerberos through JDBC and ODBC API. This ticket requests 
> to add Kerberos (using [SPENGO|https://en.wikipedia.org/wiki/SPNEGO]) for 
> HTTP connections.
> This requires creating "direct" web sessions; currently web sessions are 
> sessions over Java client sessions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5911) Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json library

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222931#comment-16222931
 ] 

ASF GitHub Bot commented on DRILL-5911:
---

Github user julianhyde commented on the issue:

https://github.com/apache/drill/pull/1012
  
If I recall correctly -- we went through this because Calcite uses ESRI -- 
the only change was to remove org.json. It's a pretty important change because 
org.json is not compatible with Apache license. And I think org.json data 
structures were used in the API (e.g. as arguments and method return values) so 
obsoleting it would be an API change.


> Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json 
> library
> --
>
> Key: DRILL-5911
> URL: https://issues.apache.org/jira/browse/DRILL-5911
> Project: Apache Drill
>  Issue Type: Task
>Affects Versions: 1.11.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Currently, {{drill-gis}} module uses {{esri-geometry-api}} library version 
> 1.2.1. This version of the library has the dependency on {{org.json}} library:
> {noformat}
> [INFO] org.apache.drill.contrib:drill-gis:jar:1.12.0-SNAPSHOT
> [INFO] \- com.esri.geometry:esri-geometry-api:jar:1.2.1:compile
> [INFO]\- org.json:json:jar:20090211:compile
> {noformat}
> In {{esri-geometry-api}} v.2.0.0 this dependency on {{org.json}} was removed: 
> https://github.com/Esri/geometry-api-java/commit/9bedde397f2f61675bc687b95875893aa7cd7f2f.
>  
> So we need also update the version of this library to avoid transitive 
> dependency on {{org.json}} form {{drill-gis}} module.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (DRILL-5912) Hash Join Enhancement: Avoid copying probe side values

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker reassigned DRILL-5912:


Assignee: Boaz Ben-Zvi

> Hash Join Enhancement: Avoid copying probe side values
> --
>
> Key: DRILL-5912
> URL: https://issues.apache.org/jira/browse/DRILL-5912
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Affects Versions: 1.11.0
>Reporter: Boaz Ben-Zvi
>Assignee: Boaz Ben-Zvi
>Priority: Minor
>
> When the Hash Join Operator (inner, or left outer) performs the "probe and 
> project" task, it copies each probe side values to be projected. Example:
> {code}
> public void projectProbeRecord(int probeIndex, int outIndex)
> throws SchemaChangeException
> {
> {
> vv15 .copyFromSafe((probeIndex), (outIndex), vv12);
> }
> {
> vv21 .copyFromSafe((probeIndex), (outIndex), vv18);
> }
> }
> {code}
> In the case where there are no duplicate-key entries in the build side, and 
> no spilling took place, then each of the outer values is projected exactly 
> once (for left outer), or at most once (for inner join). 
> In such (common) cases, we could avoid the above copy, and just transfer the 
> value vectors as is (or add a Selection Vector 2 for the inner join, to 
> eliminate the unmatched entries).
> This can be a significant performance enhancement, as copying each set of 
> values is much more expensive than transposing vectors (e.g., perform the 
> copy 64K times, plus allocation of the vectors, and possible resizing for 
> variable sized types).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5912) Hash Join Enhancement: Avoid copying probe side values

[Boaz Ben-Zvi (JIRA)]

Boaz Ben-Zvi created DRILL-5912:
---

 Summary: Hash Join Enhancement: Avoid copying probe side values
 Key: DRILL-5912
 URL: https://issues.apache.org/jira/browse/DRILL-5912
 Project: Apache Drill
  Issue Type: Improvement
  Components: Execution - Relational Operators
Affects Versions: 1.11.0
Reporter: Boaz Ben-Zvi
Priority: Minor


When the Hash Join Operator (inner, or left outer) performs the "probe and 
project" task, it copies each probe side values to be projected. Example:
{code}
public void projectProbeRecord(int probeIndex, int outIndex)
throws SchemaChangeException
{
{
vv15 .copyFromSafe((probeIndex), (outIndex), vv12);
}
{
vv21 .copyFromSafe((probeIndex), (outIndex), vv18);
}
}
{code}

In the case where there are no duplicate-key entries in the build side, and no 
spilling took place, then each of the outer values is projected exactly once 
(for left outer), or at most once (for inner join). 

In such (common) cases, we could avoid the above copy, and just transfer the 
value vectors as is (or add a Selection Vector 2 for the inner join, to 
eliminate the unmatched entries).

This can be a significant performance enhancement, as copying each set of 
values is much more expensive than transposing vectors (e.g., perform the copy 
64K times, plus allocation of the vectors, and possible resizing for variable 
sized types).




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5365) FileNotFoundException when reading a parquet file

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5365:
---
Reviewer:   (was: Paul Rogers)

> FileNotFoundException when reading a parquet file
> -
>
> Key: DRILL-5365
> URL: https://issues.apache.org/jira/browse/DRILL-5365
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Hive
>Affects Versions: 1.10.0
>Reporter: Chun Chang
>Assignee: Chunhui Shi
>
> The parquet file is generated through the following CTAS.
> To reproduce the issue: 1) two or more nodes cluster; 2) enable 
> impersonation; 3) set "fs.default.name": "file:///" in hive storage plugin; 
> 4) restart drillbits; 5) as a regular user, on node A, drop the table/file; 
> 6) ctas from a large enough hive table as source to recreate the table/file; 
> 7) query the table from node A should work; 8) query from node B as same user 
> should reproduce the issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5772) Enable UTF-8 support in query string by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222815#comment-16222815
 ] 

ASF GitHub Bot commented on DRILL-5772:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/936#discussion_r147515235
  
--- Diff: exec/java-exec/src/test/resources/saffron.properties ---
@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to you under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This properties file is used by Apache Calcite to define allowed charset 
in string literals,
+# which is by default ISO-8859-1.
+# Current configuration allows parsing UTF-8 by default, i.e. queries that 
contain utf-8 string literal.
+# To take affect this file should be present in classpath.
+
+saffron.default.charset=UTF-16LE
+saffron.default.nationalcharset=UTF-16LE
+saffron.default.collation.name=UTF-16LE$en_US
--- End diff --

Do we need a separate test properties file if it is identical with the main 
one?


> Enable UTF-8 support in query string by default
> ---
>
> Key: DRILL-5772
> URL: https://issues.apache.org/jira/browse/DRILL-5772
> Project: Apache Drill
>  Issue Type: Task
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.12.0
>
>
> Now saffron.propertries file will be added into Drill conf directory which 
> will define default encoding used to parse query string.
> Content:
> {noformat}
> saffron.default.charset=UTF-16LE
> saffron.default.nationalcharset=UTF-16LE
> saffron.default.collation.name=UTF-16LE$en_US 
> {noformat}
> This information should be also documented, probably in 
> https://drill.apache.org/docs/data-type-conversion/.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5874) NPE in AnonWebUserConnection.cleanupSession()

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5874:
---
Labels: ready-to-commit  (was: )

> NPE in AnonWebUserConnection.cleanupSession()
> -
>
> Key: DRILL-5874
> URL: https://issues.apache.org/jira/browse/DRILL-5874
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Paul Rogers
>Assignee: Sorabh Hamirwasia
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> When debugging another issue, I tried to use the Web UI to run the example 
> query:
> {code}
> SELECT * FROM cp.`employee.json` LIMIT 20
> {code}
> The query failed with this error:
> {noformat}
> Query Failed: An Error Occurred
> java.lang.NullPointerException
> {noformat}
> No stack trace was provided in the log, even at DEBUG level.
> Debugging, the problem appears to be deep inside 
> {{AnonWebUserConnection.cleanupSession()}}:
> {code}
> package io.netty.channel;
> public class DefaultChannelPromise ...
> protected EventExecutor executor() {
> EventExecutor e = super.executor();
> if (e == null) {
> return channel().eventLoop();
> } else {
> return e;
> }
> }
> {code}
> In the above, {{channel()}} is null. the {{channel}} field is also null.
> This may indicate that some part of the Web UI was not set up correctly. This 
> is a recent change, as this code worked several days ago.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5905) Exclude jdk-tools from project dependencies

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5905:
---
Labels: ready-to-commit  (was: )

> Exclude jdk-tools from project dependencies
> ---
>
> Key: DRILL-5905
> URL: https://issues.apache.org/jira/browse/DRILL-5905
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Tools, Build & Test
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Minor
>  Labels: ready-to-commit
>
> hadoop-annotations and hbase-annotations have system scope dependency on JDK 
> tools.jar. This dependency is provided by JDK and should be excluded from the 
> project dependencies



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222779#comment-16222779
 ] 

ASF GitHub Bot commented on DRILL-4091:
---

Github user amansinha100 commented on the issue:

https://github.com/apache/drill/pull/258
  
@cgivre regarding the commit process, you can send me a note at 
amansi...@apache.org  and I can walk through it.  Although someone else 
familiar with the process could do the commit, it would be a good learning 
exercise for you :)  



> Support more functions in gis contrib module
> 
>
> Key: DRILL-4091
> URL: https://issues.apache.org/jira/browse/DRILL-4091
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>Reporter: Karol Potocki
>
> Support for commonly used gis functions in gis contrib module: relate, 
> contains, crosses, intersects, touches, difference, disjoint, buffer, union 
> etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222712#comment-16222712
 ] 

ASF GitHub Bot commented on DRILL-4091:
---

Github user k255 commented on the issue:

https://github.com/apache/drill/pull/258
  
It's good that now there's commiter which is aware of the GIS context! The 
list of functions added in this PR is as follows:
ST_Buffer, ST_Contains, ST_Crosses, ST_Difference, ST_Disjoint, 
ST_DiST_ance, ST_Envelope, ST_Equals, ST_Intersects, ST_Overlaps, ST_Relate, 
ST_Touches, ST_Transform, ST_Union, ST_UnionAggregate, ST_X, ST_Y, ST_XMin, 
ST_XMax, ST_YMin, ST_YMax
 
Regarding the documentation, I wouldn't like to duplicate it because I 
followed what is available in PostGIS (which actually uses GEOS lib, in similar 
way as drill-gis uses relevant java libs - esri, proj4j) and these are defined 
in open geospatial consortium (OGC) specs. Of course here we have just a subset 
of what PostGIS is capable of, but I think it's valuable subset.
So i.e. for ST_X function the docs are at 
http://www.postgis.net/docs/ST_X.html
Also on example usage please refer to examples contained in readme at:
https://github.com/k255/drill-gis

I'll also finally need to think about blog post/presentation on this 
extension, but most probably not in following days but later in the future.




> Support more functions in gis contrib module
> 
>
> Key: DRILL-4091
> URL: https://issues.apache.org/jira/browse/DRILL-4091
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>Reporter: Karol Potocki
>
> Support for commonly used gis functions in gis contrib module: relate, 
> contains, crosses, intersects, touches, difference, disjoint, buffer, union 
> etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5911) Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json library

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222693#comment-16222693
 ] 

ASF GitHub Bot commented on DRILL-5911:
---

Github user vvysotskyi commented on the issue:

https://github.com/apache/drill/pull/1012
  
@vrozov As I understand from the [release 
notes](https://github.com/Esri/geometry-api-java/releases/tag/v2.0.0), the main 
reason for the major version change was changing the interface due to the 
removing org.json dependency. Gis storage plugin provides UDF functions, and 
they are covered by the existing unit tests, so since all tests are passed this 
change does not break drill-gis. 
I think this change should be merged in the current release, but I don't 
know when other PR will be merged. Also, another pull request isn't affected by 
changing esri-geometry-api library, ie this change does not break changes made 
there.


> Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json 
> library
> --
>
> Key: DRILL-5911
> URL: https://issues.apache.org/jira/browse/DRILL-5911
> Project: Apache Drill
>  Issue Type: Task
>Affects Versions: 1.11.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Currently, {{drill-gis}} module uses {{esri-geometry-api}} library version 
> 1.2.1. This version of the library has the dependency on {{org.json}} library:
> {noformat}
> [INFO] org.apache.drill.contrib:drill-gis:jar:1.12.0-SNAPSHOT
> [INFO] \- com.esri.geometry:esri-geometry-api:jar:1.2.1:compile
> [INFO]\- org.json:json:jar:20090211:compile
> {noformat}
> In {{esri-geometry-api}} v.2.0.0 this dependency on {{org.json}} was removed: 
> https://github.com/Esri/geometry-api-java/commit/9bedde397f2f61675bc687b95875893aa7cd7f2f.
>  
> So we need also update the version of this library to avoid transitive 
> dependency on {{org.json}} form {{drill-gis}} module.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5910) ClassNotFoundException message enhancement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222674#comment-16222674
 ] 

ASF GitHub Bot commented on DRILL-5910:
---

Github user prasadns14 commented on a diff in the pull request:

https://github.com/apache/drill/pull/1013#discussion_r147463501
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/security/ClientAuthenticatorProvider.java
 ---
@@ -57,17 +57,17 @@ private ClientAuthenticatorProvider() {
 
 // then, custom factories
 if (customFactories != null) {
-  try {
-final String[] factories = customFactories.split(",");
-for (final String factory : factories) {
+  final String[] factories = customFactories.split(",");
+  for (final String factory : factories) {
+try {
   final Class clazz = Class.forName(factory);
   if (AuthenticatorFactory.class.isAssignableFrom(clazz)) {
 final AuthenticatorFactory instance = (AuthenticatorFactory) 
clazz.newInstance();
 authFactories.put(instance.getSimpleName(), instance);
   }
+} catch (final ClassNotFoundException | IllegalAccessException | 
InstantiationException e) {
+  throw new DrillRuntimeException(String.format("Failed to create 
auth factory '%s'", factory), e);
--- End diff --

I don't think we should continue. When a custom factory is requested, in 
this case a custom authenticator factory it doesn't make sense to continue if 
it can't be instantiated.


> ClassNotFoundException message enhancement 
> ---
>
> Key: DRILL-5910
> URL: https://issues.apache.org/jira/browse/DRILL-5910
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> We need to add factory name in exception message  when ClassNotFoundException 
> is caught and DrillRuntimeException is than re-thrown in 
> ClientAuthenticatorProvider constructor.
> Steps to repoduce:
> 1) Configure plain authentication
> 2) Add 
> -Ddrill.customAuthFactories=org.apache.drill.exec.rpc.security.maprsasl.MapRSaslFactory
>  to SQLLINE_JAVA_OPTS or another class that is not present in classpath.
> 3) Run sqlline and connect to drillbit



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5911) Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json library

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222598#comment-16222598
 ] 

ASF GitHub Bot commented on DRILL-5911:
---

Github user vrozov commented on the issue:

https://github.com/apache/drill/pull/1012
  
@vvysotskyi It is a major version change of the dependency, but except for 
the pom file changes, there are no other changes. It will be good to understand 
what triggered a major version change (do release notes cover that?) and how it 
may affect gis plugin. Also, there is another PR on the review related to gis, 
can this change be combined with the other changes?


> Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json 
> library
> --
>
> Key: DRILL-5911
> URL: https://issues.apache.org/jira/browse/DRILL-5911
> Project: Apache Drill
>  Issue Type: Task
>Affects Versions: 1.11.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Currently, {{drill-gis}} module uses {{esri-geometry-api}} library version 
> 1.2.1. This version of the library has the dependency on {{org.json}} library:
> {noformat}
> [INFO] org.apache.drill.contrib:drill-gis:jar:1.12.0-SNAPSHOT
> [INFO] \- com.esri.geometry:esri-geometry-api:jar:1.2.1:compile
> [INFO]\- org.json:json:jar:20090211:compile
> {noformat}
> In {{esri-geometry-api}} v.2.0.0 this dependency on {{org.json}} was removed: 
> https://github.com/Esri/geometry-api-java/commit/9bedde397f2f61675bc687b95875893aa7cd7f2f.
>  
> So we need also update the version of this library to avoid transitive 
> dependency on {{org.json}} form {{drill-gis}} module.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5910) ClassNotFoundException message enhancement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222569#comment-16222569
 ] 

ASF GitHub Bot commented on DRILL-5910:
---

Github user vrozov commented on a diff in the pull request:

https://github.com/apache/drill/pull/1013#discussion_r147445561
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/security/ClientAuthenticatorProvider.java
 ---
@@ -57,17 +57,17 @@ private ClientAuthenticatorProvider() {
 
 // then, custom factories
 if (customFactories != null) {
-  try {
-final String[] factories = customFactories.split(",");
-for (final String factory : factories) {
+  final String[] factories = customFactories.split(",");
+  for (final String factory : factories) {
+try {
   final Class clazz = Class.forName(factory);
   if (AuthenticatorFactory.class.isAssignableFrom(clazz)) {
 final AuthenticatorFactory instance = (AuthenticatorFactory) 
clazz.newInstance();
 authFactories.put(instance.getSimpleName(), instance);
   }
+} catch (final ClassNotFoundException | IllegalAccessException | 
InstantiationException e) {
+  throw new DrillRuntimeException(String.format("Failed to create 
auth factory '%s'", factory), e);
--- End diff --

Not directly related to this PR, but should it continue with the default 
set of factories if one of custom factories can't be instantiated?


> ClassNotFoundException message enhancement 
> ---
>
> Key: DRILL-5910
> URL: https://issues.apache.org/jira/browse/DRILL-5910
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> We need to add factory name in exception message  when ClassNotFoundException 
> is caught and DrillRuntimeException is than re-thrown in 
> ClientAuthenticatorProvider constructor.
> Steps to repoduce:
> 1) Configure plain authentication
> 2) Add 
> -Ddrill.customAuthFactories=org.apache.drill.exec.rpc.security.maprsasl.MapRSaslFactory
>  to SQLLINE_JAVA_OPTS or another class that is not present in classpath.
> 3) Run sqlline and connect to drillbit



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5910) ClassNotFoundException message enhancement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222570#comment-16222570
 ] 

ASF GitHub Bot commented on DRILL-5910:
---

Github user vrozov commented on a diff in the pull request:

https://github.com/apache/drill/pull/1013#discussion_r147443404
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/security/ClientAuthenticatorProvider.java
 ---
@@ -57,17 +57,17 @@ private ClientAuthenticatorProvider() {
 
 // then, custom factories
 if (customFactories != null) {
-  try {
-final String[] factories = customFactories.split(",");
-for (final String factory : factories) {
+  final String[] factories = customFactories.split(",");
+  for (final String factory : factories) {
+try {
   final Class clazz = Class.forName(factory);
   if (AuthenticatorFactory.class.isAssignableFrom(clazz)) {
 final AuthenticatorFactory instance = (AuthenticatorFactory) 
clazz.newInstance();
 authFactories.put(instance.getSimpleName(), instance);
   }
+} catch (final ClassNotFoundException | IllegalAccessException | 
InstantiationException e) {
--- End diff --

catch ReflectiveOperationException.


> ClassNotFoundException message enhancement 
> ---
>
> Key: DRILL-5910
> URL: https://issues.apache.org/jira/browse/DRILL-5910
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> We need to add factory name in exception message  when ClassNotFoundException 
> is caught and DrillRuntimeException is than re-thrown in 
> ClientAuthenticatorProvider constructor.
> Steps to repoduce:
> 1) Configure plain authentication
> 2) Add 
> -Ddrill.customAuthFactories=org.apache.drill.exec.rpc.security.maprsasl.MapRSaslFactory
>  to SQLLINE_JAVA_OPTS or another class that is not present in classpath.
> 3) Run sqlline and connect to drillbit



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222544#comment-16222544
 ] 

ASF GitHub Bot commented on DRILL-4091:
---

Github user cgivre commented on the issue:

https://github.com/apache/drill/pull/258
  
Hi there, 
This will be my first commit, (AH!) so any assistance would be greatly 
appreciated.  I would like to see some documentation, or at minimum a list of 
what functions are included and sentence about what they do.  I think this 
could be a great addition to Drill!

> On Oct 27, 2017, at 11:21, Aman Sinha  wrote:
> 
> Thanks @k255  . @cgivre 
 once you have reviewed and are satisfied, pls mark 
the JIRA as ready-to-commit label. Also, what do you both think about 
documentation for these GIS functions ?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub 
, or mute the 
thread 
.
> 




> Support more functions in gis contrib module
> 
>
> Key: DRILL-4091
> URL: https://issues.apache.org/jira/browse/DRILL-4091
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>Reporter: Karol Potocki
>
> Support for commonly used gis functions in gis contrib module: relate, 
> contains, crosses, intersects, touches, difference, disjoint, buffer, union 
> etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222541#comment-16222541
 ] 

ASF GitHub Bot commented on DRILL-4091:
---

Github user amansinha100 commented on the issue:

https://github.com/apache/drill/pull/258
  
Thanks @k255 .   @cgivre once you have reviewed and are satisfied, pls mark 
the JIRA as ready-to-commit label.  Also, what do you both think about 
documentation for these GIS functions ?  


> Support more functions in gis contrib module
> 
>
> Key: DRILL-4091
> URL: https://issues.apache.org/jira/browse/DRILL-4091
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>Reporter: Karol Potocki
>
> Support for commonly used gis functions in gis contrib module: relate, 
> contains, crosses, intersects, touches, difference, disjoint, buffer, union 
> etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5911) Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json library

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5911:

Labels: ready-to-commit  (was: )

> Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json 
> library
> --
>
> Key: DRILL-5911
> URL: https://issues.apache.org/jira/browse/DRILL-5911
> Project: Apache Drill
>  Issue Type: Task
>Affects Versions: 1.11.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Currently, {{drill-gis}} module uses {{esri-geometry-api}} library version 
> 1.2.1. This version of the library has the dependency on {{org.json}} library:
> {noformat}
> [INFO] org.apache.drill.contrib:drill-gis:jar:1.12.0-SNAPSHOT
> [INFO] \- com.esri.geometry:esri-geometry-api:jar:1.2.1:compile
> [INFO]\- org.json:json:jar:20090211:compile
> {noformat}
> In {{esri-geometry-api}} v.2.0.0 this dependency on {{org.json}} was removed: 
> https://github.com/Esri/geometry-api-java/commit/9bedde397f2f61675bc687b95875893aa7cd7f2f.
>  
> So we need also update the version of this library to avoid transitive 
> dependency on {{org.json}} form {{drill-gis}} module.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5911) Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json library

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5911:

Reviewer: Arina Ielchiieva

> Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json 
> library
> --
>
> Key: DRILL-5911
> URL: https://issues.apache.org/jira/browse/DRILL-5911
> Project: Apache Drill
>  Issue Type: Task
>Affects Versions: 1.11.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Currently, {{drill-gis}} module uses {{esri-geometry-api}} library version 
> 1.2.1. This version of the library has the dependency on {{org.json}} library:
> {noformat}
> [INFO] org.apache.drill.contrib:drill-gis:jar:1.12.0-SNAPSHOT
> [INFO] \- com.esri.geometry:esri-geometry-api:jar:1.2.1:compile
> [INFO]\- org.json:json:jar:20090211:compile
> {noformat}
> In {{esri-geometry-api}} v.2.0.0 this dependency on {{org.json}} was removed: 
> https://github.com/Esri/geometry-api-java/commit/9bedde397f2f61675bc687b95875893aa7cd7f2f.
>  
> So we need also update the version of this library to avoid transitive 
> dependency on {{org.json}} form {{drill-gis}} module.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5771) Fix serDe errors for format plugins

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5771:

Fix Version/s: 1.12.0

> Fix serDe errors for format plugins
> ---
>
> Key: DRILL-5771
> URL: https://issues.apache.org/jira/browse/DRILL-5771
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
> Fix For: 1.12.0
>
>
> Create unit tests to check that all storage format plugins can be 
> successfully serialized  / deserialized.
> Usually this happens when query has several major fragments. 
> One way to check serde is to generate physical plan (generated as json) and 
> then submit it back to Drill.
> One example of found errors is described in the first comment. Another 
> example is described in DRILL-5166.
> *Serde issues:*
> 1. Could not obtain format plugin during deserialization
> Format plugin is created based on format plugin configuration or its name. 
> On Drill start up we load information about available plugins (its reloaded 
> each time storage plugin is updated, can be done only by admin).
> When query is parsed, we try to get plugin from the available ones, it we can 
> not find one we try to [create 
> one|https://github.com/apache/drill/blob/3e8b01d5b0d3013e3811913f0fd6028b22c1ac3f/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java#L136-L144]
> but on other query execution stages we always assume that [plugin exists 
> based on 
> configuration|https://github.com/apache/drill/blob/3e8b01d5b0d3013e3811913f0fd6028b22c1ac3f/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java#L156-L162].
> For example, during query parsing we had to create format plugin on one node 
> based on format configuration.
> Then we have sent major fragment to the different node where we used this 
> format configuration we could not get format plugin based on it and 
> deserialization has failed.
> To fix this problem we need to create format plugin during query 
> deserialization if it's absent.
>   
> 2.  Absent hash code and equals.
> Format plugins are stored in hash map where key is format plugin config.
> Since some format plugin configs did not have overridden hash code and 
> equals, we could not find format plugin based on its configuration.
> 3. Named format plugin usage
> Named format plugins configs allow to get format plugin by its name for 
> configuration shared among all drillbits.
> They are used as alias for pre-configured format plugiins. User with admin 
> priliges can modify them at runtime.
> Named format plugins configs are used instead of sending all non-default 
> parameters of format plugin config, in this case only name is sent.
> Their usage in distributed system may cause raise conditions.
> For example, 
> 1. Query is submitted. 
> 2. Parquet format plugin is created with the following configuration 
> (autoCorrectCorruptDates=>true).
> 3. Seralized named format plugin config with name as parquet.
> 4. Major fragment is sent to the different node.
> 5. Admin has changed parquet configuration for the alias 'parquet' on all 
> nodes to autoCorrectCorruptDates=>false.
> 6. Named format is deserialized on the different node into parquet format 
> plugin with configuration (autoCorrectCorruptDates=>false).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5910) ClassNotFoundException message enhancement

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5910:

Reviewer: Arina Ielchiieva

> ClassNotFoundException message enhancement 
> ---
>
> Key: DRILL-5910
> URL: https://issues.apache.org/jira/browse/DRILL-5910
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> We need to add factory name in exception message  when ClassNotFoundException 
> is caught and DrillRuntimeException is than re-thrown in 
> ClientAuthenticatorProvider constructor.
> Steps to repoduce:
> 1) Configure plain authentication
> 2) Add 
> -Ddrill.customAuthFactories=org.apache.drill.exec.rpc.security.maprsasl.MapRSaslFactory
>  to SQLLINE_JAVA_OPTS or another class that is not present in classpath.
> 3) Run sqlline and connect to drillbit



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5911) Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json library

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222510#comment-16222510
 ] 

ASF GitHub Bot commented on DRILL-5911:
---

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/1012
  
+1, LGTM.


> Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json 
> library
> --
>
> Key: DRILL-5911
> URL: https://issues.apache.org/jira/browse/DRILL-5911
> Project: Apache Drill
>  Issue Type: Task
>Affects Versions: 1.11.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
> Fix For: 1.12.0
>
>
> Currently, {{drill-gis}} module uses {{esri-geometry-api}} library version 
> 1.2.1. This version of the library has the dependency on {{org.json}} library:
> {noformat}
> [INFO] org.apache.drill.contrib:drill-gis:jar:1.12.0-SNAPSHOT
> [INFO] \- com.esri.geometry:esri-geometry-api:jar:1.2.1:compile
> [INFO]\- org.json:json:jar:20090211:compile
> {noformat}
> In {{esri-geometry-api}} v.2.0.0 this dependency on {{org.json}} was removed: 
> https://github.com/Esri/geometry-api-java/commit/9bedde397f2f61675bc687b95875893aa7cd7f2f.
>  
> So we need also update the version of this library to avoid transitive 
> dependency on {{org.json}} form {{drill-gis}} module.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5910) ClassNotFoundException message enhancement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222507#comment-16222507
 ] 

ASF GitHub Bot commented on DRILL-5910:
---

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/1013
  
+1, LGTM.


> ClassNotFoundException message enhancement 
> ---
>
> Key: DRILL-5910
> URL: https://issues.apache.org/jira/browse/DRILL-5910
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> We need to add factory name in exception message  when ClassNotFoundException 
> is caught and DrillRuntimeException is than re-thrown in 
> ClientAuthenticatorProvider constructor.
> Steps to repoduce:
> 1) Configure plain authentication
> 2) Add 
> -Ddrill.customAuthFactories=org.apache.drill.exec.rpc.security.maprsasl.MapRSaslFactory
>  to SQLLINE_JAVA_OPTS or another class that is not present in classpath.
> 3) Run sqlline and connect to drillbit



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5910) ClassNotFoundException message enhancement

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5910:

Labels: ready-to-commit  (was: )

> ClassNotFoundException message enhancement 
> ---
>
> Key: DRILL-5910
> URL: https://issues.apache.org/jira/browse/DRILL-5910
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> We need to add factory name in exception message  when ClassNotFoundException 
> is caught and DrillRuntimeException is than re-thrown in 
> ClientAuthenticatorProvider constructor.
> Steps to repoduce:
> 1) Configure plain authentication
> 2) Add 
> -Ddrill.customAuthFactories=org.apache.drill.exec.rpc.security.maprsasl.MapRSaslFactory
>  to SQLLINE_JAVA_OPTS or another class that is not present in classpath.
> 3) Run sqlline and connect to drillbit



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5771) Fix serDe errors for format plugins

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222504#comment-16222504
 ] 

ASF GitHub Bot commented on DRILL-5771:
---

GitHub user arina-ielchiieva opened a pull request:

https://github.com/apache/drill/pull/1014

DRILL-5771: Fix serDe errors for format plugins

1. Fix various serde issues for format plugins described in 
[DRILL-5771](https://issues.apache.org/jira/browse/DRILL-5771).
2. Throw meaninful exception instead of NPE when table is not found when 
table function is used.
3. Add unit tests for all format plugins for ensure serde is checked 
(physical plan is generated in json format and then submitted).
4. Fix physical plan submission on Windows 
([DRILL-4640](https://issues.apache.org/jira/browse/DRILL-4640)).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arina-ielchiieva/drill DRILL-5771

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1014.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1014


commit 72544d5aa9f0add3075c2eb174c49b355d3c85e2
Author: Arina Ielchiieva 
Date:   2017-10-26T13:09:00Z

DRILL-5771: Fix serDe errors for format plugins

1. Fix various serde issues for format plugins described in DRILL-5771.
2. Throw meaninful exception instead of NPE when table is not found when 
table function is used.
3. Added unit tests for all format plugins for ensure serde is checked 
(physical plan is generated in json format and then submitted).
4. Fix physical plan submission on Windows (DRILL-4640).




> Fix serDe errors for format plugins
> ---
>
> Key: DRILL-5771
> URL: https://issues.apache.org/jira/browse/DRILL-5771
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
>
> Create unit tests to check that all storage format plugins can be 
> successfully serialized  / deserialized.
> Usually this happens when query has several major fragments. 
> One way to check serde is to generate physical plan (generated as json) and 
> then submit it back to Drill.
> One example of found errors is described in the first comment. Another 
> example is described in DRILL-5166.
> *Serde issues:*
> 1. Could not obtain format plugin during deserialization
> Format plugin is created based on format plugin configuration or its name. 
> On Drill start up we load information about available plugins (its reloaded 
> each time storage plugin is updated, can be done only by admin).
> When query is parsed, we try to get plugin from the available ones, it we can 
> not find one we try to [create 
> one|https://github.com/apache/drill/blob/3e8b01d5b0d3013e3811913f0fd6028b22c1ac3f/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java#L136-L144]
> but on other query execution stages we always assume that [plugin exists 
> based on 
> configuration|https://github.com/apache/drill/blob/3e8b01d5b0d3013e3811913f0fd6028b22c1ac3f/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java#L156-L162].
> For example, during query parsing we had to create format plugin on one node 
> based on format configuration.
> Then we have sent major fragment to the different node where we used this 
> format configuration we could not get format plugin based on it and 
> deserialization has failed.
> To fix this problem we need to create format plugin during query 
> deserialization if it's absent.
>   
> 2.  Absent hash code and equals.
> Format plugins are stored in hash map where key is format plugin config.
> Since some format plugin configs did not have overridden hash code and 
> equals, we could not find format plugin based on its configuration.
> 3. Named format plugin usage
> Named format plugins configs allow to get format plugin by its name for 
> configuration shared among all drillbits.
> They are used as alias for pre-configured format plugiins. User with admin 
> priliges can modify them at runtime.
> Named format plugins configs are used instead of sending all non-default 
> parameters of format plugin config, in this case only name is sent.
> Their usage in distributed system may cause raise conditions.
> For example, 
> 1. Query is submitted. 
> 2. Parquet format plugin is created with the following configuration 
> (autoCorrectCorruptDates=>true).
> 3. Seralized named format plugin config with name as parquet.
> 4. Major fragment is sent to the different node.
> 5. Admin has changed parquet configuration for the alias 'parquet' on all 
> 

[jira] [Updated] (DRILL-5771) Fix serDe errors for format plugins

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5771:

Issue Type: Bug  (was: Task)

> Fix serDe errors for format plugins
> ---
>
> Key: DRILL-5771
> URL: https://issues.apache.org/jira/browse/DRILL-5771
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
>
> Create unit tests to check that all storage format plugins can be 
> successfully serialized  / deserialized.
> Usually this happens when query has several major fragments. 
> One way to check serde is to generate physical plan (generated as json) and 
> then submit it back to Drill.
> One example of found errors is described in the first comment. Another 
> example is described in DRILL-5166.
> *Serde issues:*
> 1. Could not obtain format plugin during deserialization
> Format plugin is created based on format plugin configuration or its name. 
> On Drill start up we load information about available plugins (its reloaded 
> each time storage plugin is updated, can be done only by admin).
> When query is parsed, we try to get plugin from the available ones, it we can 
> not find one we try to [create 
> one|https://github.com/apache/drill/blob/3e8b01d5b0d3013e3811913f0fd6028b22c1ac3f/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java#L136-L144]
> but on other query execution stages we always assume that [plugin exists 
> based on 
> configuration|https://github.com/apache/drill/blob/3e8b01d5b0d3013e3811913f0fd6028b22c1ac3f/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java#L156-L162].
> For example, during query parsing we had to create format plugin on one node 
> based on format configuration.
> Then we have sent major fragment to the different node where we used this 
> format configuration we could not get format plugin based on it and 
> deserialization has failed.
> To fix this problem we need to create format plugin during query 
> deserialization if it's absent.
>   
> 2.  Absent hash code and equals.
> Format plugins are stored in hash map where key is format plugin config.
> Since some format plugin configs did not have overridden hash code and 
> equals, we could not find format plugin based on its configuration.
> 3. Named format plugin usage
> Named format plugins configs allow to get format plugin by its name for 
> configuration shared among all drillbits.
> They are used as alias for pre-configured format plugiins. User with admin 
> priliges can modify them at runtime.
> Named format plugins configs are used instead of sending all non-default 
> parameters of format plugin config, in this case only name is sent.
> Their usage in distributed system may cause raise conditions.
> For example, 
> 1. Query is submitted. 
> 2. Parquet format plugin is created with the following configuration 
> (autoCorrectCorruptDates=>true).
> 3. Seralized named format plugin config with name as parquet.
> 4. Major fragment is sent to the different node.
> 5. Admin has changed parquet configuration for the alias 'parquet' on all 
> nodes to autoCorrectCorruptDates=>false.
> 6. Named format is deserialized on the different node into parquet format 
> plugin with configuration (autoCorrectCorruptDates=>false).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5771) Fix serDe errors for format plugins

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5771:

Description: 
Create unit tests to check that all storage format plugins can be successfully 
serialized  / deserialized.
Usually this happens when query has several major fragments. 

One way to check serde is to generate physical plan (generated as json) and 
then submit it back to Drill.

One example of found errors is described in the first comment. Another example 
is described in DRILL-5166.

*Serde issues:*

1. Could not obtain format plugin during deserialization
Format plugin is created based on format plugin configuration or its name. 
On Drill start up we load information about available plugins (its reloaded 
each time storage plugin is updated, can be done only by admin).
When query is parsed, we try to get plugin from the available ones, it we can 
not find one we try to [create 
one|https://github.com/apache/drill/blob/3e8b01d5b0d3013e3811913f0fd6028b22c1ac3f/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java#L136-L144]
but on other query execution stages we always assume that [plugin exists based 
on 
configuration|https://github.com/apache/drill/blob/3e8b01d5b0d3013e3811913f0fd6028b22c1ac3f/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java#L156-L162].

For example, during query parsing we had to create format plugin on one node 
based on format configuration.
Then we have sent major fragment to the different node where we used this 
format configuration we could not get format plugin based on it and 
deserialization has failed.
To fix this problem we need to create format plugin during query 
deserialization if it's absent.
  
2.  Absent hash code and equals.
Format plugins are stored in hash map where key is format plugin config.
Since some format plugin configs did not have overridden hash code and equals, 
we could not find format plugin based on its configuration.

3. Named format plugin usage
Named format plugins configs allow to get format plugin by its name for 
configuration shared among all drillbits.
They are used as alias for pre-configured format plugiins. User with admin 
priliges can modify them at runtime.
Named format plugins configs are used instead of sending all non-default 
parameters of format plugin config, in this case only name is sent.
Their usage in distributed system may cause raise conditions.
For example, 
1. Query is submitted. 
2. Parquet format plugin is created with the following configuration 
(autoCorrectCorruptDates=>true).
3. Seralized named format plugin config with name as parquet.
4. Major fragment is sent to the different node.
5. Admin has changed parquet configuration for the alias 'parquet' on all nodes 
to autoCorrectCorruptDates=>false.
6. Named format is deserialized on the different node into parquet format 
plugin with configuration (autoCorrectCorruptDates=>false).


  was:
Create unit tests to check that all storage format plugins can be successfully 
serialized  / deserialized.
Usually this happens when query has several major fragments. 

One way to check serde is to generate physical plan (generated as json) and 
then submit it back to Drill.

One example of found errors is described in the first comment. Another example 
is described in DRILL-5166.


> Fix serDe errors for format plugins
> ---
>
> Key: DRILL-5771
> URL: https://issues.apache.org/jira/browse/DRILL-5771
> Project: Apache Drill
>  Issue Type: Task
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
>
> Create unit tests to check that all storage format plugins can be 
> successfully serialized  / deserialized.
> Usually this happens when query has several major fragments. 
> One way to check serde is to generate physical plan (generated as json) and 
> then submit it back to Drill.
> One example of found errors is described in the first comment. Another 
> example is described in DRILL-5166.
> *Serde issues:*
> 1. Could not obtain format plugin during deserialization
> Format plugin is created based on format plugin configuration or its name. 
> On Drill start up we load information about available plugins (its reloaded 
> each time storage plugin is updated, can be done only by admin).
> When query is parsed, we try to get plugin from the available ones, it we can 
> not find one we try to [create 
> one|https://github.com/apache/drill/blob/3e8b01d5b0d3013e3811913f0fd6028b22c1ac3f/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java#L136-L144]
> but on other query execution stages we always assume that [plugin exists 
> based on 
> 

[jira] [Updated] (DRILL-5771) Fix serDe errors for format plugins

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5771:

Affects Version/s: 1.11.0

> Fix serDe errors for format plugins
> ---
>
> Key: DRILL-5771
> URL: https://issues.apache.org/jira/browse/DRILL-5771
> Project: Apache Drill
>  Issue Type: Task
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
>
> Create unit tests to check that all storage format plugins can be 
> successfully serialized  / deserialized.
> Usually this happens when query has several major fragments. 
> One way to check serde is to generate physical plan (generated as json) and 
> then submit it back to Drill.
> One example of found errors is described in the first comment. Another 
> example is described in DRILL-5166.
> *Serde issues:*
> 1. Could not obtain format plugin during deserialization
> Format plugin is created based on format plugin configuration or its name. 
> On Drill start up we load information about available plugins (its reloaded 
> each time storage plugin is updated, can be done only by admin).
> When query is parsed, we try to get plugin from the available ones, it we can 
> not find one we try to [create 
> one|https://github.com/apache/drill/blob/3e8b01d5b0d3013e3811913f0fd6028b22c1ac3f/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java#L136-L144]
> but on other query execution stages we always assume that [plugin exists 
> based on 
> configuration|https://github.com/apache/drill/blob/3e8b01d5b0d3013e3811913f0fd6028b22c1ac3f/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java#L156-L162].
> For example, during query parsing we had to create format plugin on one node 
> based on format configuration.
> Then we have sent major fragment to the different node where we used this 
> format configuration we could not get format plugin based on it and 
> deserialization has failed.
> To fix this problem we need to create format plugin during query 
> deserialization if it's absent.
>   
> 2.  Absent hash code and equals.
> Format plugins are stored in hash map where key is format plugin config.
> Since some format plugin configs did not have overridden hash code and 
> equals, we could not find format plugin based on its configuration.
> 3. Named format plugin usage
> Named format plugins configs allow to get format plugin by its name for 
> configuration shared among all drillbits.
> They are used as alias for pre-configured format plugiins. User with admin 
> priliges can modify them at runtime.
> Named format plugins configs are used instead of sending all non-default 
> parameters of format plugin config, in this case only name is sent.
> Their usage in distributed system may cause raise conditions.
> For example, 
> 1. Query is submitted. 
> 2. Parquet format plugin is created with the following configuration 
> (autoCorrectCorruptDates=>true).
> 3. Seralized named format plugin config with name as parquet.
> 4. Major fragment is sent to the different node.
> 5. Admin has changed parquet configuration for the alias 'parquet' on all 
> nodes to autoCorrectCorruptDates=>false.
> 6. Named format is deserialized on the different node into parquet format 
> plugin with configuration (autoCorrectCorruptDates=>false).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5910) ClassNotFoundException message enhancement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222407#comment-16222407
 ] 

ASF GitHub Bot commented on DRILL-5910:
---

GitHub user vladimirtkach opened a pull request:

https://github.com/apache/drill/pull/1013

DRILL-5910 Add factory name to ClassNotFoundException message in 
ClientAuthenticatorProvider constructor

Added factory name to exception message.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vladimirtkach/drill DRILL-5910

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1013.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1013


commit 10516783aa7db5edfd1c5f149477af4b5a1213cb
Author: vladimir 
Date:   2017-10-27T08:32:32Z

DRILL-5910 Add factory name to ClassNotFoundException message in 
ClientAuthenticatorProvider constructor




> ClassNotFoundException message enhancement 
> ---
>
> Key: DRILL-5910
> URL: https://issues.apache.org/jira/browse/DRILL-5910
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Minor
> Fix For: 1.12.0
>
>
> We need to add factory name in exception message  when ClassNotFoundException 
> is caught and DrillRuntimeException is than re-thrown in 
> ClientAuthenticatorProvider constructor.
> Steps to repoduce:
> 1) Configure plain authentication
> 2) Add 
> -Ddrill.customAuthFactories=org.apache.drill.exec.rpc.security.maprsasl.MapRSaslFactory
>  to SQLLINE_JAVA_OPTS or another class that is not present in classpath.
> 3) Run sqlline and connect to drillbit



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222338#comment-16222338
 ] 

ASF GitHub Bot commented on DRILL-4091:
---

Github user k255 commented on the issue:

https://github.com/apache/drill/pull/258
  
@amansinha100 better later than never! The PR is updated now.
@cgivre offered that he could help reviewing this.
@joeauty probably in further development we can consider adding geojson 
support. I'm happy that you like it!


> Support more functions in gis contrib module
> 
>
> Key: DRILL-4091
> URL: https://issues.apache.org/jira/browse/DRILL-4091
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>Reporter: Karol Potocki
>
> Support for commonly used gis functions in gis contrib module: relate, 
> contains, crosses, intersects, touches, difference, disjoint, buffer, union 
> etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5906) java.lang.NullPointerException while quering Hive ORC tables on MapR cluster.

[Vitalii Diravka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vitalii Diravka updated DRILL-5906:
---
Attachment: bucketed_table.zip

> java.lang.NullPointerException while quering Hive ORC tables on MapR cluster. 
> --
>
> Key: DRILL-5906
> URL: https://issues.apache.org/jira/browse/DRILL-5906
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
> Attachments: bucketed_table.zip
>
>
> Record reader throw an exception when trying to read an empty split.
> To reproduce the issue - put the bucketed table with ORC files (one or more 
> are empty) onto the maprfs and 
> run the following Hive DDL: 
> {code}
> CREATE TABLE `orc_bucketed`(
>   `id` int,
>   `name` string)
> CLUSTERED BY (
>   id)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>   'maprfs:/tmp/bucketed_table/';
> {code}
> Possible fix: upgrade drill to 
> [1.2.0-mapr-1707|https://maprdocs.mapr.com/52/EcosystemRN/HiveRN-1.2.1-1707.html]
>  hive.version, where this issue was fixed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5906) java.lang.NullPointerException while quering Hive ORC tables on MapR cluster.

[Vitalii Diravka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vitalii Diravka updated DRILL-5906:
---
Description: 
Record reader throw an exception when trying to read an empty split.
To reproduce the issue - put the bucketed table with ORC files (one or more are 
empty) onto the maprfs and 
run the following Hive DDL: 
{code}
CREATE TABLE `orc_bucketed`(
  `id` int,
  `name` string)
CLUSTERED BY (
  id)
INTO 2 BUCKETS
ROW FORMAT SERDE
  'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
  'maprfs:/tmp/bucketed_table/';
{code}

Possible fix: upgrade drill to 
[1.2.0-mapr-1707|https://maprdocs.mapr.com/52/EcosystemRN/HiveRN-1.2.1-1707.html]
 hive.version, where this issue was fixed.

  was:
Record reader throw an exception when trying to read an empty split.

Possible fix: upgrade drill to 
[1.2.0-mapr-1707|https://maprdocs.mapr.com/52/EcosystemRN/HiveRN-1.2.1-1707.html]
 hive.version, where this issue was fixed.


> java.lang.NullPointerException while quering Hive ORC tables on MapR cluster. 
> --
>
> Key: DRILL-5906
> URL: https://issues.apache.org/jira/browse/DRILL-5906
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Record reader throw an exception when trying to read an empty split.
> To reproduce the issue - put the bucketed table with ORC files (one or more 
> are empty) onto the maprfs and 
> run the following Hive DDL: 
> {code}
> CREATE TABLE `orc_bucketed`(
>   `id` int,
>   `name` string)
> CLUSTERED BY (
>   id)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>   'maprfs:/tmp/bucketed_table/';
> {code}
> Possible fix: upgrade drill to 
> [1.2.0-mapr-1707|https://maprdocs.mapr.com/52/EcosystemRN/HiveRN-1.2.1-1707.html]
>  hive.version, where this issue was fixed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5911) Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json library

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222311#comment-16222311
 ] 

ASF GitHub Bot commented on DRILL-5911:
---

GitHub user vvysotskyi opened a pull request:

https://github.com/apache/drill/pull/1012

DRILL-5911: Upgrade esri-geometry-api version to 2.0.0 to avoid depen…

…dency on org.json library

Please see [DRILL-5911](https://issues.apache.org/jira/browse/DRILL-5911) 
for details.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vvysotskyi/drill DRILL-5911

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1012.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1012


commit 87e892515bfd731a088a3e03ffdc0bef184cf29c
Author: Volodymyr Vysotskyi 
Date:   2017-10-27T14:45:17Z

DRILL-5911: Upgrade esri-geometry-api version to 2.0.0 to avoid dependency 
on org.json library




> Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json 
> library
> --
>
> Key: DRILL-5911
> URL: https://issues.apache.org/jira/browse/DRILL-5911
> Project: Apache Drill
>  Issue Type: Task
>Affects Versions: 1.11.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
> Fix For: 1.12.0
>
>
> Currently, {{drill-gis}} module uses {{esri-geometry-api}} library version 
> 1.2.1. This version of the library has the dependency on {{org.json}} library:
> {noformat}
> [INFO] org.apache.drill.contrib:drill-gis:jar:1.12.0-SNAPSHOT
> [INFO] \- com.esri.geometry:esri-geometry-api:jar:1.2.1:compile
> [INFO]\- org.json:json:jar:20090211:compile
> {noformat}
> In {{esri-geometry-api}} v.2.0.0 this dependency on {{org.json}} was removed: 
> https://github.com/Esri/geometry-api-java/commit/9bedde397f2f61675bc687b95875893aa7cd7f2f.
>  
> So we need also update the version of this library to avoid transitive 
> dependency on {{org.json}} form {{drill-gis}} module.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5911) Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json library

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5911:

Issue Type: Task  (was: Bug)

> Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json 
> library
> --
>
> Key: DRILL-5911
> URL: https://issues.apache.org/jira/browse/DRILL-5911
> Project: Apache Drill
>  Issue Type: Task
>Affects Versions: 1.11.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
> Fix For: 1.12.0
>
>
> Currently, {{drill-gis}} module uses {{esri-geometry-api}} library version 
> 1.2.1. This version of the library has the dependency on {{org.json}} library:
> {noformat}
> [INFO] org.apache.drill.contrib:drill-gis:jar:1.12.0-SNAPSHOT
> [INFO] \- com.esri.geometry:esri-geometry-api:jar:1.2.1:compile
> [INFO]\- org.json:json:jar:20090211:compile
> {noformat}
> In {{esri-geometry-api}} v.2.0.0 this dependency on {{org.json}} was removed: 
> https://github.com/Esri/geometry-api-java/commit/9bedde397f2f61675bc687b95875893aa7cd7f2f.
>  
> So we need also update the version of this library to avoid transitive 
> dependency on {{org.json}} form {{drill-gis}} module.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5911) Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json library

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5911:

Fix Version/s: 1.12.0

> Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json 
> library
> --
>
> Key: DRILL-5911
> URL: https://issues.apache.org/jira/browse/DRILL-5911
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
> Fix For: 1.12.0
>
>
> Currently, {{drill-gis}} module uses {{esri-geometry-api}} library version 
> 1.2.1. This version of the library has the dependency on {{org.json}} library:
> {noformat}
> [INFO] org.apache.drill.contrib:drill-gis:jar:1.12.0-SNAPSHOT
> [INFO] \- com.esri.geometry:esri-geometry-api:jar:1.2.1:compile
> [INFO]\- org.json:json:jar:20090211:compile
> {noformat}
> In {{esri-geometry-api}} v.2.0.0 this dependency on {{org.json}} was removed: 
> https://github.com/Esri/geometry-api-java/commit/9bedde397f2f61675bc687b95875893aa7cd7f2f.
>  
> So we need also update the version of this library to avoid transitive 
> dependency on {{org.json}} form {{drill-gis}} module.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5911) Upgrade esri-geometry-api version to 2.0.0 to avoid dependency on org.json library

[Volodymyr Vysotskyi (JIRA)]

Volodymyr Vysotskyi created DRILL-5911:
--

 Summary: Upgrade esri-geometry-api version to 2.0.0 to avoid 
dependency on org.json library
 Key: DRILL-5911
 URL: https://issues.apache.org/jira/browse/DRILL-5911
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.11.0
Reporter: Volodymyr Vysotskyi
Assignee: Volodymyr Vysotskyi


Currently, {{drill-gis}} module uses {{esri-geometry-api}} library version 
1.2.1. This version of the library has the dependency on {{org.json}} library:
{noformat}
[INFO] org.apache.drill.contrib:drill-gis:jar:1.12.0-SNAPSHOT
[INFO] \- com.esri.geometry:esri-geometry-api:jar:1.2.1:compile
[INFO]\- org.json:json:jar:20090211:compile
{noformat}

In {{esri-geometry-api}} v.2.0.0 this dependency on {{org.json}} was removed: 
https://github.com/Esri/geometry-api-java/commit/9bedde397f2f61675bc687b95875893aa7cd7f2f.
 
So we need also update the version of this library to avoid transitive 
dependency on {{org.json}} form {{drill-gis}} module.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4640) Unable to submit physical plan from Web UI on Windows

[Arina Ielchiieva (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222052#comment-16222052
 ] 

Arina Ielchiieva commented on DRILL-4640:
-

Will be fixed in the scope of DRILL-5771.

> Unable to submit physical plan from Web UI on Windows
> -
>
> Key: DRILL-4640
> URL: https://issues.apache.org/jira/browse/DRILL-4640
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Unable to submit physical plan from Web UI on Windows:
> Steps to reproduce:
> 1. Generate physical plan using query:
> explain plan for SELECT * FROM cp.`employee.json` LIMIT 20
> 2. Copy output in json column:
> {noformat}
> { "head" : { "version" : 1, "generator" : { "type" : "ExplainHandler", "info" 
> : "" }, "type" : "APACHE_DRILL_PHYSICAL", "options" : [ ], "queue" : 0, 
> "resultMode" : "EXEC" }, "graph" : [ { "pop" : "fs-scan", "@id" : 4, 
> "userName" : "User", "files" : [ "classpath:/employee.json" ], "storage" : { 
> "type" : "file", "enabled" : true, "connection" : "classpath:///", "config" : 
> null, "workspaces" : null, "formats" : { "csv" : { "type" : "text", 
> "extensions" : [ "csv" ], "delimiter" : "," }, "tsv" : { "type" : "text", 
> "extensions" : [ "tsv" ], "delimiter" : "\t" }, "json" : { "type" : "json", 
> "extensions" : [ "json" ] }, "parquet" : { "type" : "parquet" }, "avro" : { 
> "type" : "avro" }, "csvh" : { "type" : "text", "extensions" : [ "csvh" ], 
> "extractHeader" : true, "delimiter" : "," } } }, "format" : { "type" : 
> "json", "extensions" : [ "json" ] }, "columns" : [ "`*`" ], "selectionRoot" : 
> "classpath:/employee.json", "cost" : 463.0 }, { "pop" : "limit", "@id" : 3, 
> "child" : 4, "first" : 0, "last" : 20, "initialAllocation" : 100, 
> "maxAllocation" : 100, "cost" : 20.0 }, { "pop" : 
> "selection-vector-remover", "@id" : 2, "child" : 3, "initialAllocation" : 
> 100, "maxAllocation" : 100, "cost" : 20.0 }, { "pop" : "project", 
> "@id" : 1, "exprs" : [ { "ref" : "`*`", "expr" : "`*`" } ], "child" : 2, 
> "initialAllocation" : 100, "maxAllocation" : 100, "cost" : 20.0 
> }, { "pop" : "screen", "@id" : 0, "child" : 1, "initialAllocation" : 100, 
> "maxAllocation" : 100, "cost" : 20.0 } ] }
> {noformat}
> 3. Submit physical plan from Web UI
> Error:
> {noformat}
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: 
> PatternSyntaxException: Unexpected internal error near index 1 \ ^ [Error Id: 
> 4310ee06-2e84-4241-9317-553382948718 on 10.2.2.62:31010] 
> {noformat}
> Checked in Chrome and Mozilla Firefox.
> Works fine on Linux.
> The problem was in 
> [FileSelection|https://github.com/apache/drill/blob/44d5cc8eb1cd197d942a8ac52e9d42e282d30c88/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSelection.java#L40]
>  class. It used {{System.getProperty("file.separator")}} to split given path.
> But since we use {{org.apache.hadoop.fs.Path}} which stores path using its 
> default separator, not systems -> {{org.apache.hadoop.fs.Path.SEPARATOR}}.
> Default separator on Windows is \ but given path we had / (even if it came 
> from Windows)  and that's why the error occurred. To fix this problem we just 
> need to use {{org.apache.hadoop.fs.Path.SEPARATOR}} instead of 
> {{System.getProperty("file.separator")}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-4640) Unable to submit physical plan from Web UI on Windows

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-4640:

Description: 
Unable to submit physical plan from Web UI on Windows:

Steps to reproduce:
1. Generate physical plan using query:
explain plan for SELECT * FROM cp.`employee.json` LIMIT 20

2. Copy output in json column:
{noformat}
{ "head" : { "version" : 1, "generator" : { "type" : "ExplainHandler", "info" : 
"" }, "type" : "APACHE_DRILL_PHYSICAL", "options" : [ ], "queue" : 0, 
"resultMode" : "EXEC" }, "graph" : [ { "pop" : "fs-scan", "@id" : 4, "userName" 
: "User", "files" : [ "classpath:/employee.json" ], "storage" : { "type" : 
"file", "enabled" : true, "connection" : "classpath:///", "config" : null, 
"workspaces" : null, "formats" : { "csv" : { "type" : "text", "extensions" : [ 
"csv" ], "delimiter" : "," }, "tsv" : { "type" : "text", "extensions" : [ "tsv" 
], "delimiter" : "\t" }, "json" : { "type" : "json", "extensions" : [ "json" ] 
}, "parquet" : { "type" : "parquet" }, "avro" : { "type" : "avro" }, "csvh" : { 
"type" : "text", "extensions" : [ "csvh" ], "extractHeader" : true, "delimiter" 
: "," } } }, "format" : { "type" : "json", "extensions" : [ "json" ] }, 
"columns" : [ "`*`" ], "selectionRoot" : "classpath:/employee.json", "cost" : 
463.0 }, { "pop" : "limit", "@id" : 3, "child" : 4, "first" : 0, "last" : 20, 
"initialAllocation" : 100, "maxAllocation" : 100, "cost" : 20.0 }, 
{ "pop" : "selection-vector-remover", "@id" : 2, "child" : 3, 
"initialAllocation" : 100, "maxAllocation" : 100, "cost" : 20.0 }, 
{ "pop" : "project", "@id" : 1, "exprs" : [ { "ref" : "`*`", "expr" : "`*`" } 
], "child" : 2, "initialAllocation" : 100, "maxAllocation" : 100, 
"cost" : 20.0 }, { "pop" : "screen", "@id" : 0, "child" : 1, 
"initialAllocation" : 100, "maxAllocation" : 100, "cost" : 20.0 } ] 
}
{noformat}

3. Submit physical plan from Web UI

Error:
{noformat}
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: 
PatternSyntaxException: Unexpected internal error near index 1 \ ^ [Error Id: 
4310ee06-2e84-4241-9317-553382948718 on 10.2.2.62:31010] 
{noformat}

Checked in Chrome and Mozilla Firefox.
Works fine on Linux.


The problem was in 
[FileSelection|https://github.com/apache/drill/blob/44d5cc8eb1cd197d942a8ac52e9d42e282d30c88/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSelection.java#L40]
 class. It used {{System.getProperty("file.separator")}} to split given path.
But since we use {{org.apache.hadoop.fs.Path}} which stores path using its 
default separator, not systems -> {{org.apache.hadoop.fs.Path.SEPARATOR}}.
Default separator on Windows is \ but given path we had / (even if it came from 
Windows)  and that's why the error occurred. To fix this problem we just need 
to use {{org.apache.hadoop.fs.Path.SEPARATOR}} instead of 
{{System.getProperty("file.separator")}}.

  was:
Unable to submit physical plan from Web UI on Windows:

Steps to reproduce:
1. Generate physical plan using query:
explain plan for SELECT * FROM cp.`employee.json` LIMIT 20

2. Copy output in json column:
{noformat}
{ "head" : { "version" : 1, "generator" : { "type" : "ExplainHandler", "info" : 
"" }, "type" : "APACHE_DRILL_PHYSICAL", "options" : [ ], "queue" : 0, 
"resultMode" : "EXEC" }, "graph" : [ { "pop" : "fs-scan", "@id" : 4, "userName" 
: "User", "files" : [ "classpath:/employee.json" ], "storage" : { "type" : 
"file", "enabled" : true, "connection" : "classpath:///", "config" : null, 
"workspaces" : null, "formats" : { "csv" : { "type" : "text", "extensions" : [ 
"csv" ], "delimiter" : "," }, "tsv" : { "type" : "text", "extensions" : [ "tsv" 
], "delimiter" : "\t" }, "json" : { "type" : "json", "extensions" : [ "json" ] 
}, "parquet" : { "type" : "parquet" }, "avro" : { "type" : "avro" }, "csvh" : { 
"type" : "text", "extensions" : [ "csvh" ], "extractHeader" : true, "delimiter" 
: "," } } }, "format" : { "type" : "json", "extensions" : [ "json" ] }, 
"columns" : [ "`*`" ], "selectionRoot" : "classpath:/employee.json", "cost" : 
463.0 }, { "pop" : "limit", "@id" : 3, "child" : 4, "first" : 0, "last" : 20, 
"initialAllocation" : 100, "maxAllocation" : 100, "cost" : 20.0 }, 
{ "pop" : "selection-vector-remover", "@id" : 2, "child" : 3, 
"initialAllocation" : 100, "maxAllocation" : 100, "cost" : 20.0 }, 
{ "pop" : "project", "@id" : 1, "exprs" : [ { "ref" : "`*`", "expr" : "`*`" } 
], "child" : 2, "initialAllocation" : 100, "maxAllocation" : 100, 
"cost" : 20.0 }, { "pop" : "screen", "@id" : 0, "child" : 1, 
"initialAllocation" : 100, "maxAllocation" : 100, "cost" : 20.0 } ] 
}
{noformat}

3. Submit physical plan from Web UI

Error:
{noformat}
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: 
PatternSyntaxException: Unexpected internal error near index 1 \ ^ 

[jira] [Updated] (DRILL-5771) Fix serDe errors for format plugins

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5771:

Description: 
Create unit tests to check that all storage format plugins can be successfully 
serialized  / deserialized.
Usually this happens when query has several major fragments. 

One way to check serde is to generate physical plan (generated as json) and 
then submit it back to Drill.

One example of found errors is described in the first comment. Another example 
is described in DRILL-5166.

  was:
Create unit tests to check that all storage format plugins can be successfully 
serialized  / deserialized.
Usually this happens when query has several major fragments. 

One example of found errors is described in the first comment. Another example 
is described in DRILL-5166.


> Fix serDe errors for format plugins
> ---
>
> Key: DRILL-5771
> URL: https://issues.apache.org/jira/browse/DRILL-5771
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
>
> Create unit tests to check that all storage format plugins can be 
> successfully serialized  / deserialized.
> Usually this happens when query has several major fragments. 
> One way to check serde is to generate physical plan (generated as json) and 
> then submit it back to Drill.
> One example of found errors is described in the first comment. Another 
> example is described in DRILL-5166.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5771) Fix serDe errors for format plugins

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5771:

Description: 
Create unit tests to check that all storage format plugins can be successfully 
serialized  / deserialized.
Usually this happens when query has several major fragments. 

One example of found errors is described in the first comment. Another example 
is described in DRILL-5166.

  was:
Create unit tests to check that all storage format plugins can be successfully 
serialized  / deserialized. Usually this happens when query has several major 
fragments. 

One example of found errors is described in the first comment. Another example 
is describe in DRILL-5166.


> Fix serDe errors for format plugins
> ---
>
> Key: DRILL-5771
> URL: https://issues.apache.org/jira/browse/DRILL-5771
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
>
> Create unit tests to check that all storage format plugins can be 
> successfully serialized  / deserialized.
> Usually this happens when query has several major fragments. 
> One example of found errors is described in the first comment. Another 
> example is described in DRILL-5166.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5771) Fix serDe errors for format plugins

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5771:

Description: 
Create unit tests to check that all storage format plugins can be successfully 
serialized  / deserialized. Usually this happens when query has several major 
fragments. 

One example of found errors is described in the first comment. Another example 
is describe in DRILL-5166.

  was:Create unit tests to check that all storage format plugins can be 
successfully serialized  / deserialized. Usually this happens when query has 
several major fragments. To ensure that we can use {{slice_target=1}} option.


> Fix serDe errors for format plugins
> ---
>
> Key: DRILL-5771
> URL: https://issues.apache.org/jira/browse/DRILL-5771
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
>
> Create unit tests to check that all storage format plugins can be 
> successfully serialized  / deserialized. Usually this happens when query has 
> several major fragments. 
> One example of found errors is described in the first comment. Another 
> example is describe in DRILL-5166.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (DRILL-4640) Unable to submit physical plan from Web UI on Windows

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva reassigned DRILL-4640:
---

Assignee: Arina Ielchiieva

> Unable to submit physical plan from Web UI on Windows
> -
>
> Key: DRILL-4640
> URL: https://issues.apache.org/jira/browse/DRILL-4640
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Unable to submit physical plan from Web UI on Windows:
> Steps to reproduce:
> 1. Generate physical plan using query:
> explain plan for SELECT * FROM cp.`employee.json` LIMIT 20
> 2. Copy output in json column:
> {noformat}
> { "head" : { "version" : 1, "generator" : { "type" : "ExplainHandler", "info" 
> : "" }, "type" : "APACHE_DRILL_PHYSICAL", "options" : [ ], "queue" : 0, 
> "resultMode" : "EXEC" }, "graph" : [ { "pop" : "fs-scan", "@id" : 4, 
> "userName" : "User", "files" : [ "classpath:/employee.json" ], "storage" : { 
> "type" : "file", "enabled" : true, "connection" : "classpath:///", "config" : 
> null, "workspaces" : null, "formats" : { "csv" : { "type" : "text", 
> "extensions" : [ "csv" ], "delimiter" : "," }, "tsv" : { "type" : "text", 
> "extensions" : [ "tsv" ], "delimiter" : "\t" }, "json" : { "type" : "json", 
> "extensions" : [ "json" ] }, "parquet" : { "type" : "parquet" }, "avro" : { 
> "type" : "avro" }, "csvh" : { "type" : "text", "extensions" : [ "csvh" ], 
> "extractHeader" : true, "delimiter" : "," } } }, "format" : { "type" : 
> "json", "extensions" : [ "json" ] }, "columns" : [ "`*`" ], "selectionRoot" : 
> "classpath:/employee.json", "cost" : 463.0 }, { "pop" : "limit", "@id" : 3, 
> "child" : 4, "first" : 0, "last" : 20, "initialAllocation" : 100, 
> "maxAllocation" : 100, "cost" : 20.0 }, { "pop" : 
> "selection-vector-remover", "@id" : 2, "child" : 3, "initialAllocation" : 
> 100, "maxAllocation" : 100, "cost" : 20.0 }, { "pop" : "project", 
> "@id" : 1, "exprs" : [ { "ref" : "`*`", "expr" : "`*`" } ], "child" : 2, 
> "initialAllocation" : 100, "maxAllocation" : 100, "cost" : 20.0 
> }, { "pop" : "screen", "@id" : 0, "child" : 1, "initialAllocation" : 100, 
> "maxAllocation" : 100, "cost" : 20.0 } ] }
> {noformat}
> 3. Submit physical plan from Web UI
> Error:
> {noformat}
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: 
> PatternSyntaxException: Unexpected internal error near index 1 \ ^ [Error Id: 
> 4310ee06-2e84-4241-9317-553382948718 on 10.2.2.62:31010] 
> {noformat}
> Checked in Chrome and Mozilla Firefox.
> Works fine on Linux.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5771) Fix serDe errors for format plugins

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5771:

Summary: Fix serDe errors for format plugins  (was: Add unit tests to check 
that all storage plugin can be successfully serialized / deserialized)

> Fix serDe errors for format plugins
> ---
>
> Key: DRILL-5771
> URL: https://issues.apache.org/jira/browse/DRILL-5771
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
>
> Create unit tests to check that all storage format plugins can be 
> successfully serialized  / deserialized. Usually this happens when query has 
> several major fragments. To ensure that we can use {{slice_target=1}} option.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (DRILL-5771) Add unit tests to check that all storage plugin can be successfully serialized / deserialized

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva reassigned DRILL-5771:
---

Assignee: Arina Ielchiieva

> Add unit tests to check that all storage plugin can be successfully 
> serialized / deserialized
> -
>
> Key: DRILL-5771
> URL: https://issues.apache.org/jira/browse/DRILL-5771
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
>
> Create unit tests to check that all storage format plugins can be 
> successfully serialized  / deserialized. Usually this happens when query has 
> several major fragments. To ensure that we can use {{slice_target=1}} option.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5906) java.lang.NullPointerException while quering Hive ORC tables on MapR cluster.

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5906:

Reviewer: Arina Ielchiieva

> java.lang.NullPointerException while quering Hive ORC tables on MapR cluster. 
> --
>
> Key: DRILL-5906
> URL: https://issues.apache.org/jira/browse/DRILL-5906
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Record reader throw an exception when trying to read an empty split.
> Possible fix: upgrade drill to 
> [1.2.0-mapr-1707|https://maprdocs.mapr.com/52/EcosystemRN/HiveRN-1.2.1-1707.html]
>  hive.version, where this issue was fixed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5910) ClassNotFoundException message enhancement

[Volodymyr Tkach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Volodymyr Tkach updated DRILL-5910:
---
Fix Version/s: (was: Future)
   1.12.0

> ClassNotFoundException message enhancement 
> ---
>
> Key: DRILL-5910
> URL: https://issues.apache.org/jira/browse/DRILL-5910
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Minor
> Fix For: 1.12.0
>
>
> We need to add factory name in exception message  when ClassNotFoundException 
> is caught and DrillRuntimeException is than re-thrown in 
> ClientAuthenticatorProvider constructor.
> Steps to repoduce:
> 1) Configure plain authentication
> 2) Add 
> -Ddrill.customAuthFactories=org.apache.drill.exec.rpc.security.maprsasl.MapRSaslFactory
>  to SQLLINE_JAVA_OPTS or another class that is not present in classpath.
> 3) Run sqlline and connect to drillbit



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)