[jira] [Created] (DRILL-5171) TestSort unit test tests the external sort, not the in-memory SortBatch

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5171:
--

 Summary: TestSort unit test tests the external sort, not the 
in-memory SortBatch
 Key: DRILL-5171
 URL: https://issues.apache.org/jira/browse/DRILL-5171
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.8.0
Reporter: Paul Rogers


Drill provides two sort operators: an in-memory {{SortBatch}} and a spillable 
{{ExternalSortBatch}}. The external sort is adaptive: it sorts in memory when 
it can, and spills when necessary. Perhaps for this reason, the in-memory sort 
appears to be deprecated (but is not marked as such.)

The in-memory sort has associated test case: {{TestSort}} and 
{{TestSimpleSort}}. When run, {{TestSort}} actually uses the external sort. 
{{TestSimpleSort}} has a single test case which is disabled.

The result is that no tests exist for the in-memory sort. That operator should 
be marked as deprecated, or the test cases adjusted to actually exercise the 
in-memory sort operator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-5170) Mockito-based unit tests fail when run under Java 8

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5170:
--

 Summary: Mockito-based unit tests fail when run under Java 8
 Key: DRILL-5170
 URL: https://issues.apache.org/jira/browse/DRILL-5170
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.8.0
Reporter: Paul Rogers
Priority: Minor


Drill makes extensive use of the Mockito mocking library. Drill uses a very old 
version of Mockito. Mockito appears to perform byte-code manipulations that are 
incompatible with Java 8. For example:

{code}
cd exec/java-exec/
mvn surefire:test -Dtest=TestSimpleProjection
Running org.apache.drill.exec.physical.impl.project.TestSimpleProjection#project
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 6.794 sec <<< 
FAILURE! - in org.apache.drill.exec.physical.impl.project.TestSimpleProjection
project(org.apache.drill.exec.physical.impl.project.TestSimpleProjection)  Time 
elapsed: 0.215 sec  <<< ERROR!
java.lang.VerifyError: null
at sun.instrument.InstrumentationImpl.redefineClasses0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.lang.reflect.Method.invoke(Method.java:498)
{code}

A similar error occurs when running the same unit test under Eclipse.

Fixing this issue is non-trivial. Mockito is subject to rapid change. The 
semantics of mocks has changed significantly. Code that works fine in our old 
version requires extensive rework to work with the current version.

For now, the only workaround is to compile Drill under the (obsolete) Java 7 
and use that version to run unit tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-5169) Reconsider use of AutoCloseable within Drill

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5169:
--

 Summary: Reconsider use of AutoCloseable within Drill
 Key: DRILL-5169
 URL: https://issues.apache.org/jira/browse/DRILL-5169
 Project: Apache Drill
  Issue Type: Improvement
Affects Versions: 1.8.0
Reporter: Paul Rogers
Priority: Minor


Drill has many resources that must be closed: value vectors, threads, operators 
and on and on. The {{close()}} method may sometimes throw an exception or take 
a long time. Drill has developed, or borrowed from Guava, many utilities to 
help manage the close operation.

Java has two forms of "predefined" closeable interfaces: {{Closeable}} and 
{{AutoCloseable}}. {{Closeable}} is for I/O resources and thus can throw an 
{{IOException}}. {{AutoCloseable}} throws no exception, and is integrated into 
the language for use in try-with-resources blocks. Because {{AutoCloseable}} is 
intended only for this use, any creation or return of an {{AutoCloseable}} 
outside of a try-with-resources block produces compiler warnings.

Neither of the two Java interfaces fit Drill's needs. {{Closeable}} throws a 
particular exception ({{IOException}}) which Drill seldom throws, but does not 
throw exceptions that Drill does throw.

Drill has settled on {{AutoCloseable}}, but few of Drill's resources are 
limited in life to a single try-with-resources block. The result is either 
hundreds of resource warnings (which developers learn to ignore), or hundreds 
of insertions of {{@SuppressWarnings("resource")}} tags, which just clutter the 
code.

Note that there is nothing special about either of the Java-provided 
interfaces. {{Closeable}} is simply a convention to allow easy closing of IO 
resources such as streams and so on. {{AutoCloseable}} exists for the sole 
purpose of implementing try-with-resource blocks.

What we need is a Drill-specific interface that provides a common {{close()}} 
method which throws only unchecked exceptions, but is not required to be used 
in try-with-resources. Perhaps call this {{DrillCloseable}}.

Next, reimplement the various close utilities. For example: 
{{DrillCloseables.closeAll()}} would close a set of resources, suppressing 
exceptions, and throwing a single combined exception if any operations fail.

Then, convert all uses of {{AutoCloseable}} to {{DrillCloseable}}, or at least 
all that are not used in try-with-resources block. Doing so will eliminate many 
compiler warnings and or suppress warnings tags. Because Java allows classes to 
implement multiple interfaces, it is even possible for a class to implement 
both {{DrillCloseable}} and {{AutoCloseable}} in the rare instance where both 
are needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request #714: DRILL-4919: Fix select count(1) / count(*) on csv w...

[arina-ielchiieva]

GitHub user arina-ielchiieva opened a pull request:

https://github.com/apache/drill/pull/714

DRILL-4919: Fix select count(1) / count(*) on csv with header

When columns from file are not needed in result (`isSkipQuery = true`), 
empty list of columns is replaced for text files with `columns[0]`. But 
currently text files with headers do not support `columns` syntax, so for these 
type of files we need to replace empty list of columns with default named 
column.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arina-ielchiieva/drill DRILL-4919

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/714.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #714


commit c2d1e222fac64dc6c5b177c983a2e5b2d5227190
Author: Arina Ielchiieva 
Date:   2016-12-29T15:42:53Z

DRILL-4919: Fix select count(1) / count(*) on csv with header




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #713: DRILL-3562: Query fails when using flatten on JSON ...

[Serhii-Harnyk]

GitHub user Serhii-Harnyk opened a pull request:

https://github.com/apache/drill/pull/713

DRILL-3562: Query fails when using flatten on JSON data where some do…

…cuments have an empty array
1. Added set for ListWriters tracking to keep empty arrays for further 
initializing in ensureAtLeastOneField method. 
2. Added check to avoid schema generating with field type "Late" and mode 
"Optional", replaced it to "Int" type in FlattenRecordBatch class.
3. Added unit tests to cover cases querying Json with empty arrays with 
flatten.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Serhii-Harnyk/drill DRILL-3562

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/713.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #713


commit 815de0cc6f16b247b3a655007241a074e38394c7
Author: Serhii-Harnyk 
Date:   2016-12-20T16:55:41Z

DRILL-3562: Query fails when using flatten on JSON data where some 
documents have an empty array




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---