[GitHub] drill issue #866: Drill 5657: Implement size-aware result set loader

[paul-rogers]

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/866
  
This commit provides another two levels of foundation for size-aware vector 
writers in the Drill record readers.

Much of the material below appears in Javadoc throughout the code. But, it 
is gathered here for quick reference to speed the code review.

The PR is broken into commits by layer of function. May be easier to review 
each commit one-by-one rather than looking at the whole mess in one big diff.

## Column Accessors

A recent extension to Drill's set of test tools created a "row set" 
abstraction to allow us to create, and verify, record batches with very few 
lines of code. Part of this work involved creating a set of "column accessors" 
in the vector subsystem. Column readers provide a uniform API to obtain data 
from columns (vectors), while column writers provide a uniform writing 
interface.

DRILL-5211 discusses a set of changes to limit value vectors to 16 MB in 
size (to avoid memory fragmentation due to Drill's two memory allocators.) The 
column accessors have proven to be so useful that they will be the basis for 
the new, size-aware writers used by Drill's record readers.

Changes include:

* Implement fill-empties logic for vectors that do not provide it.
* Use the new size-aware methods, throwing vector overflow exceptions which 
can now occur.
* Some fiddling to handle the non-standard names of vector functions.
* Modify strings to use a default type of bytes[], but offset a String 
version for convenience.
* Add “finish batch” logic to handle values omitted at the end of a 
batch. (This is a bug in some existing record readers.)

## Result Set Loader

The second layer of this commit is the new “result set loader.” This 
abstraction is an evolution of the “Mutator” class in the scan batch, when 
used with the existing column writers (which some readers use and others do 
not.)

A result set loader loads a set of tuples (AKA records, rows) from any 
source (such as a record reader) into a set of record batches. The loader:

* Divides records into batches based on a maximum row count or a maximum 
vector size, whichever occurs first. (Later revisions may limit overall batch 
size.)
* Tracks the start and end of each batch.
* Tracks the start and end of each row.
* Provides column loaders to write each column value.
* Handles overflow when a vector becomes full, but the client still must 
finish writing the current row.

The original Mutator class divided up responsibilities:

* The Mutator handled the entire record batch
* An optional VectorContainerWriter writes each record

The result set loader follows this same general pattern.

* The result set loader handles the entire record batch (really, a series 
of batches that make up the entire result set: hence the name.)
* The TupleLoader class provides per-tuple services which mostly consists 
of access to the column loaders.
* A tuple schema defines the schema for the result set (see below.)

To hide this complexity from the client, a ResultSetLoader interface 
defines the public API. Then, a ResultSetLoaderImpl class implements the 
interface with all the gory details. Separate classes handle each column, the 
result set schema, and so on.

This class is pretty complex, with a state machine per batch and per 
column, so take your time reviewing it.

## Column Loaders

The column writers are low-level classes that interface between a consumer 
and a value vector. To create the tuple loader we need a higher-level 
abstraction: the column loader. (Not that there is no equivalent for reading 
columns at this time: generated code does the reading in its own special way 
for each operator.)

Column loaders have a number of responsibilities:

* Single class used for all data types. No more casting.
* Transparently handle vector overflow and rollover.
* Provide generic (Object-based) setters, most useful for testing.

Because this commit seeks to prove the concept; the column loader supports 
a subset of types. Adding the other types is simply a matter of copy & paste, 
and will be done once things settle down. For now, the focus is on int and 
Varchar types (though the generic version supports all types.)

To handle vector overflow, each “set” method:

* Tries to write the value into the current vector (using a column writer)
* If overflow occurs, tell the listener (the row set mutator) to create a 
new vector
* Try the write a second time using the new vector

The set of column writers must be synchronized (not in a multi-thread 
sense) on the current row position. As in the row set test utilities, a 
WriterIndex performs this task. (In fact, this is derived from the 

[GitHub] drill pull request #866: Drill 5657: Implement size-aware result set loader

[paul-rogers]

GitHub user paul-rogers opened a pull request:

https://github.com/apache/drill/pull/866

Drill 5657: Implement size-aware result set loader



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/paul-rogers/drill DRILL-5657

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/866.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #866


commit 2c95326e1108e9299d6742c0cf5c35b86605bbe6
Author: Paul Rogers 
Date:   2017-07-04T05:43:40Z

DRILL-5657: Revised column accessors

commit bb4a4ed8eadbaa339570e8adabc8709e3d393719
Author: Paul Rogers 
Date:   2017-07-04T05:45:43Z

Revisions to row set test utilities

commit bec870fa849fdc39a12a1d1d48d18f3a0c6e8371
Author: Paul Rogers 
Date:   2017-07-04T05:48:14Z

Result set loader implementation and tests

commit 0c8727f5adfa1ffd2e4be584ba7f967e72265661
Author: Paul Rogers 
Date:   2017-07-04T05:48:47Z

Collateral damage: other affected files




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5657) Extend column accessors to be size-aware

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5657:
--

 Summary: Extend column accessors to be size-aware
 Key: DRILL-5657
 URL: https://issues.apache.org/jira/browse/DRILL-5657
 Project: Apache Drill
  Issue Type: Improvement
Affects Versions: 1.11.0
Reporter: Paul Rogers
Assignee: Paul Rogers
 Fix For: 1.11.0


A recent extension to Drill's set of test tools created a "row set" abstraction 
to allow us to create, and verify, record batches with very few lines of code. 
Part of this work involved creating a set of "column accessors" in the vector 
subsystem. Column readers provide a uniform API to obtain data from columns 
(vectors), while column writers provide a uniform writing interface.

DRILL-5211 discusses a set of changes to limit value vectors to 16 MB in size 
(to avoid memory fragmentation due to Drill's two memory allocators.) The 
column accessors have proven to be so useful that they will be the basis for 
the new, size-aware writers used by Drill's record readers.

A step in that direction is to retrofit the column writers to use the 
size-aware {{setScalar()}} and {{setArray()}} methods introduced in DRILL-5517.

Since the test framework row set classes are (at present) the only consumer of 
the accessors, those classes must also be updated with the changes. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] drill pull request #861: DRILL-4722: Fix EqualityVisitor for interval day ex...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/861


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #863: DRILL-4970: Prevent changing the negative value of ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/863


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #862: DRILL-5420: ParquetAsyncPgReader goes into infinite...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/862


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #863: DRILL-4970: Prevent changing the negative value of input h...

[paul-rogers]

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/863
  
Not entirely thrilled with focusing on one bug without cleaning up the 
entire area, but that is another discussion.
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #831: DRILL-5432: Added pcap-format support

[paul-rogers]

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/831
  
Squashed commits and committed to Apache master. Congratulations!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #831: DRILL-5432: Added pcap-format support

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/831


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #851: DRILL-5518: Test framework enhancements

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/851


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #840: DRILL-5517: Size-aware set methods in value vectors

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/840


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #831: DRILL-5432: Added pcap-format support

[paul-rogers]

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/831
  
Looks like we're good to go. Will be working on pulling the PR into Apache 
master today and will post any issues that arise as I run the code through the 
detailed tests used in that process.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill issue #831: DRILL-5432: Added pcap-format support

[dmitriyHavrilovich]

Github user dmitriyHavrilovich commented on the issue:

https://github.com/apache/drill/pull/831
  
@paul-rogers , is anything else we can do for this PR ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #863: DRILL-4970: Prevent changing the negative value of ...

[vvysotskyi]

Github user vvysotskyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/863#discussion_r125332767
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestCastFunctions.java
 ---
@@ -78,4 +81,71 @@ public void testToDateForTimeStamp() throws Exception {
 .build()
 .run();
   }
+
+  @Test // DRILL-4970
+  public void testCastNegativeFloatToInt() throws Exception {
+try {
+  test("create table dfs_test.tmp.table_with_float as\n" +
+  "(select cast(-255.0 as double) as double_col,\n" +
+  "cast(-255.0 as float) as float_col\n" +
+  "from (values(1)))");
+
+  final List columnNames = Lists.newArrayList();
+  columnNames.add("float_col");
+  columnNames.add("double_col");
+
+  final List castTypes = Lists.newArrayList();
+  castTypes.add("int");
+  castTypes.add("bigInt");
+
+  final String query = "select count(*) as c from 
dfs_test.tmp.table_with_float\n" +
+"where (cast(%1$s as %2$s) >= -255 and (%1$s 
<= -5)) or (%1$s <= -256)";
+
+  for (String columnName : columnNames) {
+for (String castType : castTypes) {
+  testBuilder()
+.sqlQuery(query, columnName, castType)
+.unOrdered()
+.baselineColumns("c")
+.baselineValues(1L)
+.build()
+.run();
+}
+  }
+} finally {
+  test("drop table if exists dfs_test.tmp.table_with_float");
+}
+  }
+
+  @Test // DRILL-4970
+  public void testCastNegativeDecimalToVarChar() throws Exception {
--- End diff --

The value of input holder was changing only for types that was tested. Cast 
function makes only explicit cast of the value for most of the types, so this 
bug couldn't appear for this types.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #863: DRILL-4970: Prevent changing the negative value of ...

[vvysotskyi]

Github user vvysotskyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/863#discussion_r125331522
  
--- Diff: exec/java-exec/src/main/codegen/templates/CastFunctions.java ---
@@ -54,14 +56,14 @@ public void setup() {}
   public void eval() {
 <#if (type.from.startsWith("Float") && type.to.endsWith("Int"))>
 boolean sign = (in.value < 0);
-in.value = java.lang.Math.abs(in.value);
-${type.native} fractional = in.value % 1;
+${type.native} value = java.lang.Math.abs(in.value);
+${type.native} fractional = value % 1;
--- End diff --

Thanks for the pointing this. Drill uses `half away from zero` rounding 
mode, so method `Math.round()` wouldn't help. I rewrote current code and now it 
handles boundary cases in the same way as the `Math.round()`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #863: DRILL-4970: Prevent changing the negative value of ...

[vvysotskyi]

Github user vvysotskyi commented on a diff in the pull request:

https://github.com/apache/drill/pull/863#discussion_r125332691
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestCastFunctions.java
 ---
@@ -78,4 +81,71 @@ public void testToDateForTimeStamp() throws Exception {
 .build()
 .run();
   }
+
+  @Test // DRILL-4970
+  public void testCastNegativeFloatToInt() throws Exception {
+try {
+  test("create table dfs_test.tmp.table_with_float as\n" +
--- End diff --

The initial bug was caused by changing the input holder when negative 
values casting, so current test checks the correctness of the fix only.
Added tests to check cast functions with boundary values. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (DRILL-4511) refresh over empty folder results in error, we need a better error message

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva resolved DRILL-4511.
-
Resolution: Fixed

> refresh over empty folder results in error, we need a better error message
> --
>
> Key: DRILL-4511
> URL: https://issues.apache.org/jira/browse/DRILL-4511
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Parquet
>Affects Versions: 1.7.0
>Reporter: Khurram Faraaz
>
> refresh table metadata  results in error on sqlline prompt. We 
> need a better message on sqlline prompt, something like, "The folder is 
> empty, and there is no metadata to refresh"
> [root@centos-01 ~]# hadoop fs -ls /tmp
> ...
> drwxr-xr-x   - root root  0 2015-07-21 21:16 /tmp/complex
> ...
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> refresh table metadata complex;
> ++--+
> |   ok   |   summary|
> ++--+
> | false  | Error: null  |
> ++--+
> 1 row selected (0.131 seconds)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2016-03-15 10:50:23,747 [2918170f-84ee-a6fb-38eb-266108bd64f3:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 2918170f-84ee-a6fb-38eb-266108bd64f3: refresh table metadata complex
> 2016-03-15 10:50:23,790 [2918170f-84ee-a6fb-38eb-266108bd64f3:foreman] ERROR 
> o.a.d.e.p.s.h.RefreshMetadataHandler - Failed to update metadata for table 
> 'complex'
> java.lang.NullPointerException: null
> at 
> org.apache.drill.exec.store.dfs.FileSelection.minusDirectories(FileSelection.java:153)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create(WorkspaceSchemaFactory.java:594)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create(WorkspaceSchemaFactory.java:370)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.getNewEntry(ExpandingConcurrentMap.java:96)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.get(ExpandingConcurrentMap.java:90)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.getTable(WorkspaceSchemaFactory.java:524)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.calcite.jdbc.SimpleCalciteSchema.getTable(SimpleCalciteSchema.java:82)
>  ~[calcite-core-1.4.0-drill-r10.jar:1.4.0-drill-r10]
> at 
> org.apache.calcite.jdbc.CalciteAbstractSchema$SchemaPlusImpl.getTable(CalciteAbstractSchema.java:177)
>  ~[calcite-core-1.4.0-drill-r10.jar:1.4.0-drill-r10]
> at 
> org.apache.drill.exec.planner.sql.handlers.RefreshMetadataHandler.getPlan(RefreshMetadataHandler.java:73)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:94)
>  [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:927) 
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:251) 
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> 2016-03-15 10:50:23,801 [2918170f-84ee-a6fb-38eb-266108bd64f3:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2918170f-84ee-a6fb-38eb-266108bd64f3:0:0: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2016-03-15 10:50:23,801 [2918170f-84ee-a6fb-38eb-266108bd64f3:frag:0:0] INFO  
> o.a.d.e.w.f.FragmentStatusReporter - 
> 2918170f-84ee-a6fb-38eb-266108bd64f3:0:0: State to report: RUNNING
> 2016-03-15 10:50:23,808 [2918170f-84ee-a6fb-38eb-266108bd64f3:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2918170f-84ee-a6fb-38eb-266108bd64f3:0:0: State change requested RUNNING --> 
> FINISHED
> 2016-03-15 10:50:23,808 [2918170f-84ee-a6fb-38eb-266108bd64f3:frag:0:0] INFO  
> o.a.d.e.w.f.FragmentStatusReporter - 
> 2918170f-84ee-a6fb-38eb-266108bd64f3:0:0: State to report: FINISHED
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (DRILL-3434) Applying kvgen on a scalar (field within a map) results in "Unable to get holder type for minor type [LATE] and mode [OPTIONAL]"

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva resolved DRILL-3434.
-
   Resolution: Fixed
Fix Version/s: (was: Future)

> Applying kvgen on a scalar (field within a map) results in "Unable to get 
> holder type for minor type [LATE] and mode [OPTIONAL]"
> 
>
> Key: DRILL-3434
> URL: https://issues.apache.org/jira/browse/DRILL-3434
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow, Functions - Drill
>Reporter: Rahul Challapalli
>Assignee: Mehant Baid
> Attachments: error.log
>
>
> git.commit.id.abbrev=e3fc7e9
> Data :
> {code}
> {
>   "id":1,
>   "map": {
>   "map1":{"col1":1, "col2":2}
>   }
> }
> {code}
> The below query returns an error now
> {code}
> 0: jdbc:drill:schema=dfs_eea> select kvgen(d.map.abcd) from `data.json` d;
> Error: SYSTEM ERROR: UnsupportedOperationException: Unable to get holder type 
> for minor type [LATE] and mode [OPTIONAL]
> Fragment 0:0
> [Error Id: 62d8b3ad-62d1-43e8-a39a-18b19399fca0 on qa-node190.qa.lab:31010] 
> (state=,code=0)
> {code}
> It used to return a proper error message in the past (kvgen function only 
> supports Simple maps as input)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (DRILL-4519) File system directory-based partition pruning doesn't work correctly with parquet metadata

[Miroslav Holubec (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Miroslav Holubec resolved DRILL-4519.
-
Resolution: Fixed

Fixed probably as part of some other fix. Haven't observed it since 1.6.0...

> File system directory-based partition pruning doesn't work correctly with 
> parquet metadata
> --
>
> Key: DRILL-4519
> URL: https://issues.apache.org/jira/browse/DRILL-4519
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.4.0, 1.5.0, 1.6.0
>Reporter: Miroslav Holubec
>
> We have parquet files in folders with following convention /MM/DD/HH.
> Without drill's parquet metadata directory prunning works seamlessly.
> {noformat}
> select dir0, dir1, dir2 from hdfs.test.indexed;
> dir0 = ,  dir1 = MM, dir2 = DD, dir3 = HH
> {noformat}
> After creating metadata and executing same query, dir0 contains HH folder 
> name instead yearly folder name. dir1...3 are null.
> {noformat}
> refresh table metadata hdfs.test.indexed;
> select dir0, dir1, dir2 from hdfs.test.indexed;
> dir0 = HH,  dir1 = null, dir2 = null, dir3 = null
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (DRILL-3837) Metadata Caching : refresh metadata command on an empty dir should throw proper error message

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva resolved DRILL-3837.
-
Resolution: Fixed

> Metadata Caching : refresh metadata command on an empty dir should throw 
> proper error message
> -
>
> Key: DRILL-3837
> URL: https://issues.apache.org/jira/browse/DRILL-3837
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Metadata
>Reporter: Rahul Challapalli
>Priority: Minor
>
> git.commit.id.abbrev=3c89b30
> The 'mc' folder is empty without any files. The below error message should be 
> improved
> {code}
> refresh table metadata dfs.`/drill/mc`;
> ++---+
> |   ok   |  summary  |
> ++---+
> | false  | Error: Index: 0, Size: 0  |
> ++---+
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] drill pull request #865: DRILL-5634 - Add Crypto and Hash Functions

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/865#discussion_r125243537
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/CryptoFunctions.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.annotations.Workspace;
+import org.apache.drill.exec.expr.holders.VarCharHolder;
+
+import javax.inject.Inject;
+
+public class CryptoFunctions{
+static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CryptoFunctions.class);
+
+private CryptoFunctions() {}
+
+@FunctionTemplate(
+name = "md5",
+scope = FunctionTemplate.FunctionScope.SIMPLE,
+nulls = FunctionTemplate.NullHandling.NULL_IF_NULL
+)
+public static class md5Function implements DrillSimpleFunc {
+
+@Param
+VarCharHolder raw_input;
+
+@Output
+VarCharHolder out;
+
+@Inject
+DrillBuf buffer;
+
+@Workspace
+java.security.MessageDigest md;
+
+@Override
+public void setup() {
+try {
+md = java.security.MessageDigest.getInstance("MD5");
+} catch( Exception e ) {
--- End diff --

Please use proper formatting here and throughout the code [1].

[1] https://drill.apache.org/docs/apache-drill-contribution-guidelines/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #865: DRILL-5634 - Add Crypto and Hash Functions

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/865#discussion_r125248545
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/CryptoHelperFunctions.java
 ---
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.expr.fn.impl;
+import javax.crypto.Cipher;
+import javax.crypto.spec.SecretKeySpec;
+import javax.xml.bind.DatatypeConverter;
+import java.io.UnsupportedEncodingException;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.Arrays;
+
+public class CryptoHelperFunctions {
+
+private static SecretKeySpec secretKey;
+private static byte[] key;
--- End diff --

In concurrent environment this code will be prone to produce incorrect 
results, since each thread may have its own key. Saving state in static 
variables is not thread-safe.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #865: DRILL-5634 - Add Crypto and Hash Functions

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/865#discussion_r125248660
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/CryptoHelperFunctions.java
 ---
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.expr.fn.impl;
+import javax.crypto.Cipher;
+import javax.crypto.spec.SecretKeySpec;
+import javax.xml.bind.DatatypeConverter;
+import java.io.UnsupportedEncodingException;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.Arrays;
+
+public class CryptoHelperFunctions {
--- End diff --

Please add java doc describing the purpose of this class and also please 
add java doc to the methods below.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #865: DRILL-5634 - Add Crypto and Hash Functions

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/865#discussion_r125249038
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/CryptoFunctions.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.annotations.Workspace;
+import org.apache.drill.exec.expr.holders.VarCharHolder;
+
+import javax.inject.Inject;
+
+public class CryptoFunctions{
--- End diff --

Please add java-doc to all the functions below (you have provided good 
description in Jira but its better if we'll have it in code).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #865: DRILL-5634 - Add Crypto and Hash Functions

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/865#discussion_r125242571
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/CryptoFunctions.java
 ---
@@ -0,0 +1,315 @@
+/**
--- End diff --

Please include Apache header in a form of comment. Please correct here and 
in files below.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #865: DRILL-5634 - Add Crypto and Hash Functions

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/865#discussion_r125246241
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestCryptoFunctions.java
 ---
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.fn.impl;
+
+import org.apache.drill.BaseTestQuery;
+import org.junit.Test;
+
+public class TestCryptoFunctions extends BaseTestQuery {
+
+@Test
+public void testMD5() throws Exception {
+final String query = "select md5( 'testing' ) as md5_hash from 
(values(1))";
+testBuilder()
+.sqlQuery(query)
+.ordered()
+.baselineColumns("md5_hash")
+.baselineValues("ae2b1fca515949e5d54fb22b8ed95575")
+.go();
+}
+
+@Test
+public void testSHA1() throws Exception {
+final String query = "select sha( 'testing' ) as sha_hash from 
(values(1))";
--- End diff --

Please remove extra space here and in test below: `sha( 'testing' ) ` -> 
`sha('testing') `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #865: DRILL-5634 - Add Crypto and Hash Functions

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/865#discussion_r125244029
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/CryptoFunctions.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.annotations.Workspace;
+import org.apache.drill.exec.expr.holders.VarCharHolder;
+
+import javax.inject.Inject;
+
+public class CryptoFunctions{
+static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CryptoFunctions.class);
+
+private CryptoFunctions() {}
+
+@FunctionTemplate(
+name = "md5",
+scope = FunctionTemplate.FunctionScope.SIMPLE,
+nulls = FunctionTemplate.NullHandling.NULL_IF_NULL
+)
+public static class md5Function implements DrillSimpleFunc {
+
+@Param
+VarCharHolder raw_input;
+
+@Output
+VarCharHolder out;
+
+@Inject
+DrillBuf buffer;
+
+@Workspace
+java.security.MessageDigest md;
+
+@Override
+public void setup() {
+try {
+md = java.security.MessageDigest.getInstance("MD5");
+} catch( Exception e ) {
+}
+}
+
+@Override
+public void eval() {
+
+String input = 
org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(raw_input.start,
 raw_input.end, raw_input.buffer);
+byte[] thedigest = null;
+String output_string = "";
+
+try {
+byte[] bytesOfMessage = input.getBytes("UTF-8");
+thedigest = md.digest(bytesOfMessage);
+output_string = String.format("%032X", new 
java.math.BigInteger(1, thedigest));
+output_string = output_string.toLowerCase();
+
+} catch( Exception e ) {
+}
--- End diff --

1. Please mind formatting.
2. If in case of error, we leave the defaults, then it's better  to at 
least log the exception.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #865: DRILL-5634 - Add Crypto and Hash Functions

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/865#discussion_r125243335
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/CryptoFunctions.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.annotations.Workspace;
+import org.apache.drill.exec.expr.holders.VarCharHolder;
+
+import javax.inject.Inject;
+
+public class CryptoFunctions{
+static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CryptoFunctions.class);
+
+private CryptoFunctions() {}
+
+@FunctionTemplate(
+name = "md5",
+scope = FunctionTemplate.FunctionScope.SIMPLE,
+nulls = FunctionTemplate.NullHandling.NULL_IF_NULL
+)
+public static class md5Function implements DrillSimpleFunc {
+
+@Param
+VarCharHolder raw_input;
+
+@Output
+VarCharHolder out;
+
+@Inject
+DrillBuf buffer;
+
+@Workspace
+java.security.MessageDigest md;
+
+@Override
+public void setup() {
+try {
+md = java.security.MessageDigest.getInstance("MD5");
+} catch( Exception e ) {
+}
--- End diff --

What will happen if we couldn't get MD5 instance? Shouldn't we throw an 
exception in this case?
We won't be able to provide correct result anyway.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #865: DRILL-5634 - Add Crypto and Hash Functions

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/865#discussion_r125244592
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/CryptoFunctions.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.expr.DrillSimpleFunc;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import org.apache.drill.exec.expr.annotations.Output;
+import org.apache.drill.exec.expr.annotations.Param;
+import org.apache.drill.exec.expr.annotations.Workspace;
+import org.apache.drill.exec.expr.holders.VarCharHolder;
+
+import javax.inject.Inject;
+
+public class CryptoFunctions{
+static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CryptoFunctions.class);
+
+private CryptoFunctions() {}
+
+@FunctionTemplate(
+name = "md5",
+scope = FunctionTemplate.FunctionScope.SIMPLE,
+nulls = FunctionTemplate.NullHandling.NULL_IF_NULL
+)
+public static class md5Function implements DrillSimpleFunc {
--- End diff --

I guess according to java naming convention class names should start with 
letter in upper case.
Please correct here and in classes below.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Problems building Drill

[Muhammad Gelbana]

I tried a building from source using the same branch (i.e. *master*) after
creating a fresh clone and I faced some trouble
 but its none of what you
are mentioning here. Are you running the tests from the command line or
from eclipse ? Try the command line (i.e. mvn clean install)

One thing that caused the test failures (None of the issues you are
mentioning) on my machine was that my system had other things running and
that caused some timeout-oriented tests to fails. There are 2 constant
failures though but I haven't got the chance to look into that yet.

-- Gelbana

On Sun, Jul 2, 2017 at 10:43 PM, Charles Givre  wrote:

> Hello all,
> I’m having a small problem building Drill from source.  I keep getting the
> errors below when I try to run tests.  It builds fine when I skip the
> tests.  I’ve googled the errors and haven’t really found anything helpful.
> I’m not an expert on Maven so any suggestions would be helpful. Full stack
> trace below.  I’m on a Mac using Sierra.
>
> I tried mvn dependencies::tree -U and am getting this error as well:
>
> [ERROR] Failed to execute goal on project drill-jdbc: Could not resolve
> dependencies for project org.apache.drill.exec:drill-jdbc:jar:1.11.0-SNAPSHOT:
> The following artifacts could not be resolved: 
> org.apache.drill.exec:drill-java-exec:jar:1.11.0-SNAPSHOT,
> org.apache.drill.exec:drill-java-exec:jar:tests:1.11.0-SNAPSHOT: Could
> not find artifact org.apache.drill.exec:drill-java-exec:jar:1.11.0-SNAPSHOT
> in mapr-drill-optiq-snapshots (http://repository.mapr.com/
> nexus/content/repositories/drill-optiq/) -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal on project drill-jdbc: Could not resolve dependencies for project
> org.apache.drill.exec:drill-jdbc:jar:1.11.0-SNAPSHOT: The following
> artifacts could not be resolved: 
> org.apache.drill.exec:drill-java-exec:jar:1.11.0-SNAPSHOT,
> org.apache.drill.exec:drill-java-exec:jar:tests:1.11.0-SNAPSHOT: Could
> not find artifact org.apache.drill.exec:drill-java-exec:jar:1.11.0-SNAPSHOT
> in mapr-drill-optiq-snapshots (http://repository.mapr.com/
> nexus/content/repositories/drill-optiq/)
> at org.apache.maven.lifecycle.internal.
> LifecycleDependencyResolver.getDependencies(LifecycleDependencyRe
>
>
> Thanks,
> — C
>
>
> [ERROR] Failed to execute goal org.apache.maven.plugins:
> maven-surefire-plugin:2.17:test (default-test) on project
> drill-java-exec: ExecutionException: java.lang.RuntimeException: There was
> an error in the forked process
> [ERROR] java.lang.NoClassDefFoundError: mockit/internal/state/TestRun
> [ERROR] at org.junit.runner.notification.RunNotifier.
> fireTestRunStarted(RunNotifier.java)
> [ERROR] at org.junit.runner.JUnitCore.run(JUnitCore.java:136)
> [ERROR] at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> [ERROR] at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.
> createRequestAndRun(JUnitCoreWrapper.java:113)
> [ERROR] at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.
> executeLazy(JUnitCoreWrapper.java:94)
> [ERROR] at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.
> execute(JUnitCoreWrapper.java:58)
> [ERROR] at org.apache.maven.surefire.junitcore.JUnitCoreProvider.
> invoke(JUnitCoreProvider.java:134)
> [ERROR] at org.apache.maven.surefire.booter.ForkedBooter.
> invokeProviderInSameClassLoader(ForkedBooter.java:200)
> [ERROR] at org.apache.maven.surefire.booter.ForkedBooter.
> runSuitesInProcess(ForkedBooter.java:153)
> [ERROR] at org.apache.maven.surefire.booter.ForkedBooter.main(
> ForkedBooter.java:103)
> [ERROR] Caused by: java.lang.ClassNotFoundException:
> mockit.internal.state.TestRun
> [ERROR] at java.net.URLClassLoader.findClass(URLClassLoader.java:
> 381)
> [ERROR] at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> [ERROR] at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> [ERROR] ... 10 more
> [ERROR] -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test
> (default-test) on project drill-java-exec: ExecutionException
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(
> MojoExecutor.java:213)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(
> MojoExecutor.java:154)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(
> MojoExecutor.java:146)
> at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.
> buildProject(LifecycleModuleBuilder.java:117)
> at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.
> buildProject(LifecycleModuleBuilder.java:81)
> at org.apache.maven.lifecycle.internal.builder.singlethreaded.
> SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> at 

Re: Why rules from all plugins contribute into optimizing any type of query ?

[Muhammad Gelbana]

​Thanks a lot everyone.

Aman, your answer is very convincing. You made it clear that since a single
query can involve multiple plugins, then all rules provided by at least the
involved plugins must be considered by the planner.

-- Gelbana

On Mon, Jul 3, 2017 at 4:04 AM, Aman Sinha  wrote:

> Agree with Ted and Julian's comments and would add one more point: The
> Planner registers the Storage plugin optimizer rules from all the plugins
> [1].  The assumption is that a single query could be querying multiple data
> sources, joining them in Drill etc, so it is the rule's responsibility to
> have the proper constructor and matches() method to specialize it.   For
> example,  if you have a logical planning rule and have a DrillScanRel which
> is supposed to have a GroupScan of JDBCGroupScan,  you could check that in
> your matches() and return False if otherwise.
>
>
> [1]
> https://github.com/apache/drill/blob/master/exec/java-
> exec/src/main/java/org/apache/drill/exec/planner/PlannerPhase.java#L197
>
> Aman
>
> On Sun, Jul 2, 2017 at 11:47 AM, Julian Hyde  wrote:
>
> > What Ted said.
> >
> > But also, conversely, you should know that in Calcite you can write a
> > general-purpose rule. Or better, re-use a general-purpose rule that
> someone
> > else has written. There are logical rules, for example constant reduction
> > and logic simplification, that work regardless of the data source. And
> > there are common patterns, for example pushing down projects and filters,
> > that apply to many different data sources. You would want to push down
> > projects and filters to Parquet just as would would want to push them
> into
> > a JDBC source, albeit that Parquet cannot handle as rich an expression
> > language.
> >
> > General-purpose rules often take classes as constructor parameters, so
> you
> > can instantiate the rule to look for a FooProject.class rather than a
> > JdbcProject.class or LogicalProject.class.
> >
> > Julian
> >
> >
> > > On Jul 2, 2017, at 11:08 AM, Ted Dunning 
> wrote:
> > >
> > > It all depends on how you write your rules. If you write them so that
> > they
> > > apply too generally, then the rules themselves are at fault.
> > >
> > > If you write rules that only optimize for your input format, then you
> > > should be fine.
> > >
> > >
> > > On Jul 2, 2017 9:41 AM, "Muhammad Gelbana" 
> wrote:
> > >
> > >> I wrote a plugin for a custom JDBC datasource. This plugin registers a
> > >> couple of rules.
> > >>
> > >> When I ran an SQL query that uses parquet files, I found that a rule
> of
> > my
> > >> JDBC plugin was invoked to optimize the query !
> > >>
> > >> I believe this is a mistake. Please correct me if I'm wrong.
> > >>
> > >> I'm saying this is a mistake because a rule registered by a plugin
> that
> > >> utilizes a specific datasource should only be concerned about queries
> > >> executed by that plugin.
> > >>
> > >> Query optimizations for a JDBC datasource won't probably work for
> > queries
> > >> targeted for parquet files !
> > >>
> > >> What do you think ?
> > >>
> > >> Gelbana
> > >>
> >
> >
>