date:20151221

[jira] [Created] (DRILL-4215) Transfer ownership of buffers when doing transfers

2015-12-21 Thread Steven Phillips (JIRA)

Steven Phillips created DRILL-4215:
--

 Summary: Transfer ownership of buffers when doing transfers
 Key: DRILL-4215
 URL: https://issues.apache.org/jira/browse/DRILL-4215
 Project: Apache Drill
  Issue Type: Bug
Reporter: Steven Phillips
Assignee: Steven Phillips


The new allocator has the feature of allowing the transfer of ownership of 
buffers from one allocator to another. We should make use of this feature by 
transferring ownership whenever we transfer buffers between vectors. This will 
allow better tracking of how much memory operators are holding on to.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4215) Transfer ownership of buffers when doing transfers

2015-12-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067057#comment-15067057
 ] 

ASF GitHub Bot commented on DRILL-4215:
---

GitHub user StevenMPhillips opened a pull request:

https://github.com/apache/drill/pull/308

DRILL-4215: Transfer ownership in TransferPair

Only commit 79bbfcb is part of this commit. But since the other commits 
haven't been merged to master, they are also showing up in this PR.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StevenMPhillips/drill drill-4215

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/308.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #308


commit ea7bbc58f3e4bed5dc8093381d6c7dc4911f520e
Author: Chris Westin 
Date:   2015-11-11T22:57:16Z

DRILL-4144: Clean up close behaviors.

commit da4f9821644520bf2c4eb698770ed87713b1a039
Author: Chris Westin 
Date:   2015-11-11T22:57:47Z

DRILL-4144: Add new allocator

commit 728687f8d7c28d398ac993705dc0e4fb189b01d4
Author: Jacques Nadeau 
Date:   2015-11-16T01:26:02Z

DRILL-4144: Allocator Improvements

- make Allocator mostly lockless
- change BaseAllocator maps to direct references
- add documentation around memory management model
- move transfer and ownership methods to DrillBuf
- Improve debug messaging.
- Fix/revert sort changes
- Remove unused fragment limit flag
- Add time to HistoricalLog events
- Remove reservation amount from RootAllocator constructor (since not 
allowed)
- Fix concurrency issue where allocator is closing at same moment as 
incoming batch transfer, causing leaked memory and/or query failure.
- Add new AutoCloseables.close(Iterable)
- Remove extraneous DataResponseHandler and Impl (and update TestBitRpc to 
use smarter mock of FragmentManager)
- Remove the concept of poison pill record batches, using instead 
FragmentContext.isOverMemoryLimit()
- Update incoming data batches so that they are transferred under 
protection of a close lock
- Improve field names in IncomingBuffers and move synchronization to 
collectors as opposed to IncomingBuffers (also change decrementing to 
decrementToZero rather than two part check).

commit 79bbfcb018429ad3a02daf2ac4dea119126041e1
Author: Steven Phillips 
Date:   2015-12-01T08:34:41Z

DRILL-4215: Transfer buffer ownership in TransferPair




> Transfer ownership of buffers when doing transfers
> --
>
> Key: DRILL-4215
> URL: https://issues.apache.org/jira/browse/DRILL-4215
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Steven Phillips
>Assignee: Steven Phillips
>
> The new allocator has the feature of allowing the transfer of ownership of 
> buffers from one allocator to another. We should make use of this feature by 
> transferring ownership whenever we transfer buffers between vectors. This 
> will allow better tracking of how much memory operators are holding on to.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4192) Dir0 and Dir1 from drill-1.4 are messed up

2015-12-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066925#comment-15066925
 ] 

ASF GitHub Bot commented on DRILL-4192:
---

Github user hnfgns commented on a diff in the pull request:

https://github.com/apache/drill/pull/307#discussion_r48180127
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSelection.java 
---
@@ -213,14 +214,15 @@ public static FileSelection create(final 
List statuses, final List Dir0 and Dir1 from drill-1.4 are messed up
> --
>
> Key: DRILL-4192
> URL: https://issues.apache.org/jira/browse/DRILL-4192
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.4.0
>Reporter: Krystal
>Assignee: Aman Sinha
>Priority: Blocker
>
> I have the following directories:
> /drill/testdata/temp1/abc/dt=2014-12-30/lineitem.parquet
> /drill/testdata/temp1/abc/dt=2014-12-31/lineitem.parquet
> The following queries returned incorrect data.
> select dir0,dir1 from dfs.`/drill/testdata/temp1` limit 2;
> ++---+
> |  dir0  | dir1  |
> ++---+
> | dt=2014-12-30  | null  |
> | dt=2014-12-30  | null  |
> ++---+
> select dir0 from dfs.`/drill/testdata/temp1` limit 2;
> ++
> |  dir0  |
> ++
> | dt=2014-12-31  |
> | dt=2014-12-31  |
> ++



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4192) Dir0 and Dir1 from drill-1.4 are messed up

2015-12-21 Thread Mehant Baid (JIRA)


 [ 
https://issues.apache.org/jira/browse/DRILL-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mehant Baid updated DRILL-4192:
---
Assignee: Aman Sinha  (was: Mehant Baid)

> Dir0 and Dir1 from drill-1.4 are messed up
> --
>
> Key: DRILL-4192
> URL: https://issues.apache.org/jira/browse/DRILL-4192
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.4.0
>Reporter: Krystal
>Assignee: Aman Sinha
>Priority: Blocker
>
> I have the following directories:
> /drill/testdata/temp1/abc/dt=2014-12-30/lineitem.parquet
> /drill/testdata/temp1/abc/dt=2014-12-31/lineitem.parquet
> The following queries returned incorrect data.
> select dir0,dir1 from dfs.`/drill/testdata/temp1` limit 2;
> ++---+
> |  dir0  | dir1  |
> ++---+
> | dt=2014-12-30  | null  |
> | dt=2014-12-30  | null  |
> ++---+
> select dir0 from dfs.`/drill/testdata/temp1` limit 2;
> ++
> |  dir0  |
> ++
> | dt=2014-12-31  |
> | dt=2014-12-31  |
> ++



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4192) Dir0 and Dir1 from drill-1.4 are messed up

2015-12-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066937#comment-15066937
 ] 

ASF GitHub Bot commented on DRILL-4192:
---

Github user hnfgns commented on a diff in the pull request:

https://github.com/apache/drill/pull/307#discussion_r48180622
  
--- Diff: exec/java-exec/src/test/java/org/apache/drill/TestBugFixes.java 
---
@@ -144,4 +147,15 @@ public void testDRILL2361_JoinColumnAliasWithDots() 
throws Exception {
 .build().run();
   }
 
+  @Test
+  public void testDRILL4192() throws Exception {
+String query = (String.format("select dir0, dir1 from 
dfs_test.`%s/bugs/DRILL-4192` order by dir1", TEST_RES_PATH));
--- End diff --

I am wondering if this test should also include an HDFS wildcard.


> Dir0 and Dir1 from drill-1.4 are messed up
> --
>
> Key: DRILL-4192
> URL: https://issues.apache.org/jira/browse/DRILL-4192
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.4.0
>Reporter: Krystal
>Assignee: Aman Sinha
>Priority: Blocker
>
> I have the following directories:
> /drill/testdata/temp1/abc/dt=2014-12-30/lineitem.parquet
> /drill/testdata/temp1/abc/dt=2014-12-31/lineitem.parquet
> The following queries returned incorrect data.
> select dir0,dir1 from dfs.`/drill/testdata/temp1` limit 2;
> ++---+
> |  dir0  | dir1  |
> ++---+
> | dt=2014-12-30  | null  |
> | dt=2014-12-30  | null  |
> ++---+
> select dir0 from dfs.`/drill/testdata/temp1` limit 2;
> ++
> |  dir0  |
> ++
> | dt=2014-12-31  |
> | dt=2014-12-31  |
> ++



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4192) Dir0 and Dir1 from drill-1.4 are messed up

2015-12-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066919#comment-15066919
 ] 

ASF GitHub Bot commented on DRILL-4192:
---

Github user hnfgns commented on a diff in the pull request:

https://github.com/apache/drill/pull/307#discussion_r48179942
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSelection.java 
---
@@ -201,7 +202,7 @@ public static FileSelection create(final 
DrillFileSystem fs, final String parent
*
* @see FileSelection#FileSelection(List, List, String)
*/
-  public static FileSelection create(final List statuses, 
final List files, final String root) {
+  public static FileSelection create(final List statuses, 
final List files, String root) {
--- End diff --

I would recommend avoiding the re-assignment to root.


> Dir0 and Dir1 from drill-1.4 are messed up
> --
>
> Key: DRILL-4192
> URL: https://issues.apache.org/jira/browse/DRILL-4192
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.4.0
>Reporter: Krystal
>Assignee: Aman Sinha
>Priority: Blocker
>
> I have the following directories:
> /drill/testdata/temp1/abc/dt=2014-12-30/lineitem.parquet
> /drill/testdata/temp1/abc/dt=2014-12-31/lineitem.parquet
> The following queries returned incorrect data.
> select dir0,dir1 from dfs.`/drill/testdata/temp1` limit 2;
> ++---+
> |  dir0  | dir1  |
> ++---+
> | dt=2014-12-30  | null  |
> | dt=2014-12-30  | null  |
> ++---+
> select dir0 from dfs.`/drill/testdata/temp1` limit 2;
> ++
> |  dir0  |
> ++
> | dt=2014-12-31  |
> | dt=2014-12-31  |
> ++



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4192) Dir0 and Dir1 from drill-1.4 are messed up

2015-12-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066931#comment-15066931
 ] 

ASF GitHub Bot commented on DRILL-4192:
---

Github user hnfgns commented on a diff in the pull request:

https://github.com/apache/drill/pull/307#discussion_r48180355
  
--- Diff: exec/java-exec/src/test/java/org/apache/drill/TestBugFixes.java 
---
@@ -18,12 +18,15 @@
 package org.apache.drill;
 
 import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.util.TestTools;
 import org.apache.drill.exec.planner.physical.PlannerSettings;
 import org.junit.Ignore;
 import org.junit.Test;
 
 public class TestBugFixes extends BaseTestQuery {
   private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(TestBugFixes.class);
+  static final String WORKING_PATH = TestTools.getWorkingPath();
+  static final String TEST_RES_PATH = WORKING_PATH + "/src/test/resources";
--- End diff --

You may like to use TestTools.getTestResourcesPath() instead, less 
fail-prone.


> Dir0 and Dir1 from drill-1.4 are messed up
> --
>
> Key: DRILL-4192
> URL: https://issues.apache.org/jira/browse/DRILL-4192
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.4.0
>Reporter: Krystal
>Assignee: Aman Sinha
>Priority: Blocker
>
> I have the following directories:
> /drill/testdata/temp1/abc/dt=2014-12-30/lineitem.parquet
> /drill/testdata/temp1/abc/dt=2014-12-31/lineitem.parquet
> The following queries returned incorrect data.
> select dir0,dir1 from dfs.`/drill/testdata/temp1` limit 2;
> ++---+
> |  dir0  | dir1  |
> ++---+
> | dt=2014-12-30  | null  |
> | dt=2014-12-30  | null  |
> ++---+
> select dir0 from dfs.`/drill/testdata/temp1` limit 2;
> ++
> |  dir0  |
> ++
> | dt=2014-12-31  |
> | dt=2014-12-31  |
> ++



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (DRILL-4205) Simple query hit IndexOutOfBoundException

2015-12-21 Thread Dechang Gu (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067182#comment-15067182
 ] 

Dechang Gu edited comment on DRILL-4205 at 12/21/15 10:37 PM:
--

Parth,  which repo is this fix ( abdfc6a )  in?I looked in your repo 
https://github.com/parthchandra/incubator-drill/commits/DRILL-4205  , but could 
not find this commit (the top commit id is  
aaa6bb850b9de1e86afe2e0a5afad90d753fca04)


was (Author: dechanggu):
Parth,  which repo is this fix ( abdfc6a )  in?I looked in your repo 
https://github.com/parthchandra/incubator-drill/commits/DRILL-4205  , but could 
not find this commit (the top commit id is  
aaa6bb850b9de1e86afe2e0a5afad90d753fca04

>  Simple query hit IndexOutOfBoundException
> --
>
> Key: DRILL-4205
> URL: https://issues.apache.org/jira/browse/DRILL-4205
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.4.0
>Reporter: Dechang Gu
>Assignee: Parth Chandra
>
> The following query failed due to IOB:
> 0: jdbc:drill:schema=wf_pigprq100> select * from 
> `store_sales/part-m-00073.parquet`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: srcIndex: 1048587
> Fragment 0:0
> [Error Id: ad8d2bc0-259f-483c-9024-93865963541e on ucs-node4.perf.lab:31010]
>   (org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet 
> record reader.
> Message: 
> Hadoop path: /tpcdsPigParq/SF100/store_sales/part-m-00073.parquet
> Total records read: 135280
> Mock records read: 0
> Records to read: 1424
> Row group index: 0
> Records in row group: 3775712
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message pig_schema {
>   optional int64 ss_sold_date_sk;
>   optional int64 ss_sold_time_sk;
>   optional int64 ss_item_sk;
>   optional int64 ss_customer_sk;
>   optional int64 ss_cdemo_sk;
>   optional int64 ss_hdemo_sk;
>   optional int64 ss_addr_sk;
>   optional int64 ss_store_sk;
>   optional int64 ss_promo_sk;
>   optional int64 ss_ticket_number;
>   optional int64 ss_quantity;
>   optional double ss_wholesale_cost;
>   optional double ss_list_price;
>   optional double ss_sales_price;
>   optional double ss_ext_discount_amt;
>   optional double ss_ext_sales_price;
>   optional double ss_ext_wholesale_cost;
>   optional double ss_ext_list_price;
>   optional double ss_ext_tax;
>   optional double ss_coupon_amt;
>   optional double ss_net_paid;
>   optional double ss_net_paid_inc_tax;
>   optional double ss_net_profit;
> }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4205) Simple query hit IndexOutOfBoundException

2015-12-21 Thread Dechang Gu (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067182#comment-15067182
 ] 

Dechang Gu commented on DRILL-4205:
---

Parth,  which repo is this fix ( abdfc6a )  in?I looked in your repo 
https://github.com/parthchandra/incubator-drill/commits/DRILL-4205  , but could 
not find this commit (the top commit id is  
aaa6bb850b9de1e86afe2e0a5afad90d753fca04

>  Simple query hit IndexOutOfBoundException
> --
>
> Key: DRILL-4205
> URL: https://issues.apache.org/jira/browse/DRILL-4205
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.4.0
>Reporter: Dechang Gu
>Assignee: Parth Chandra
>
> The following query failed due to IOB:
> 0: jdbc:drill:schema=wf_pigprq100> select * from 
> `store_sales/part-m-00073.parquet`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: srcIndex: 1048587
> Fragment 0:0
> [Error Id: ad8d2bc0-259f-483c-9024-93865963541e on ucs-node4.perf.lab:31010]
>   (org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet 
> record reader.
> Message: 
> Hadoop path: /tpcdsPigParq/SF100/store_sales/part-m-00073.parquet
> Total records read: 135280
> Mock records read: 0
> Records to read: 1424
> Row group index: 0
> Records in row group: 3775712
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message pig_schema {
>   optional int64 ss_sold_date_sk;
>   optional int64 ss_sold_time_sk;
>   optional int64 ss_item_sk;
>   optional int64 ss_customer_sk;
>   optional int64 ss_cdemo_sk;
>   optional int64 ss_hdemo_sk;
>   optional int64 ss_addr_sk;
>   optional int64 ss_store_sk;
>   optional int64 ss_promo_sk;
>   optional int64 ss_ticket_number;
>   optional int64 ss_quantity;
>   optional double ss_wholesale_cost;
>   optional double ss_list_price;
>   optional double ss_sales_price;
>   optional double ss_ext_discount_amt;
>   optional double ss_ext_sales_price;
>   optional double ss_ext_wholesale_cost;
>   optional double ss_ext_list_price;
>   optional double ss_ext_tax;
>   optional double ss_coupon_amt;
>   optional double ss_net_paid;
>   optional double ss_net_paid_inc_tax;
>   optional double ss_net_profit;
> }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4205) Simple query hit IndexOutOfBoundException

2015-12-21 Thread Dechang Gu (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067358#comment-15067358
 ] 

Dechang Gu commented on DRILL-4205:
---

Found the patch in the Apache Drill master repo. And tested it -- looks likes 
the issue is resolved.

Thanks,
Dechang

>  Simple query hit IndexOutOfBoundException
> --
>
> Key: DRILL-4205
> URL: https://issues.apache.org/jira/browse/DRILL-4205
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.4.0
>Reporter: Dechang Gu
>Assignee: Parth Chandra
>
> The following query failed due to IOB:
> 0: jdbc:drill:schema=wf_pigprq100> select * from 
> `store_sales/part-m-00073.parquet`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: srcIndex: 1048587
> Fragment 0:0
> [Error Id: ad8d2bc0-259f-483c-9024-93865963541e on ucs-node4.perf.lab:31010]
>   (org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet 
> record reader.
> Message: 
> Hadoop path: /tpcdsPigParq/SF100/store_sales/part-m-00073.parquet
> Total records read: 135280
> Mock records read: 0
> Records to read: 1424
> Row group index: 0
> Records in row group: 3775712
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message pig_schema {
>   optional int64 ss_sold_date_sk;
>   optional int64 ss_sold_time_sk;
>   optional int64 ss_item_sk;
>   optional int64 ss_customer_sk;
>   optional int64 ss_cdemo_sk;
>   optional int64 ss_hdemo_sk;
>   optional int64 ss_addr_sk;
>   optional int64 ss_store_sk;
>   optional int64 ss_promo_sk;
>   optional int64 ss_ticket_number;
>   optional int64 ss_quantity;
>   optional double ss_wholesale_cost;
>   optional double ss_list_price;
>   optional double ss_sales_price;
>   optional double ss_ext_discount_amt;
>   optional double ss_ext_sales_price;
>   optional double ss_ext_wholesale_cost;
>   optional double ss_ext_list_price;
>   optional double ss_ext_tax;
>   optional double ss_coupon_amt;
>   optional double ss_net_paid;
>   optional double ss_net_paid_inc_tax;
>   optional double ss_net_profit;
> }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4134) Incorporate remaining patches from DRILL-1942 Allocator refactor

2015-12-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067332#comment-15067332
 ] 

ASF GitHub Bot commented on DRILL-4134:
---

Github user adeneche commented on the pull request:

https://github.com/apache/drill/pull/283#issuecomment-166466013
  
Results of the performance test are good, no regressions.
Only one comment left to address (peakAllocation computation), but other 
than that:
+1


> Incorporate remaining patches from DRILL-1942 Allocator refactor
> 
>
> Key: DRILL-4134
> URL: https://issues.apache.org/jira/browse/DRILL-4134
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Execution - Flow
>Reporter: Jacques Nadeau
>Assignee: Jacques Nadeau
> Fix For: 1.5.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3478) Bson Record Reader for Mongo storage plugin

2015-12-21 Thread B Anil Kumar (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067341#comment-15067341
 ] 

B Anil Kumar commented on DRILL-3478:
-

[~jnadeau] Can you please review latest patch with review comments fixes?

> Bson Record Reader for Mongo storage plugin
> ---
>
> Key: DRILL-3478
> URL: https://issues.apache.org/jira/browse/DRILL-3478
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - MongoDB
>Reporter: B Anil Kumar
>Assignee: B Anil Kumar
> Fix For: Future
>
> Attachments: 0001-DRILL-3478_1-Review-comments-fixes.patch, 
> Test_queries_with_review_comment_fixes, drill_bson_sqlline_test_2015_1
>
>
> Improve the mongo query performance.
> We are considering the suggestions provided by [~dragoncurve] and [~hgunes] 
> in drill mailing chain.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4192) Dir0 and Dir1 from drill-1.4 are messed up

2015-12-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067303#comment-15067303
 ] 

ASF GitHub Bot commented on DRILL-4192:
---

Github user mehant commented on the pull request:

https://github.com/apache/drill/pull/307#issuecomment-166463713
  
@hnfgns updated PR based on your review comments.


> Dir0 and Dir1 from drill-1.4 are messed up
> --
>
> Key: DRILL-4192
> URL: https://issues.apache.org/jira/browse/DRILL-4192
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.4.0
>Reporter: Krystal
>Assignee: Aman Sinha
>Priority: Blocker
>
> I have the following directories:
> /drill/testdata/temp1/abc/dt=2014-12-30/lineitem.parquet
> /drill/testdata/temp1/abc/dt=2014-12-31/lineitem.parquet
> The following queries returned incorrect data.
> select dir0,dir1 from dfs.`/drill/testdata/temp1` limit 2;
> ++---+
> |  dir0  | dir1  |
> ++---+
> | dt=2014-12-30  | null  |
> | dt=2014-12-30  | null  |
> ++---+
> select dir0 from dfs.`/drill/testdata/temp1` limit 2;
> ++
> |  dir0  |
> ++
> | dt=2014-12-31  |
> | dt=2014-12-31  |
> ++



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3584) Drill Kerberos HDFS Support / Documentation

2015-12-21 Thread Christopher Soza (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066280#comment-15066280
 ] 

Christopher Soza commented on DRILL-3584:
-

4 months have gone by and still this is pending...Wonder who uses Apache Drill 
in Enterprise environment without Kerberos support..I would;ve thought that 
this should be the no. 1 priority. 

> Drill Kerberos HDFS Support / Documentation
> ---
>
> Key: DRILL-3584
> URL: https://issues.apache.org/jira/browse/DRILL-3584
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.1.0
>Reporter: Hari Sekhon
>Priority: Blocker
>
> I'm trying to find Drill docs for Kerberos support for secure HDFS clusters 
> and it doesn't appear to be well tested / supported / documented yet.
> This product is Dead-on-Arrival if it doesn't integrate well with secure 
> Hadoop clusters, specifically HDFS + Kerberos (plus obviously secure 
> kerberized Hive/HCatalog etc.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3584) Drill Kerberos HDFS Support / Documentation

2015-12-21 Thread Christopher Soza (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066315#comment-15066315
 ] 

Christopher Soza commented on DRILL-3584:
-

But this is an Apache Project !! 

> Drill Kerberos HDFS Support / Documentation
> ---
>
> Key: DRILL-3584
> URL: https://issues.apache.org/jira/browse/DRILL-3584
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.1.0
>Reporter: Hari Sekhon
>Priority: Blocker
>
> I'm trying to find Drill docs for Kerberos support for secure HDFS clusters 
> and it doesn't appear to be well tested / supported / documented yet.
> This product is Dead-on-Arrival if it doesn't integrate well with secure 
> Hadoop clusters, specifically HDFS + Kerberos (plus obviously secure 
> kerberized Hive/HCatalog etc.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4216) Aggregate Window Function COUNT() With GROUP BY Clause expected: range(0, 32768)

2015-12-21 Thread PIPELINE (JIRA)

PIPELINE created DRILL-4216:
---

 Summary: Aggregate Window Function COUNT() With GROUP BY Clause 
expected: range(0, 32768)
 Key: DRILL-4216
 URL: https://issues.apache.org/jira/browse/DRILL-4216
 Project: Apache Drill
  Issue Type: Bug
  Components: Functions - Drill
Affects Versions: 1.3.0
 Environment: Hadoop-2.5.2
Hbase-0.9.15
Java-1.7.0_85
Reporter: PIPELINE


*When column is row_key,it work well !*
0: jdbc:drill:> select count(row_key) over() from hbase.web_initial_20151222 wi 
group by row_key limit 3;
+-+
| EXPR$0  |
+-+
| 102906  |
| 102906  |
| 102906  |
+-+
3 rows selected (1.645 seconds)

*When column is Hbase.Talbename.ColumnFamily.Qualify, and count(column) less 
than 32768,it work well !*
0: jdbc:drill:> select count(wi.cf1.q5) over() from hbase.web_initial_20151214 
wi group by wi.cf1.q5 limit 3;
+-+
| EXPR$0  |
+-+
| 10383   |
| 10383   |
| 10383   |
+-+
3 rows selected (1.044 seconds)

{color:red}
When column is Hbase.Talbename.ColumnFamily.Qualify, and count(column) more 
than 32768,IndexOutOfBoundsException
{color}

0: jdbc:drill:> select count(wi.cf1.q5) over() from hbase.web_initial_20151222 
wi group by wi.cf1.q5 limit 3;
Error: SYSTEM ERROR: IndexOutOfBoundsException: index: 0, length: 62784 
(expected: range(0, 32768))

Fragment 0:0

[Error Id: 77406a8a-8389-4f1b-af6c-d26d811379b7 on slave4.hadoop:31010] 
(state=,code=0)
java.sql.SQLException: SYSTEM ERROR: IndexOutOfBoundsException: index: 0, 
length: 62784 (expected: range(0, 32768))

Fragment 0:0

[Error Id: 77406a8a-8389-4f1b-af6c-d26d811379b7 on slave4.hadoop:31010]
at 
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:247)
at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:320)
at 
net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
at 
org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:160)
at sqlline.IncrementalRows.hasNext(IncrementalRows.java:62)
at 
sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
at sqlline.SqlLine.print(SqlLine.java:1593)
at sqlline.Commands.execute(Commands.java:852)
at sqlline.Commands.sql(Commands.java:751)
at sqlline.SqlLine.dispatch(SqlLine.java:746)
at sqlline.SqlLine.begin(SqlLine.java:621)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM 
ERROR: IndexOutOfBoundsException: index: 0, length: 62784 (expected: range(0, 
32768))

Fragment 0:0

[Error Id: 77406a8a-8389-4f1b-af6c-d26d811379b7 on slave4.hadoop:31010]
at 
org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:118)
at 
org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:112)
at 
org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:47)
at 
org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:32)
at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:69)
at org.apache.drill.exec.rpc.RpcBus$RequestEvent.run(RpcBus.java:400)
at 
org.apache.drill.common.SerializedExecutor$RunnableProcessor.run(SerializedExecutor.java:105)
at 
org.apache.drill.exec.rpc.RpcBus$SameExecutor.execute(RpcBus.java:264)
at 
org.apache.drill.common.SerializedExecutor.execute(SerializedExecutor.java:142)
at 
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:298)
at 
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:269)
at 
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at 
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at 
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at

[jira] [Closed] (DRILL-3765) Partition prune rule is unnecessary fired multiple times.

2015-12-21 Thread Dechang Gu (JIRA)


 [ 
https://issues.apache.org/jira/browse/DRILL-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dechang Gu closed DRILL-3765.
-

Verified and tested.  It looks fine. 

alter session set `planner.enable_hep_partition_pruning` = false;
explain plan for select ss_sold_date_sk, ss_sold_time_sk, ss_item_sk, 
ss_customer_sk from store_pb_store_sk where ss_store_sk in (11,15, 7, 202, 278, 
100, 200, 300, 400, 500) and ss_customer_sk = 96479;

1 row selected (140.404 seconds)


alter session set `planner.enable_hep_partition_pruning` = true;
explain plan for select ss_sold_date_sk, ss_sold_time_sk, ss_item_sk, 
ss_customer_sk from store_pb_store_sk where ss_store_sk in (11,15, 7, 202, 278, 
100, 200, 300, 400, 500) and ss_customer_sk = 96479;

1 row selected (51.001 seconds)

> Partition prune rule is unnecessary fired multiple times. 
> --
>
> Key: DRILL-3765
> URL: https://issues.apache.org/jira/browse/DRILL-3765
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
> Fix For: 1.4.0
>
>
> It seems that the partition prune rule may be fired multiple times, even 
> after the first rule execution has pushed the filter into the scan operator. 
> Since partition prune has to build the vectors to contain the partition /file 
> / directory information, to invoke the partition prune rule unnecessary may 
> lead to big memory overhead.
> Drill planner should avoid the un-necessary partition prune rule, in order to 
> reduce the chance of hitting OOM exception, while the partition prune rule is 
> executed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (DRILL-3584) Drill Kerberos HDFS Support / Documentation

2015-12-21 Thread Hari Sekhon (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066323#comment-15066323
 ] 

Hari Sekhon edited comment on DRILL-3584 at 12/21/15 11:06 AM:
---

Last time I pressed this issue Ted Dunning suggested that I implemented the 
HDFS Kerberos integration myself. If only If I was still on a vendor payroll 
and getting steady salary to spend time on this kind of thing...

I'm not holding my breath for this to happen soon... we ended up dropping Drill 
here - as I mentioned in my original post it's Dead-On-Arrival without the HDFS 
Kerberos integration for most governed enterprise environments.


was (Author: harisekhon):
Last time I pressed this issue Ted Dunning suggested that I implemented the 
HDFS Kerberos integration myself. If only If I was still on a vendor payroll 
and getting steady salary to spend time on this kind of thing...

> Drill Kerberos HDFS Support / Documentation
> ---
>
> Key: DRILL-3584
> URL: https://issues.apache.org/jira/browse/DRILL-3584
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.1.0
>Reporter: Hari Sekhon
>Priority: Blocker
>
> I'm trying to find Drill docs for Kerberos support for secure HDFS clusters 
> and it doesn't appear to be well tested / supported / documented yet.
> This product is Dead-on-Arrival if it doesn't integrate well with secure 
> Hadoop clusters, specifically HDFS + Kerberos (plus obviously secure 
> kerberized Hive/HCatalog etc.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3584) Drill Kerberos HDFS Support / Documentation

2015-12-21 Thread Hari Sekhon (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066302#comment-15066302
 ] 

Hari Sekhon commented on DRILL-3584:


MapR preferred not to run Kerberos on their clusters last I used them (about a 
year ago). While they have some kerberos integration they usually recommended 
their own ticketing system, MapR "native security", to secure the base 
filesystem... it's not clear how much of the Apache ecosystem actually 
integrates with this alternative mechanism though. I agree this is a 
disadvantage to the Drill project and the limited adoption will prevent the 
implicit additional QA of wider usage on the HDFS based platforms which require 
the standard Kerberos security.

> Drill Kerberos HDFS Support / Documentation
> ---
>
> Key: DRILL-3584
> URL: https://issues.apache.org/jira/browse/DRILL-3584
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.1.0
>Reporter: Hari Sekhon
>Priority: Blocker
>
> I'm trying to find Drill docs for Kerberos support for secure HDFS clusters 
> and it doesn't appear to be well tested / supported / documented yet.
> This product is Dead-on-Arrival if it doesn't integrate well with secure 
> Hadoop clusters, specifically HDFS + Kerberos (plus obviously secure 
> kerberized Hive/HCatalog etc.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3584) Drill Kerberos HDFS Support / Documentation

2015-12-21 Thread Hari Sekhon (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066323#comment-15066323
 ] 

Hari Sekhon commented on DRILL-3584:


Last time I pressed this issue Ted Dunning suggested that I implemented the 
HDFS Kerberos integration myself. If only If I was still on a vendor payroll 
and getting steady salary to spend time on this kind of thing...

> Drill Kerberos HDFS Support / Documentation
> ---
>
> Key: DRILL-3584
> URL: https://issues.apache.org/jira/browse/DRILL-3584
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.1.0
>Reporter: Hari Sekhon
>Priority: Blocker
>
> I'm trying to find Drill docs for Kerberos support for secure HDFS clusters 
> and it doesn't appear to be well tested / supported / documented yet.
> This product is Dead-on-Arrival if it doesn't integrate well with secure 
> Hadoop clusters, specifically HDFS + Kerberos (plus obviously secure 
> kerberized Hive/HCatalog etc.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4214) JDBC connectivity through Cloudera

2015-12-21 Thread Jacques Nadeau (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1503#comment-1503
 ] 

Jacques Nadeau commented on DRILL-4214:
---

In later versions yes.



> JDBC connectivity through Cloudera
> --
>
> Key: DRILL-4214
> URL: https://issues.apache.org/jira/browse/DRILL-4214
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.0.0
> Environment: Drill 1.0.0 Cloudera cdh 5.4.0
>Reporter: Jinith Joseph
>  Labels: newbie, test
>
> Drill JDBC  Java API works fine when executing from IDE Development and the 
> cosole. But when attempting to run using Job Designer in Cloudera, its giving 
> error as below :
> Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], 
> main() threw exception, java.lang.NoSuchMethodError: 
> com.google.common.base.Stopwatch.elapsed(Ljava/util/concurrent/TimeUnit;)J
> org.apache.oozie.action.hadoop.JavaMainException: 
> java.lang.NoSuchMethodError: 
> com.google.common.base.Stopwatch.elapsed(Ljava/util/concurrent/TimeUnit;)J
>   at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:60)
>   at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:46)
>   at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:38)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoSuchMethodError: 
> com.google.common.base.Stopwatch.elapsed(Ljava/util/concurrent/TimeUnit;)J
>   at 
> org.apache.drill.common.util.PathScanner.scanForImplementations(PathScanner.java:98)
>   at 
> org.apache.drill.common.util.PathScanner.scanForImplementationsArr(PathScanner.java:77)
>   at 
> org.apache.drill.common.logical.data.LogicalOperatorBase.getSubTypes(LogicalOperatorBase.java:88)
>   at 
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:77)
>   at 
> org.apache.drill.common.config.DrillConfig.create(DrillConfig.java:169)
>   at 
> org.apache.drill.common.config.DrillConfig.create(DrillConfig.java:142)
>   at 
> org.apache.drill.common.config.DrillConfig.create(DrillConfig.java:97)
>   at 
> org.apache.drill.exec.metrics.DrillMetrics.(DrillMetrics.java:40)
>   at 
> io.netty.buffer.PooledByteBufAllocatorL.(PooledByteBufAllocatorL.java:45)
>   at 
> io.netty.buffer.PooledByteBufAllocatorL.(PooledByteBufAllocatorL.java:43)
>   at 
> org.apache.drill.exec.memory.TopLevelAllocator.(TopLevelAllocator.java:48)
>   at 
> org.apache.drill.exec.memory.TopLevelAllocator.(TopLevelAllocator.java:74)
>   at 
> org.apache.drill.jdbc.DrillConnectionImpl.(DrillConnectionImpl.java:120)
>   at 
> org.apache.drill.jdbc.DrillJdbc41Factory$DrillJdbc41Connection.(DrillJdbc41Factory.java:97)
>   at 
> org.apache.drill.jdbc.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:60)
>   at 
> org.apache.drill.jdbc.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:46)
>   at 
> org.apache.drill.jdbc.DrillFactory.newConnection(DrillFactory.java:54)
>   at 
> net.hydromatic.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:126)
>   at java.sql.DriverManager.getConnection(DriverManager.java:571)
>   at java.sql.DriverManager.getConnection(DriverManager.java:215)
>   at 
> com.alshaya.drill.Drill_Jdbc_Test.GetLastModifiedDate(Drill_Jdbc_Test.java:29)
>   at

[jira] [Commented] (DRILL-3584) Drill Kerberos HDFS Support / Documentation

2015-12-21 Thread Jacques Nadeau (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066531#comment-15066531
 ] 

Jacques Nadeau commented on DRILL-3584:
---

Hold your breath. I'm going to take this on. I thought this was a configuration 
issue but it sure sounds like something is missing. 

> Drill Kerberos HDFS Support / Documentation
> ---
>
> Key: DRILL-3584
> URL: https://issues.apache.org/jira/browse/DRILL-3584
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.1.0
>Reporter: Hari Sekhon
>Priority: Blocker
>
> I'm trying to find Drill docs for Kerberos support for secure HDFS clusters 
> and it doesn't appear to be well tested / supported / documented yet.
> This product is Dead-on-Arrival if it doesn't integrate well with secure 
> Hadoop clusters, specifically HDFS + Kerberos (plus obviously secure 
> kerberized Hive/HCatalog etc.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4214) JDBC connectivity through Cloudera

2015-12-21 Thread Jacques Nadeau (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066540#comment-15066540
 ] 

Jacques Nadeau commented on DRILL-4214:
---

Can you try with a newer version. Preferably 1.4. Some of the earlier versions 
of Drill had jdbc driver packaging issues. 

> JDBC connectivity through Cloudera
> --
>
> Key: DRILL-4214
> URL: https://issues.apache.org/jira/browse/DRILL-4214
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.0.0
> Environment: Drill 1.0.0 Cloudera cdh 5.4.0
>Reporter: Jinith Joseph
>  Labels: newbie, test
>
> Drill JDBC  Java API works fine when executing from IDE Development and the 
> cosole. But when attempting to run using Job Designer in Cloudera, its giving 
> error as below :
> Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], 
> main() threw exception, java.lang.NoSuchMethodError: 
> com.google.common.base.Stopwatch.elapsed(Ljava/util/concurrent/TimeUnit;)J
> org.apache.oozie.action.hadoop.JavaMainException: 
> java.lang.NoSuchMethodError: 
> com.google.common.base.Stopwatch.elapsed(Ljava/util/concurrent/TimeUnit;)J
>   at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:60)
>   at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:46)
>   at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:38)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoSuchMethodError: 
> com.google.common.base.Stopwatch.elapsed(Ljava/util/concurrent/TimeUnit;)J
>   at 
> org.apache.drill.common.util.PathScanner.scanForImplementations(PathScanner.java:98)
>   at 
> org.apache.drill.common.util.PathScanner.scanForImplementationsArr(PathScanner.java:77)
>   at 
> org.apache.drill.common.logical.data.LogicalOperatorBase.getSubTypes(LogicalOperatorBase.java:88)
>   at 
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:77)
>   at 
> org.apache.drill.common.config.DrillConfig.create(DrillConfig.java:169)
>   at 
> org.apache.drill.common.config.DrillConfig.create(DrillConfig.java:142)
>   at 
> org.apache.drill.common.config.DrillConfig.create(DrillConfig.java:97)
>   at 
> org.apache.drill.exec.metrics.DrillMetrics.(DrillMetrics.java:40)
>   at 
> io.netty.buffer.PooledByteBufAllocatorL.(PooledByteBufAllocatorL.java:45)
>   at 
> io.netty.buffer.PooledByteBufAllocatorL.(PooledByteBufAllocatorL.java:43)
>   at 
> org.apache.drill.exec.memory.TopLevelAllocator.(TopLevelAllocator.java:48)
>   at 
> org.apache.drill.exec.memory.TopLevelAllocator.(TopLevelAllocator.java:74)
>   at 
> org.apache.drill.jdbc.DrillConnectionImpl.(DrillConnectionImpl.java:120)
>   at 
> org.apache.drill.jdbc.DrillJdbc41Factory$DrillJdbc41Connection.(DrillJdbc41Factory.java:97)
>   at 
> org.apache.drill.jdbc.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:60)
>   at 
> org.apache.drill.jdbc.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:46)
>   at 
> org.apache.drill.jdbc.DrillFactory.newConnection(DrillFactory.java:54)
>   at 
> net.hydromatic.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:126)
>   at java.sql.DriverManager.getConnection(DriverManager.java:571)
>   at java.sql.DriverManager.getConnection(DriverManager.java:215)
>

[jira] [Commented] (DRILL-4214) JDBC connectivity through Cloudera

2015-12-21 Thread Yuliya Feldman (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066611#comment-15066611
 ] 

Yuliya Feldman commented on DRILL-4214:
---

How would packaging issues of Drill JDBC would affect it? Do we do shading?
(Not that this that much important)
  From: Jacques Nadeau (JIRA) 
 To: yufeld...@yahoo.com 
 Sent: Monday, December 21, 2015 7:13 AM
 Subject: [jira] [Commented] (DRILL-4214) JDBC connectivity through Cloudera
   

    [ 
https://issues.apache.org/jira/browse/DRILL-4214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066540#comment-15066540
 ] 

Jacques Nadeau commented on DRILL-4214:
---

Can you try with a newer version. Preferably 1.4. Some of the earlier versions 
of Drill had jdbc driver packaging issues. 




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)




> JDBC connectivity through Cloudera
> --
>
> Key: DRILL-4214
> URL: https://issues.apache.org/jira/browse/DRILL-4214
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.0.0
> Environment: Drill 1.0.0 Cloudera cdh 5.4.0
>Reporter: Jinith Joseph
>  Labels: newbie, test
>
> Drill JDBC  Java API works fine when executing from IDE Development and the 
> cosole. But when attempting to run using Job Designer in Cloudera, its giving 
> error as below :
> Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], 
> main() threw exception, java.lang.NoSuchMethodError: 
> com.google.common.base.Stopwatch.elapsed(Ljava/util/concurrent/TimeUnit;)J
> org.apache.oozie.action.hadoop.JavaMainException: 
> java.lang.NoSuchMethodError: 
> com.google.common.base.Stopwatch.elapsed(Ljava/util/concurrent/TimeUnit;)J
>   at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:60)
>   at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:46)
>   at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:38)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoSuchMethodError: 
> com.google.common.base.Stopwatch.elapsed(Ljava/util/concurrent/TimeUnit;)J
>   at 
> org.apache.drill.common.util.PathScanner.scanForImplementations(PathScanner.java:98)
>   at 
> org.apache.drill.common.util.PathScanner.scanForImplementationsArr(PathScanner.java:77)
>   at 
> org.apache.drill.common.logical.data.LogicalOperatorBase.getSubTypes(LogicalOperatorBase.java:88)
>   at 
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:77)
>   at 
> org.apache.drill.common.config.DrillConfig.create(DrillConfig.java:169)
>   at 
> org.apache.drill.common.config.DrillConfig.create(DrillConfig.java:142)
>   at 
> org.apache.drill.common.config.DrillConfig.create(DrillConfig.java:97)
>   at 
> org.apache.drill.exec.metrics.DrillMetrics.(DrillMetrics.java:40)
>   at 
> io.netty.buffer.PooledByteBufAllocatorL.(PooledByteBufAllocatorL.java:45)
>   at 
> io.netty.buffer.PooledByteBufAllocatorL.(PooledByteBufAllocatorL.java:43)
>   at 
> org.apache.drill.exec.memory.TopLevelAllocator.(TopLevelAllocator.java:48)
>   at 
> org.apache.drill.exec.memory.TopLevelAllocator.(TopLevelAllocator.java:74)
>   at 
> org.apache.drill.jdbc.DrillConnectionImpl.(DrillConnectionImpl.java:120)
>   at 
>

[jira] [Assigned] (DRILL-3584) Drill Kerberos HDFS Support / Documentation

2015-12-21 Thread Jacques Nadeau (JIRA)


 [ 
https://issues.apache.org/jira/browse/DRILL-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jacques Nadeau reassigned DRILL-3584:
-

Assignee: Jacques Nadeau

> Drill Kerberos HDFS Support / Documentation
> ---
>
> Key: DRILL-3584
> URL: https://issues.apache.org/jira/browse/DRILL-3584
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.1.0
>Reporter: Hari Sekhon
>Assignee: Jacques Nadeau
>Priority: Blocker
>
> I'm trying to find Drill docs for Kerberos support for secure HDFS clusters 
> and it doesn't appear to be well tested / supported / documented yet.
> This product is Dead-on-Arrival if it doesn't integrate well with secure 
> Hadoop clusters, specifically HDFS + Kerberos (plus obviously secure 
> kerberized Hive/HCatalog etc.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4214) JDBC connectivity through Cloudera

2015-12-21 Thread Jacques Nadeau (JIRA)


[ 
https://issues.apache.org/jira/browse/DRILL-4214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15066782#comment-15066782
 ] 

Jacques Nadeau commented on DRILL-4214:
---

Just to clarify. The Drill JDBC driver should be fully shaded in version 1.4. 
This should allow it to coexist in most any environment.

> JDBC connectivity through Cloudera
> --
>
> Key: DRILL-4214
> URL: https://issues.apache.org/jira/browse/DRILL-4214
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.0.0
> Environment: Drill 1.0.0 Cloudera cdh 5.4.0
>Reporter: Jinith Joseph
>  Labels: newbie, test
>
> Drill JDBC  Java API works fine when executing from IDE Development and the 
> cosole. But when attempting to run using Job Designer in Cloudera, its giving 
> error as below :
> Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], 
> main() threw exception, java.lang.NoSuchMethodError: 
> com.google.common.base.Stopwatch.elapsed(Ljava/util/concurrent/TimeUnit;)J
> org.apache.oozie.action.hadoop.JavaMainException: 
> java.lang.NoSuchMethodError: 
> com.google.common.base.Stopwatch.elapsed(Ljava/util/concurrent/TimeUnit;)J
>   at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:60)
>   at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:46)
>   at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:38)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoSuchMethodError: 
> com.google.common.base.Stopwatch.elapsed(Ljava/util/concurrent/TimeUnit;)J
>   at 
> org.apache.drill.common.util.PathScanner.scanForImplementations(PathScanner.java:98)
>   at 
> org.apache.drill.common.util.PathScanner.scanForImplementationsArr(PathScanner.java:77)
>   at 
> org.apache.drill.common.logical.data.LogicalOperatorBase.getSubTypes(LogicalOperatorBase.java:88)
>   at 
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:77)
>   at 
> org.apache.drill.common.config.DrillConfig.create(DrillConfig.java:169)
>   at 
> org.apache.drill.common.config.DrillConfig.create(DrillConfig.java:142)
>   at 
> org.apache.drill.common.config.DrillConfig.create(DrillConfig.java:97)
>   at 
> org.apache.drill.exec.metrics.DrillMetrics.(DrillMetrics.java:40)
>   at 
> io.netty.buffer.PooledByteBufAllocatorL.(PooledByteBufAllocatorL.java:45)
>   at 
> io.netty.buffer.PooledByteBufAllocatorL.(PooledByteBufAllocatorL.java:43)
>   at 
> org.apache.drill.exec.memory.TopLevelAllocator.(TopLevelAllocator.java:48)
>   at 
> org.apache.drill.exec.memory.TopLevelAllocator.(TopLevelAllocator.java:74)
>   at 
> org.apache.drill.jdbc.DrillConnectionImpl.(DrillConnectionImpl.java:120)
>   at 
> org.apache.drill.jdbc.DrillJdbc41Factory$DrillJdbc41Connection.(DrillJdbc41Factory.java:97)
>   at 
> org.apache.drill.jdbc.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:60)
>   at 
> org.apache.drill.jdbc.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:46)
>   at 
> org.apache.drill.jdbc.DrillFactory.newConnection(DrillFactory.java:54)
>   at 
> net.hydromatic.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:126)
>   at java.sql.DriverManager.getConnection(DriverManager.java:571)
>   at

[jira] [Updated] (DRILL-4047) Select with options

2015-12-21 Thread Chun Chang (JIRA)


 [ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chun Chang updated DRILL-4047:
--
Reviewer:   (was: Chun Chang)

> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
> Fix For: 1.4.0
>
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a 
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4215) Transfer ownership of buffers when doing transfers

[jira] [Commented] (DRILL-4215) Transfer ownership of buffers when doing transfers

[jira] [Commented] (DRILL-4192) Dir0 and Dir1 from drill-1.4 are messed up

[jira] [Updated] (DRILL-4192) Dir0 and Dir1 from drill-1.4 are messed up

[jira] [Commented] (DRILL-4192) Dir0 and Dir1 from drill-1.4 are messed up

[jira] [Commented] (DRILL-4192) Dir0 and Dir1 from drill-1.4 are messed up

[jira] [Commented] (DRILL-4192) Dir0 and Dir1 from drill-1.4 are messed up

[jira] [Comment Edited] (DRILL-4205) Simple query hit IndexOutOfBoundException

[jira] [Commented] (DRILL-4205) Simple query hit IndexOutOfBoundException

[jira] [Commented] (DRILL-4205) Simple query hit IndexOutOfBoundException

[jira] [Commented] (DRILL-4134) Incorporate remaining patches from DRILL-1942 Allocator refactor

[jira] [Commented] (DRILL-3478) Bson Record Reader for Mongo storage plugin

[jira] [Commented] (DRILL-4192) Dir0 and Dir1 from drill-1.4 are messed up

[jira] [Commented] (DRILL-3584) Drill Kerberos HDFS Support / Documentation

[jira] [Commented] (DRILL-3584) Drill Kerberos HDFS Support / Documentation

[jira] [Created] (DRILL-4216) Aggregate Window Function COUNT() With GROUP BY Clause expected: range(0, 32768)

[jira] [Closed] (DRILL-3765) Partition prune rule is unnecessary fired multiple times.

[jira] [Comment Edited] (DRILL-3584) Drill Kerberos HDFS Support / Documentation

[jira] [Commented] (DRILL-3584) Drill Kerberos HDFS Support / Documentation

[jira] [Commented] (DRILL-3584) Drill Kerberos HDFS Support / Documentation

[jira] [Commented] (DRILL-4214) JDBC connectivity through Cloudera

[jira] [Commented] (DRILL-3584) Drill Kerberos HDFS Support / Documentation

[jira] [Commented] (DRILL-4214) JDBC connectivity through Cloudera

[jira] [Commented] (DRILL-4214) JDBC connectivity through Cloudera

[jira] [Assigned] (DRILL-3584) Drill Kerberos HDFS Support / Documentation

[jira] [Commented] (DRILL-4214) JDBC connectivity through Cloudera

[jira] [Updated] (DRILL-4047) Select with options

27 matches

Site Navigation

Mail list logo

Footer information