Batch handling project next steps

Hi All, Let's discuss the next step for the "batch handling" project. [1] Thanks to Aman for committing the "hygiene" PR. I'm rebasing the remaining code on the updated master. I'll keep the "RowSetRev3" [2] branch unchanged since [1] has many links to it. Instead I'll create a new branch. It

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user ppadma commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164634551 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -215,6 +206,7 @@ public BatchHolder() {

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164624503 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -215,6 +206,7 @@ public BatchHolder()

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164623893 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/RecordBatchSizer.java --- @@ -129,11 +143,16 @@ public

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164623527 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/RecordBatchSizer.java --- @@ -232,9 +251,8 @@ else if (width > 0) {

Re: Drill Case Sensitivity

Drill is case preserving. In most cases, table names (and path) are case-sensitive (except probably on Windows). The case sensitivity / in-sensitivity of columns depends on the data source. For example, with DFS formats such as parquet and JSON, column names are case insensitive. But with MapR-DB

Drill Case Sensitivity

What is Drill's policy on case sensitivity? Some of the tests assume that Drill is case-insensitive, but how does Drill handle data sources like HBase that are case sensitive? Do we do some validation that there are no potentially conflicting column names like "employee" and "Employee"? Do we

[GitHub] drill pull request #1007: DRILL-5864 Selecting a non-existing field from a M...

Github user HanumathRao closed the pull request at: https://github.com/apache/drill/pull/1007 ---

[jira] [Created] (DRILL-6117) Query End time is not being set

Kunal Khatua created DRILL-6117: --- Summary: Query End time is not being set Key: DRILL-6117 URL: https://issues.apache.org/jira/browse/DRILL-6117 Project: Apache Drill Issue Type: Bug

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164615997 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/agg/TestHashAggrSpill.java --- @@ -53,59 +51,54 @@ @Rule public

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164615743 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/RecordBatchSizer.java --- @@ -232,9 +251,8 @@ else if (width > 0) {

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164615077 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/RecordBatchSizer.java --- @@ -129,11 +143,16 @@ public

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164614812 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTableTemplate.java --- @@ -805,7 +803,12 @@ private IntVector

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164613714 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -956,21 +925,8 @@ private void

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164612920 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -516,43 +501,48 @@ private void

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164612280 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -84,13 +85,6 @@ public abstract class

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164611358 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -956,21 +925,8 @@ private void

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164600675 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTableTemplate.java --- @@ -805,7 +803,12 @@ private IntVector

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164609742 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -84,13 +85,6 @@ public abstract class

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164571408 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/RecordBatchSizer.java --- @@ -129,11 +143,16 @@ public

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164607265 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -516,43 +501,48 @@ private void

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164604859 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -397,11 +384,9 @@ private void

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164549701 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggBatch.java --- @@ -255,7 +254,6 @@ private HashAggregator

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164576364 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/agg/TestHashAggrSpill.java --- @@ -53,59 +51,54 @@ @Rule public

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164609183 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -516,43 +501,48 @@ private void

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164574746 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/RecordBatchSizer.java --- @@ -232,9 +251,8 @@ else if (width > 0) {

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164611155 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -215,6 +206,7 @@ public BatchHolder() {

[GitHub] drill pull request #1060: DRILL-5846: Improve parquet performance for Flat D...

Github user vrozov commented on a diff in the pull request: https://github.com/apache/drill/pull/1060#discussion_r164600674 --- Diff: exec/memory/base/src/main/java/org/apache/drill/exec/util/MemoryUtils.java --- @@ -0,0 +1,186 @@ +/** + * Licensed to the Apache Software

[GitHub] drill pull request #1101: DRILL-6032: Made the batch sizing for HashAgg more...

Github user ppadma commented on a diff in the pull request: https://github.com/apache/drill/pull/1101#discussion_r164599681 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java --- @@ -215,6 +206,7 @@ public BatchHolder() {

[GitHub] drill pull request #1096: DRILL-6099 : Push limit past flatten(project) with...

Github user gparai commented on a diff in the pull request: https://github.com/apache/drill/pull/1096#discussion_r164593749 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillPushLimitToScanRule.java --- @@ -121,4 +132,50 @@ protected void

[GitHub] drill pull request #1096: DRILL-6099 : Push limit past flatten(project) with...

Github user chunhui-shi commented on a diff in the pull request: https://github.com/apache/drill/pull/1096#discussion_r164585670 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillPushLimitToScanRule.java --- @@ -121,4 +132,50 @@ protected void

[GitHub] drill issue #897: Drill-5703 Added Syntax Highlighting and Limited Autocompl...

Github user kkhatua commented on the issue: https://github.com/apache/drill/pull/897 @cgivre can we close this PR if the #1043 met the objectives? ---

[GitHub] drill issue #1099: DRILL-6106: Use valueOf method instead of constructor sin...

Github user vrozov commented on the issue: https://github.com/apache/drill/pull/1099 Travis CI failed due to a known issue `TestSortImpl.testLargeBatch:502->runJumboBatchTest:475->runLargeSortTest:454 Value of 1:0 expected:<0> but was:<1>` unrelated to the SSLConfigClient.java

[GitHub] drill issue #1099: DRILL-6106: Use valueOf method instead of constructor sin...

Github user reudismam commented on the issue: https://github.com/apache/drill/pull/1099 I reverted the edit because the code does not pass in Travis CI. ---

[GitHub] drill issue #1099: DRILL-6106: Use valueOf method instead of constructor sin...

Github user vrozov commented on the issue: https://github.com/apache/drill/pull/1099 What was the reason to revert the change in SSLConfigClient.java? ---

[jira] [Created] (DRILL-6116) MAXDIR - Unsupported operation exception // partition explorer interface?

John Humphreys created DRILL-6116: - Summary: MAXDIR - Unsupported operation exception // partition explorer interface? Key: DRILL-6116 URL: https://issues.apache.org/jira/browse/DRILL-6116 Project:

[jira] [Created] (DRILL-6115) SingleMergeExchange is not scaling up when many minor fragments are allocated for a query.

Hanumath Rao Maduri created DRILL-6115: -- Summary: SingleMergeExchange is not scaling up when many minor fragments are allocated for a query. Key: DRILL-6115 URL:

[GitHub] drill pull request #1060: DRILL-5846: Improve parquet performance for Flat D...

Github user sachouche commented on a diff in the pull request: https://github.com/apache/drill/pull/1060#discussion_r164532290 --- Diff: exec/memory/base/src/main/java/io/netty/buffer/DrillBuf.java --- @@ -703,7 +703,18 @@ protected void _setLong(int index, long value) {

[GitHub] drill pull request #1060: DRILL-5846: Improve parquet performance for Flat D...

Github user sachouche commented on a diff in the pull request: https://github.com/apache/drill/pull/1060#discussion_r164525752 --- Diff: exec/memory/base/src/main/java/org/apache/drill/exec/util/MemoryUtils.java --- @@ -0,0 +1,186 @@ +/** + * Licensed to the Apache

[GitHub] drill pull request #1060: DRILL-5846: Improve parquet performance for Flat D...

Github user vrozov commented on a diff in the pull request: https://github.com/apache/drill/pull/1060#discussion_r164520611 --- Diff: exec/memory/base/src/main/java/io/netty/buffer/DrillBuf.java --- @@ -703,7 +703,18 @@ protected void _setLong(int index, long value) {

[GitHub] drill pull request #1060: DRILL-5846: Improve parquet performance for Flat D...

Github user vrozov commented on a diff in the pull request: https://github.com/apache/drill/pull/1060#discussion_r164519147 --- Diff: exec/memory/base/src/main/java/org/apache/drill/exec/util/MemoryUtils.java --- @@ -0,0 +1,186 @@ +/** + * Licensed to the Apache Software

[GitHub] drill pull request #1060: DRILL-5846: Improve parquet performance for Flat D...

Github user sachouche commented on a diff in the pull request: https://github.com/apache/drill/pull/1060#discussion_r164509658 --- Diff: exec/memory/base/src/main/java/org/apache/drill/exec/util/MemoryUtils.java --- @@ -0,0 +1,186 @@ +/** + * Licensed to the Apache

[GitHub] drill pull request #1060: DRILL-5846: Improve parquet performance for Flat D...

Github user vrozov commented on a diff in the pull request: https://github.com/apache/drill/pull/1060#discussion_r164506523 --- Diff: exec/memory/base/src/main/java/org/apache/drill/exec/util/MemoryUtils.java --- @@ -0,0 +1,186 @@ +/** + * Licensed to the Apache Software

[GitHub] drill issue #1099: DRILL-6106: Use valueOf method instead of constructor sin...

Github user arina-ielchiieva commented on the issue: https://github.com/apache/drill/pull/1099 @reudismam you can leave `valueOf` for Double and Float for consistency. There are two other comments @vrozov has left. Could you please address hem as well? They are not directly

[GitHub] drill issue #1099: DRILL-6106: Use valueOf method instead of constructor sin...

Github user reudismam commented on the issue: https://github.com/apache/drill/pull/1099 Conflicts resolved. Do you want to revert the edits to Double and Float? I think it is ok for consistency. ---

Re: LATERAL and UNNEST support for Drill

Hi Ted, Thanks for you question. Array type aggregator is not planned along with this project. But probably after this is done we can look into it. Thanks, Sorabh Get Outlook for iOS From: Ted Dunning Sent: Sunday,

[GitHub] drill issue #1090: DRILL-6080: Sort incorrectly limits batch size to 65535 r...

Github user vrozov commented on the issue: https://github.com/apache/drill/pull/1090 +1 ---

[GitHub] drill issue #1099: DRILL-6106: Use valueOf method instead of constructor sin...

Github user arina-ielchiieva commented on the issue: https://github.com/apache/drill/pull/1099 @reudismam could you please address code review comments so changes can be pushed into master? Also please resolve merge conflict. ---

[GitHub] drill issue #1094: DRILL-6090: While connecting to drill-bits using JDBC Dri...

Github user arina-ielchiieva commented on the issue: https://github.com/apache/drill/pull/1094 @milindt could you please address code review comments so changes can be pushed into master? ---