[GitHub] drill pull request #1171: DRILL-6231: Fix memory allocation for repeated lis...

2018-03-16 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1171#discussion_r175244575 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchSizer.java --- @@ -395,11 +395,24 @@ private void allocateMap

Re: [ANNOUNCE] New Committer: Volodymyr Vysotskyi

2018-03-15 Thread Paul Rogers
Congratulations! Thanks for the great contributions. - Paul On Thursday, March 15, 2018, 10:26:26 AM PDT, Khurram Faraaz wrote: Congratulations Volodymyr! From: Arina Ielchiieva Sent: Thursday, March 15, 2018 10:16:51 AM To: dev@drill.apache.org Sub

[GitHub] drill pull request #1164: DRILL-6234: Improved documentation for VariableWid...

2018-03-13 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1164#discussion_r174328662 --- Diff: exec/vector/src/test/java/org/apache/drill/exec/vector/VariableLengthVectorTest.java --- @@ -0,0 +1,152 @@ +/** + * Licensed to the

[GitHub] drill pull request #1164: DRILL-6234: Improved documentation for VariableWid...

2018-03-13 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1164#discussion_r174327601 --- Diff: exec/vector/src/main/codegen/templates/VariableLengthVectors.java --- @@ -514,6 +516,22 @@ public boolean isNull(int index){ * The

[GitHub] drill pull request #1164: DRILL-6234: Improved documentation for VariableWid...

2018-03-13 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1164#discussion_r174327027 --- Diff: exec/vector/src/main/codegen/templates/VariableLengthVectors.java --- @@ -506,6 +506,8 @@ public boolean isNull(int index

[GitHub] drill pull request #1164: DRILL-6234: Improved documentation for VariableWid...

2018-03-13 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1164#discussion_r174328025 --- Diff: exec/vector/src/main/codegen/templates/VariableLengthVectors.java --- @@ -514,6 +516,22 @@ public boolean isNull(int index){ * The

[GitHub] drill pull request #1164: DRILL-6234: Improved documentation for VariableWid...

2018-03-13 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1164#discussion_r174327182 --- Diff: exec/vector/src/main/codegen/templates/VariableLengthVectors.java --- @@ -514,6 +516,22 @@ public boolean isNull(int index){ * The

[GitHub] drill pull request #1164: DRILL-6234: Improved documentation for VariableWid...

2018-03-13 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1164#discussion_r174327511 --- Diff: exec/vector/src/main/codegen/templates/VariableLengthVectors.java --- @@ -514,6 +516,22 @@ public boolean isNull(int index){ * The

[GitHub] drill pull request #1164: DRILL-6234: Improved documentation for VariableWid...

2018-03-13 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1164#discussion_r174327132 --- Diff: exec/vector/src/main/codegen/templates/VariableLengthVectors.java --- @@ -514,6 +516,22 @@ public boolean isNull(int index){ * The

[GitHub] drill pull request #1164: DRILL-6234: Improved documentation for VariableWid...

2018-03-13 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1164#discussion_r174328361 --- Diff: exec/vector/src/test/java/org/apache/drill/exec/vector/VariableLengthVectorTest.java --- @@ -0,0 +1,152 @@ +/** + * Licensed to the

[GitHub] drill issue #1161: DRILL-6230: Extend row set readers to handle hyper vector...

2018-03-11 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1161 @ppadma, I believe you are working in this area, can you please take a look at your convenience? Thanks. ---

[GitHub] drill pull request #1161: DRILL-6230: Extend row set readers to handle hyper...

2018-03-11 Thread paul-rogers
GitHub user paul-rogers opened a pull request: https://github.com/apache/drill/pull/1161 DRILL-6230: Extend row set readers to handle hyper vectors The current row set readers have incomplete support for hyper-vectors. To add full support, we need an interface that supports either

[jira] [Created] (DRILL-6230) Extend row set readers to handle hyper vectors

2018-03-10 Thread Paul Rogers (JIRA)
Paul Rogers created DRILL-6230: -- Summary: Extend row set readers to handle hyper vectors Key: DRILL-6230 URL: https://issues.apache.org/jira/browse/DRILL-6230 Project: Apache Drill Issue Type

[GitHub] drill issue #1150: DRILL-6210: Enhanced test schema utilities

2018-03-08 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1150 Rebased on latest master. ---

[GitHub] drill issue #1150: DRILL-6210: Enhanced test schema utilities

2018-03-07 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1150 @Ben-Zvi, the changes look right. Old style was to have a "buildFoo" method for the nested structure "foo". The new style is to have a "resumeBar" method

[GitHub] drill issue #1150: DRILL-6210: Enhanced test schema utilities

2018-03-06 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1150 Fixed the typo. @Ben-Zvi, thanks for pointing out the unit tests: I do have test file in my branch; forgot to include it with the PR. Just did. ---

Re: [1.13 - Release Blocker] - DRILL-6216: Metadata mismatch when connecting to a Drill 1.12.0 with a Drill-1.13.0-SNAPSHOT driver

2018-03-06 Thread Paul Rogers
Hi Sorabh, Thanks for tracking this one down. Our unit tests did not uncover this issue when I did the original PR, unfortunately. The name change was done to be consistent with other places where we use special names and, as I recall, help with certain tasks. Clearly, however, if the client dep

[GitHub] drill issue #1104: DRILL-6118: Handle item star columns during project / fil...

2018-03-06 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1104 I noticed this change when merging with the latest master. What does `DYNAMIC_STAR` mean? The original `WILDCARD` name was more theoretical. How is `DYNAMIC_STAR` different from `STAR` (other

[GitHub] drill issue #1138: DRILL-4120: Allow implicit columns for Avro storage forma...

2018-03-06 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1138 @vvysotskyi, thanks for addressing the schema issues! ---

Non-batch review comments?

2018-03-04 Thread Paul Rogers
Hi All, Two years ago, when I started with Drill, Github would send out an e-mail for each review comment. Later, GitHub thankfully implemented a feature that combined comments into a single e-mail. But, for some time now, GitHub is back to spamming this list with each individual comment. Anyon

[GitHub] drill pull request #1125: DRILL-6126: Allocate memory for value vectors upfr...

2018-03-04 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1125#discussion_r172105268 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchSizer.java --- @@ -525,4 +763,11 @@ public VectorInitializer

[GitHub] drill pull request #1125: DRILL-6126: Allocate memory for value vectors upfr...

2018-03-04 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1125#discussion_r172105917 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchSizer.java --- @@ -199,12 +422,18 @@ public String toString

[GitHub] drill pull request #1125: DRILL-6126: Allocate memory for value vectors upfr...

2018-03-04 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1125#discussion_r172104808 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchSizer.java --- @@ -76,110 +82,327 @@ * greater than (but

[GitHub] drill pull request #1125: DRILL-6126: Allocate memory for value vectors upfr...

2018-03-04 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1125#discussion_r172105435 --- Diff: exec/vector/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMapVector.java --- @@ -608,4 +608,22 @@ public void collectLedgers(Set

[GitHub] drill pull request #1125: DRILL-6126: Allocate memory for value vectors upfr...

2018-03-04 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1125#discussion_r172104027 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchSizer.java --- @@ -36,13 +39,17 @@ import

[GitHub] drill pull request #1125: DRILL-6126: Allocate memory for value vectors upfr...

2018-03-04 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1125#discussion_r172105104 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchSizer.java --- @@ -76,110 +82,327 @@ * greater than (but

[GitHub] drill pull request #1125: DRILL-6126: Allocate memory for value vectors upfr...

2018-03-04 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1125#discussion_r172103839 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/MergeJoinBatch.java --- @@ -492,8 +504,12 @@ private void allocateBatch

[GitHub] drill pull request #1125: DRILL-6126: Allocate memory for value vectors upfr...

2018-03-04 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1125#discussion_r172104598 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchSizer.java --- @@ -76,110 +82,327 @@ * greater than (but

[GitHub] drill pull request #1125: DRILL-6126: Allocate memory for value vectors upfr...

2018-03-04 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1125#discussion_r172104395 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchSizer.java --- @@ -76,110 +82,327 @@ * greater than (but

[GitHub] drill issue #1150: DRILL-6210: Enhanced test schema utilities

2018-03-04 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1150 @padma, can you review this? @ilooner, you might want to take a look also since you've taken an interest in improving our test structures. ---

[GitHub] drill pull request #1150: DRILL-6210: Enhanced test schema utilities

2018-03-04 Thread paul-rogers
GitHub user paul-rogers opened a pull request: https://github.com/apache/drill/pull/1150 DRILL-6210: Enhanced test schema utilities The result set loader project enhanced the schema builder used in tests to handle Drill's complex types: maps, unions, lists and repeated lists

[jira] [Created] (DRILL-6210) Enhance the test schema builder for remaining types

2018-03-04 Thread Paul Rogers (JIRA)
Paul Rogers created DRILL-6210: -- Summary: Enhance the test schema builder for remaining types Key: DRILL-6210 URL: https://issues.apache.org/jira/browse/DRILL-6210 Project: Apache Drill Issue

Re: Batch Sizing for Parquet Flat Reader

2018-03-04 Thread Paul Rogers
need the result set loader for the Parquet reader that reads nested types. Thanks, - Paul On Sunday, March 4, 2018, 1:07:38 PM PST, Paul Rogers wrote: Hi Aman, Please see my comment in DRILL-6147. For the hangout to be productive, perhaps we should create test cases that will show

Re: Batch Sizing for Parquet Flat Reader

2018-03-04 Thread Paul Rogers
imposed. > > Thanks > Padma > > > On Feb 12, 2018, at 8:25 AM, Paul Rogers mailto:par0...@yahoo.com.INVALID>> wrote: > > Agreed that allocating vectors up front is another good improvement. > The average batch size approach gets us 80% of the way to the goal: i

[GitHub] drill issue #1011: Drill 1170: Drill-on-YARN

2018-03-04 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1011 @arina-ielchiieva, thanks much for your help with this PR. Glad to see it is finally in Drill master after all this time! ---

[GitHub] drill issue #1011: Drill 1170: Drill-on-YARN

2018-03-04 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1011 @kr-arjun, thanks for the text file. The error is related to security. DoY, in its current for, is an "MVP": it works, but leaves off advanced features. One of those missing features

Re: [DISCUSS] 1.13.0 release

2018-03-03 Thread Paul Rogers
Hi Parth, Issues with DRILL-1170 are resolved. Needs one final review by Arina, then we should be good to do. Thanks to everyone for getting the other two "batch size" PRs committed recently. Thanks, - Paul On Saturday, March 3, 2018, 10:57:10 PM PST, Parth Chandra wrote: Thank you

[GitHub] drill issue #1011: Drill 1170: Drill-on-YARN

2018-03-03 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1011 @arina-ielchiieva, turned out that there were unneeded dependencies in the DoY additions to the drill-root pom.xml file. Removed these and the json.org warnings went away. Please take a

[GitHub] drill issue #1011: Drill 1170: Drill-on-YARN

2018-03-03 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1011 Rebased onto latest master. ---

[GitHub] drill pull request #1011: Drill 1170: Drill-on-YARN

2018-03-03 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1011#discussion_r172039445 --- Diff: distribution/src/assemble/bin.xml --- @@ -323,6 +333,21 @@ src/resources/sqlline.bat bin + --- End

[GitHub] drill issue #1011: Drill 1170: Drill-on-YARN

2018-03-03 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1011 @kr-arjun, thanks for your note on error handling. Where you using the `start` command? There is exactly one place where the error "Failed to start Drill application master" is thr

[GitHub] drill issue #1138: DRILL-4120: Allow implicit columns for Avro storage forma...

2018-03-01 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1138 Another thought. The removed code is at plan time. Did the original code have to open each file to retrieve schema? If so, does removing the code remove that load? If so, then this change could

[GitHub] drill issue #1138: DRILL-4120: Allow implicit columns for Avro storage forma...

2018-03-01 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1138 As @arina-ielchiieva points out, this change backs out plan-time knowledge of schema. This may not affect run-time accuracy. However, it does mean that queries can be planned, based on not

[GitHub] drill pull request #1138: DRILL-4120: Allow implicit columns for Avro storag...

2018-03-01 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1138#discussion_r171606330 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/avro/AvroRecordReader.java --- @@ -154,6 +156,12 @@ public int next

[GitHub] drill pull request #1138: DRILL-4120: Allow implicit columns for Avro storag...

2018-03-01 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1138#discussion_r171607241 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/avro/AvroRecordReader.java --- @@ -295,7 +301,8 @@ private void processPrimitive

Re: [ANNOUNCE] New Committer: Kunal Khatua

2018-02-28 Thread Paul Rogers
Congrats, Kunal! Well deserved. - Paul > On Feb 27, 2018, at 10:42 AM, Prasad Nagaraj Subramanya > wrote: > > Congratulations Kunal! > > > On Tue, Feb 27, 2018 at 10:41 AM, Padma Penumarthy > wrote: > >> Congratulations Kunal ! >> >> Thanks >> Padma >> >> >>> On Feb 27, 2018, at 8:42 A

[GitHub] drill issue #1112: DRILL-6114: Metadata revisions

2018-02-28 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1112 @arina-ielchiieva, @parthchandra can either of you perhaps give this one a committer review? Thanks! ---

[GitHub] drill pull request #1135: DRILL-6040: Added usage for graceful_stop in drill...

2018-02-28 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1135#discussion_r171319850 --- Diff: distribution/src/resources/drillbit.sh --- @@ -45,7 +45,7 @@ # configuration file. The option takes precedence over the

[GitHub] drill issue #1138: DRILL-4120: Allow implicit columns for Avro storage forma...

2018-02-28 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1138 General comment: if we could move to the new scan framework; it handles implicit columns for all file-based readers. It also handles projection, missing columns, etc... ---

[GitHub] drill pull request #1129: DRILL-6180: Use System Option "output_batch_size" ...

2018-02-28 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1129#discussion_r171314892 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/managed/SortConfig.java --- @@ -71,8 +72,8 @@ private

[GitHub] drill pull request #1129: DRILL-6180: Use System Option "output_batch_size" ...

2018-02-28 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1129#discussion_r171315310 --- Diff: exec/java-exec/src/main/resources/drill-module.conf --- @@ -421,7 +416,7 @@ drill.exec.options

[GitHub] drill issue #1011: Drill 1170: Drill-on-YARN

2018-02-27 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1011 @arina-ielchiieva, do you want to give this one a committer +1? Then I'll mark it ready-to-commit. Thanks! ---

[GitHub] drill issue #1121: DRILL-6153: Operator framework

2018-02-27 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1121 @arina-ielchiieva, can you do a committer review of this one? ---

[GitHub] drill issue #1011: Drill 1170: Drill-on-YARN

2018-02-27 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1011 Rebased onto latest master. ---

[GitHub] drill issue #1011: Drill 1170: Drill-on-YARN

2018-02-27 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1011 Fixed the drill-common dependency as @ilooner requested. ---

[GitHub] drill pull request #1125: DRILL-6126: Allocate memory for value vectors upfr...

2018-02-21 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1125#discussion_r169859674 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchSizer.java --- @@ -245,16 +251,30 @@ private void buildVectorInitializer

[GitHub] drill pull request #1125: DRILL-6126: Allocate memory for value vectors upfr...

2018-02-21 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1125#discussion_r169860846 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/flatten/FlattenRecordBatch.java --- @@ -266,12 +270,18 @@ public void

[GitHub] drill pull request #1125: DRILL-6126: Allocate memory for value vectors upfr...

2018-02-21 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1125#discussion_r169857960 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchSizer.java --- @@ -418,11 +438,13 @@ private void measureColumn

[GitHub] drill pull request #1125: DRILL-6126: Allocate memory for value vectors upfr...

2018-02-21 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1125#discussion_r169860310 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchSizer.java --- @@ -245,16 +251,30 @@ private void buildVectorInitializer

[GitHub] drill pull request #1125: DRILL-6126: Allocate memory for value vectors upfr...

2018-02-21 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1125#discussion_r169858105 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchSizer.java --- @@ -108,6 +108,12 @@ public final float

[GitHub] drill pull request #1125: DRILL-6126: Allocate memory for value vectors upfr...

2018-02-21 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1125#discussion_r169859837 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorInitializer.java --- @@ -97,7 +97,7 @@ public void allocateBatch

[GitHub] drill pull request #1094: DRILL-6090: While connecting to drill-bits using J...

2018-02-19 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1094#discussion_r169214241 --- Diff: exec/jdbc/src/main/java/org/apache/drill/jdbc/impl/DrillConnectionImpl.java --- @@ -108,73 +108,81 @@ protected DrillConnectionImpl(DriverImpl

[GitHub] drill issue #1011: Drill 1170: Drill-on-YARN

2018-02-19 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1011 Failing in Travis, apparently due to test-framework issue: ``` Caused by: java.lang.ClassNotFoundException: org.apache.drill.categories.SecurityTest ``` @ilooner, any idea

[GitHub] drill issue #1121: DRILL-6153: Operator framework

2018-02-19 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1121 Rebased on latest master. ---

[GitHub] drill pull request #1121: DRILL-6153: Operator framework

2018-02-19 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1121#discussion_r169203505 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/protocol/OperatorDriver.java --- @@ -0,0 +1,222 @@ +/* + * Licensed

[GitHub] drill pull request #1121: DRILL-6153: Operator framework

2018-02-19 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1121#discussion_r169203884 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/protocol/VectorContainerAccessor.java --- @@ -0,0 +1,132

[GitHub] drill pull request #1121: DRILL-6153: Operator framework

2018-02-19 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1121#discussion_r169203498 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/protocol/OperatorDriver.java --- @@ -0,0 +1,222 @@ +/* + * Licensed

[GitHub] drill pull request #1121: DRILL-6153: Operator framework

2018-02-19 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1121#discussion_r169203509 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/protocol/OperatorDriver.java --- @@ -0,0 +1,222 @@ +/* + * Licensed

[GitHub] drill issue #1114: Drill-6104: Added Logfile Reader

2018-02-19 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1114 See [this example](https://github.com/paul-rogers/drill/tree/regex-plugin/exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/regex), and [this test](https://github.com/paul-rogers

[GitHub] drill issue #1011: Drill 1170: Drill-on-YARN

2018-02-19 Thread paul-rogers
Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/1011 Rebased on latest master and resolved merge conflicts. Some ZK-related classes changed. Would be good if Abhishek could do a quick sanity test on his test cluster to make sure things

[jira] [Created] (DRILL-6169) Table functions do not allow list arguments

2018-02-19 Thread Paul Rogers (JIRA)
Paul Rogers created DRILL-6169: -- Summary: Table functions do not allow list arguments Key: DRILL-6169 URL: https://issues.apache.org/jira/browse/DRILL-6169 Project: Apache Drill Issue Type: Bug

[jira] [Created] (DRILL-6168) Table functions do not "inherit" default configuration

2018-02-18 Thread Paul Rogers (JIRA)
Paul Rogers created DRILL-6168: -- Summary: Table functions do not "inherit" default configuration Key: DRILL-6168 URL: https://issues.apache.org/jira/browse/DRILL-6168 Project: Ap

[jira] [Created] (DRILL-6167) Table functions give error without hidden type field

2018-02-18 Thread Paul Rogers (JIRA)
Paul Rogers created DRILL-6167: -- Summary: Table functions give error without hidden type field Key: DRILL-6167 URL: https://issues.apache.org/jira/browse/DRILL-6167 Project: Apache Drill Issue

[GitHub] drill pull request #1112: DRILL-6114: Metadata revisions

2018-02-15 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1112#discussion_r168682152 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/metadata/MetadataUtils.java --- @@ -0,0 +1,165 @@ +/* + * Licensed to the

[GitHub] drill pull request #1112: DRILL-6114: Metadata revisions

2018-02-15 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1112#discussion_r168682470 --- Diff: exec/vector/src/main/java/org/apache/drill/exec/record/metadata/ColumnMetadata.java --- @@ -15,36 +15,115 @@ * See the License for the

[GitHub] drill pull request #1112: DRILL-6114: Metadata revisions

2018-02-15 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1112#discussion_r168682331 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/metadata/RepeatedListColumnMetadata.java --- @@ -0,0 +1,94

[GitHub] drill pull request #1112: DRILL-6114: Metadata revisions

2018-02-15 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1112#discussion_r168681135 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/metadata/AbstractColumnMetadata.java --- @@ -0,0 +1,206 @@ +/* + * Licensed

[GitHub] drill pull request #1112: DRILL-6114: Metadata revisions

2018-02-15 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1112#discussion_r168681901 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/metadata/AbstractColumnMetadata.java --- @@ -0,0 +1,206 @@ +/* + * Licensed

[GitHub] drill pull request #1121: DRILL-6153: Operator framework

2018-02-15 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1121#discussion_r168676813 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/protocol/OperatorDriver.java --- @@ -0,0 +1,183 @@ +/* + * Licensed

[GitHub] drill pull request #1121: DRILL-6153: Operator framework

2018-02-15 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1121#discussion_r168675063 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/protocol/OperatorDriver.java --- @@ -0,0 +1,183 @@ +/* + * Licensed

[GitHub] drill pull request #1121: DRILL-6153: Operator framework

2018-02-15 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1121#discussion_r168677109 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/protocol/OperatorExec.java --- @@ -0,0 +1,127 @@ +/* + * Licensed to

[GitHub] drill pull request #1121: DRILL-6153: Operator framework

2018-02-15 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1121#discussion_r168676906 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/protocol/OperatorDriver.java --- @@ -0,0 +1,183 @@ +/* + * Licensed

[GitHub] drill pull request #1121: DRILL-6153: Operator framework

2018-02-15 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1121#discussion_r168674650 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/protocol/OperatorDriver.java --- @@ -0,0 +1,183 @@ +/* + * Licensed

[GitHub] drill pull request #1121: DRILL-6153: Operator framework

2018-02-15 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1121#discussion_r168676504 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/protocol/OperatorDriver.java --- @@ -0,0 +1,183 @@ +/* + * Licensed

[GitHub] drill pull request #1120: DRILL-6148: TestSortSpillWithException is sometime...

2018-02-13 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1120#discussion_r167929421 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/xsort/TestSortSpillWithException.java --- @@ -59,6 +59,7 @@ public static

[GitHub] drill pull request #1121: DRILL-6153: Operator framework

2018-02-12 Thread paul-rogers
GitHub user paul-rogers opened a pull request: https://github.com/apache/drill/pull/1121 DRILL-6153: Operator framework Includes the core files for the operator framework revision. See [this writeup](https://github.com/paul-rogers/drill/wiki/BH-Operator-Framework) for details

[GitHub] drill pull request #1120: DRILL-6148: TestSortSpillWithException is sometime...

2018-02-12 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1120#discussion_r167766107 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/xsort/TestSortSpillWithException.java --- @@ -59,6 +59,7 @@ public static

[jira] [Created] (DRILL-6153) Revised operator framework

2018-02-12 Thread Paul Rogers (JIRA)
Paul Rogers created DRILL-6153: -- Summary: Revised operator framework Key: DRILL-6153 URL: https://issues.apache.org/jira/browse/DRILL-6153 Project: Apache Drill Issue Type: Improvement

[GitHub] drill pull request #1120: DRILL-6148: TestSortSpillWithException is sometime...

2018-02-12 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1120#discussion_r167743599 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/xsort/TestSortSpillWithException.java --- @@ -59,6 +59,7 @@ public static

Review for PR #1112: Metadata revisions

2018-02-12 Thread Paul Rogers
iews of the existing stuff so it can be used. Thanks, - Paul [1]  https://github.com/paul-rogers/drill/wiki/BH-Code-Intro#pull-request-plan

Re: Batch Sizing for Parquet Flat Reader

2018-02-12 Thread Paul Rogers
nd also improve memory utilization. Thanks Padma > On Feb 11, 2018, at 3:44 PM, Paul Rogers wrote: > > One more thought: >>> 3) Assuming that you go with the average batch size calculation approach, > > The average batch size approach is a quick and dirty approach for no

Re: Batch Sizing for Parquet Flat Reader

2018-02-11 Thread Paul Rogers
umnar processing and b) refactor the Parquet code to user the framework - I agree there might be some duplicate effort but I really believe this will be minimal - DRILL-6147 is not more than one week of research & analysis and one week of implementation Regards, Salim > On Feb 11, 2018, at 1

Re: Batch Sizing for Parquet Flat Reader

2018-02-11 Thread Paul Rogers
Parth notes: Also note that memory allocations by Netty greater than the 16MB chunk sizeare returned to the OS when the memory is free'd. Both this document andthe original document on memory fragmentation state incorrectly that suchmemory is not released back to the OS. A quick thought experim

Re: Batch Sizing for Parquet Flat Reader

2018-02-11 Thread Paul Rogers
One more thought: > > 3) Assuming that you go with the average batch size calculation approach, The average batch size approach is a quick and dirty approach for non-leaf operators that can observe an incoming batch to estimate row width. Because Drill batches are large, the law of large numbers

Re: Batch Sizing for Parquet Flat Reader

2018-02-11 Thread Paul Rogers
Hi All, Perhaps this topic needs just a bit more thought and discussion to avoid working at cross purposes. I've outlined the issues, and a possible path forward, in a comment to DRILL-6147. Quick summary: creating a second batch size implementation just for Parquet will be very difficult once w

[GitHub] drill pull request #1107: DRILL-6123: Limit batch size for Merge Join based ...

2018-02-09 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1107#discussion_r167146312 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/RecordBatchSizer.java --- @@ -311,8 +311,8 @@ public static ColumnSize

[GitHub] drill pull request #1107: DRILL-6123: Limit batch size for Merge Join based ...

2018-02-09 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1107#discussion_r167282219 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractRecordBatchMemoryManager.java --- @@ -0,0 +1,63 @@ +/* + * Licensed

[GitHub] drill pull request #1107: DRILL-6123: Limit batch size for Merge Join based ...

2018-02-09 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1107#discussion_r167284278 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractRecordBatchMemoryManager.java --- @@ -0,0 +1,63 @@ +/* + * Licensed

[GitHub] drill pull request #1107: DRILL-6123: Limit batch size for Merge Join based ...

2018-02-09 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1107#discussion_r167281940 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractRecordBatchMemoryManager.java --- @@ -0,0 +1,63 @@ +/* + * Licensed

[GitHub] drill pull request #1107: DRILL-6123: Limit batch size for Merge Join based ...

2018-02-09 Thread paul-rogers
Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1107#discussion_r167146010 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/RecordBatchSizer.java --- @@ -137,10 +137,10 @@ public ColumnSize

<    2   3   4   5   6   7   8   9   10   11   >