I think we need to include DRILL-4467 <https://issues.apache.org/jira/browse/DRILL-4467>. I think it is a one line patch and it provides unpredictable plans at a minimum but may also present invalid result. Still need to think through the second half. I've seen this plan instability in some of my recent test runs (even without Java 8) when running extended HBase tests.
-- Jacques Nadeau CTO and Co-Founder, Dremio On Thu, Mar 3, 2016 at 10:02 PM, Parth Chandra <[email protected]> wrote: > Updated list (I'll follow up with the folks named here separately) - > > Committed for 1.6 - > > DRILL-4384 - Query profile is missing important information on WebUi - > Merged > DRILL-3488/pr 388 (Java 1.8 support) - Merged. > DRILL-4410/pr 380 (listvector should initiatlize bits...) - Merged > DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos, etc) - Merged > DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration) - > Waiting to be merged > DRILL-4437 (and others)/pr 394 (Operator unit test framework). Waiting to > be merged. > > DRILL-4281/pr 400 (Drill should support inbound impersonation) (Jacques to > review) > DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly expose > their types within Calcite.) - Waiting for Aman to review. (Owners: Hsuan, > Jinfeng, Aman, Sudheesh) > DRILL-4313/pr 396 (Improved client randomization. Update JIRA with > warnings about using the feature ) (Sudheesh to review.) > DRILL-4449/pr 389 (Wrong results when metadata cache is used..) (Aman to > review) > DRILL-4069/pr 352 Enable RPC thread offload by default (Owner: Sudheesh) > > Need review - > DRILL-4375/pr 402 (Fix the maven release profile) > DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite) > DRILL-4332/pr 389 (Make vector comparison order stable in test framework) > DRILL-4411/pr 381 (hash join over-memory condition) > DRILL-4387/pr 379 (GroupScan should not use star column) > DRILL-4184/pr 372 (support variable length decimal fields in parquet) > DRILL-4120 - dir0 does not work when the directory structure contains Avro > files - Partial patch available. > DRILL-4203/pr 341 (fix dates written into parquet files to conform to > parquet format spec) > > Not included (yet) - > DRILL-3149 - No patch available > DRILL-4441 - IN operator does not work with Avro reader - No patch > available > DRILL-3745/pr 399 - Hive char support - New feature - Needs QA - Not > included in 1.6 > DRILL-3623 - Limit 0 should avoid execution when querying a known schema. > (Need to add limitations of current impl). Intrusive change; should be > included at beginning of release cycle. > DRILL-4416/pr 385 (quote path separator) (Owner: Hanifi) - Causes leak. > > Others - > DRILL-2517 - Already resolved. > DRILL-3688/pr 382 (skip.header.line.count in hive). - Already merged. PR > needs to be closed. > > > On Thu, Mar 3, 2016 at 9:44 PM, Parth Chandra <[email protected]> wrote: > > > Right. My mistake. Thanks, Jacques, for reviewing. > > > > On Thu, Mar 3, 2016 at 9:08 PM, Zelaine Fong <[email protected]> wrote: > > > >> DRILL-4281/pr 400 (Drill should support inbound impersonation) (Sudheesh > >> to > >> review) > >> > >> Sudheesh is the fixer of DRILL-4281, so I don't think he can be the > >> reviewer :). > >> > >> -- Zelaine > >> > >> On Thu, Mar 3, 2016 at 6:30 PM, Parth Chandra <[email protected]> > wrote: > >> > >> > Here's an updated list with names of reviewers added. If anyone else > is > >> > reviewing the open PRs please let me know. Some PRs have owners names > >> that > >> > I will follow up with. > >> > Jason, I've included your JIRA in the list. > >> > > >> > > >> > Committed for 1.6 - > >> > > >> > DRILL-4384 - Query profile is missing important information on WebUi - > >> > Merged > >> > DRILL-3488/pr 388 (Java 1.8 support) - Merged. > >> > DRILL-4410/pr 380 (listvector should initiatlize bits...) - Merged > >> > DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos, etc) - > Merged > >> > DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration) - > >> > Waiting to be merged > >> > > >> > DRILL-4281/pr 400 (Drill should support inbound impersonation) > >> (Sudheesh to > >> > review) > >> > DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly > >> expose > >> > their types within Calcite.) - Waiting for Aman to review. (Owners: > >> Hsuan, > >> > Jinfeng, Aman, Sudheesh) > >> > DRILL-4313/pr 396 (Improved client randomization. Update JIRA with > >> > warnings about using the feature ) (Sudheesh to review.) > >> > DRILL-4437 (and others)/pr 394 (Operator unit test framework). (Parth > to > >> > review) > >> > DRILL-4449/pr 389 (Wrong results when metadata cache is used..) (Aman > to > >> > review) > >> > DRILL-4416/pr 385 (quote path separator) (Owner: Hanifi) > >> > DRILL-4069/pr 352 Enable RPC thread offload by default (Owner: > Sudheesh) > >> > > >> > Need review - > >> > DRILL-4375/pr 402 (Fix the maven release profile) > >> > DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite) > >> > DRILL-4332/pr 389 (Make vector comparison order stable in test > >> framework) > >> > DRILL-4411/pr 381 (hash join over-memory condition) > >> > DRILL-4387/pr 379 (GroupScan should not use star column) > >> > DRILL-4184/pr 372 (support variable length decimal fields in parquet) > >> > DRILL-4120 - dir0 does not work when the directory structure contains > >> Avro > >> > files - Partial patch available. > >> > DRILL-4203/pr 341 (fix dates written into parquet files to conform to > >> > parquet format spec) > >> > > >> > Not included (yet) - > >> > DRILL-3149 - No patch available > >> > DRILL-4441 - IN operator does not work with Avro reader - No patch > >> > available > >> > DRILL-3745/pr 399 - Hive char support - New feature - Needs QA - Not > >> > included in 1.6 > >> > DRILL-3623 - Limit 0 should avoid execution when querying a known > >> schema. > >> > (Need to add limitations of current impl). Intrusive change; should be > >> > included at beginning of release cycle. > >> > > >> > Others - > >> > DRILL-2517 - Already resolved. > >> > DRILL-3688/pr 382 (skip.header.line.count in hive). - Already merged. > PR > >> > needs to be closed. > >> > > >> > > >> > > >> > Thanks > >> > > >> > Parth > >> > > >> > > >> > > >> > > >> > > >> > On Thu, Mar 3, 2016 at 3:21 PM, Jason Altekruse < > >> [email protected]> > >> > wrote: > >> > > >> > > I have updated the PR for the parquet date corruption issue that > >> didn't > >> > > make it into 1.5. > >> > > > >> > > https://github.com/apache/drill/pull/341 > >> > > https://issues.apache.org/jira/browse/DRILL-4203 > >> > > > >> > > If this can get reviewed, I think it would be good to get into the > >> > release. > >> > > Any takers? > >> > > > >> > > On Wed, Mar 2, 2016 at 11:07 PM, Parth Chandra <[email protected]> > >> > wrote: > >> > > > >> > > > I've summarized the list of JIRs below. > >> > > > The first set of pull requests is under review (or have some > >> reviewer > >> > > > assigned). > >> > > > The second set contains pull requests that need review. We need > >> > > committers > >> > > > to review these. Please volunteer or these will not be able to > make > >> it > >> > > into > >> > > > the release. > >> > > > The third set is Jira's that do not have a patch and/or should not > >> be > >> > > > included because they require deeper scrutiny. > >> > > > I'm hoping we can finalize the list of PRs that can be reviewed by > >> > Friday > >> > > > morning and possibly *finalize the list of issues to be included > by > >> > > Friday > >> > > > end of day* so please take some time to review the PRs. > >> > > > Also note that the QA team has offered to do sanity testing once > we > >> > > decide > >> > > > on the final commit to be included, before the release candidate > is > >> > > rolled > >> > > > out, which helps with the release candidate moving forward > smoothly. > >> > > > > >> > > > Here's the list - > >> > > > > >> > > > *Committed for 1.6 -* > >> > > > DRILL-4281/pr 400 (Drill should support inbound impersonation) > >> > > > DRILL-4372/pr 377(?) (Drill Operators and Functions should > correctly > >> > > expose > >> > > > their types within Calcite.) - Waiting for Aman to review. > >> > > > DRILL-4313/pr 396 (Improved client randomization. Update JIRA > with > >> > > > warnings about using the feature ) Sudheesh to review. > >> > > > DRILL-3488/pr 388 (Java 1.8 support) Hanifi to review > >> > > > DRILL-4437 (and others)/pr 394 (Operator unit test framework). > >> Parth to > >> > > > review > >> > > > DRILL-4384 - Query profile is missing important information on > >> WebUi - > >> > > > Marked as resolved. Patch not applied? > >> > > > > >> > > > *Need review -* > >> > > > DRILL-4465/pr 401 (Simplify Calcite parsing & planning > integration) > >> > > > DRILL-4375/pr 402 (Fix the maven release profile) > >> > > > DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite) > >> > > > DRILL-4332/pr 389 (Make vector comparison order stable in test > >> > framework) > >> > > > DRILL-4449/pr 389 (Wrong results when metadata cache is used..) > >> > > > DRILL-4416/pr 385 (quote path separator) > >> > > > DRILL-4411/pr 381 (hash join over-memory condition) > >> > > > DRILL-4410/pr 380 (listvector should initiatlize bits...) > >> > > > DRILL-4387/pr 379 (GroupScan should not use star column) > >> > > > DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos, etc) > >> > > > DRILL-4184/pr 372 (support variable length decimal fields in > >> parquet) > >> > > > DRILL-4069/pr 352 Enable RPC thread offload by default > >> > > > DRILL-4120 - dir0 does not work when the directory structure > >> contains > >> > > Avro > >> > > > files - Partial patch available. > >> > > > > >> > > > *Not included (yet) - * > >> > > > DRILL-3149 - No patch available > >> > > > DRILL-4441 - IN operator does not work with Avro reader - No patch > >> > > > available > >> > > > DRILL-3745/pr 399 - Hive char support - New feature - Needs QA - > Not > >> > > > included in 1.6 > >> > > > DRILL-3623 - Limit 0 should avoid execution when querying a known > >> > schema. > >> > > > (Need to add limitations of current impl). Intrusive change; > should > >> be > >> > > > included at beginning of release cycle. > >> > > > > >> > > > *Others -* > >> > > > DRILL-2517 - Already resolved. > >> > > > DRILL-3688/pr 382 (skip.header.line.count in hive). - Already > >> merged. > >> > PR > >> > > > needs to be closed. > >> > > > > >> > > > > >> > > > > >> > > > On Wed, Mar 2, 2016 at 3:11 PM, Vicky Markman < > >> [email protected]> > >> > > > wrote: > >> > > > > >> > > > > You are welcome, Jacques. > >> > > > > > >> > > > > Vick*y *:) > >> > > > > > >> > > > > On Wed, Mar 2, 2016 at 3:06 PM, Jacques Nadeau < > >> [email protected]> > >> > > > wrote: > >> > > > > > >> > > > > > I just realized that we didn't merge the broken profile patch > >> > (thanks > >> > > > > > Vicki). We should get it merged as well. > >> > > > > > > >> > > > > > DRILL-4384 > >> > > > > > > >> > > > > > -- > >> > > > > > Jacques Nadeau > >> > > > > > CTO and Co-Founder, Dremio > >> > > > > > > >> > > > > > On Wed, Mar 2, 2016 at 10:46 AM, Jason Altekruse < > >> > > > > [email protected] > >> > > > > > > > >> > > > > > wrote: > >> > > > > > > >> > > > > > > I should have merged this sooner but we will need this patch > >> > that I > >> > > > had > >> > > > > > > applied to the 1.5 release branch. The change is small and > >> fixes > >> > a > >> > > > > build > >> > > > > > > problem that only appears when running the maven release > >> profile. > >> > > > > > > > >> > > > > > > https://github.com/apache/drill/pull/402 > >> > > > > > > > >> > > > > > > On Wed, Mar 2, 2016 at 9:28 AM, Jinfeng Ni < > >> > [email protected]> > >> > > > > > wrote: > >> > > > > > > > >> > > > > > > > Hi John, > >> > > > > > > > > >> > > > > > > > I think patch for DRILL-2517 has been merged to the apache > >> > master > >> > > > > > > > branch. Have you tried your query on the latest master > >> branch? > >> > > > > > > > > >> > > > > > > > In DRILL-2517, I posted some performance number for 117k > >> small > >> > > > > parquet > >> > > > > > > > files. The patch did show improvement. > >> > > > > > > > > >> > > > > > > > Before DRILL-3996 is resolved, for now if your query > relies > >> on > >> > > > filter > >> > > > > > > > pushdown logic to push partitioning filter first, then the > >> > patch > >> > > > for > >> > > > > > > > DRILL-2517 will not help. > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > On Wed, Mar 2, 2016 at 4:23 AM, John Omernik < > >> [email protected] > >> > > > >> > > > > wrote: > >> > > > > > > > > I'd like to request drill-2517 be added as a bandaid for > >> the > >> > > > > planning > >> > > > > > > > > issues when there are lots of directories of parquet > >> files. > >> > > > This > >> > > > > > > issue > >> > > > > > > > is > >> > > > > > > > > really hurting drill adoption for me in my org. > >> > > > > > > > > > >> > > > > > > > > Thanks, John > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > On Tuesday, March 1, 2016, Edmon Begoli < > >> [email protected]> > >> > > > wrote: > >> > > > > > > > > > >> > > > > > > > >> May I please ask to give this issue the attention for > >> 1.6: > >> > > > > > > > >> > >> > > > > > > > >> > > > > >> https://issues.apache.org/jira/plugins/servlet/mobile#issue/DRILL-3149 > >> > > > > > > > >> > >> > > > > > > > >> I will try to suggest a patch. Given my time > constraints > >> I > >> > > might > >> > > > > not > >> > > > > > > be > >> > > > > > > > >> able to submit a complete, unit tested code but at I > >> least I > >> > > > will > >> > > > > > try > >> > > > > > > to > >> > > > > > > > >> submit a snippet that will help with fixing it up (I > >> think > >> > we > >> > > > just > >> > > > > > > need > >> > > > > > > > to > >> > > > > > > > >> do a look ahead byte lookup to ensure that it is not > >> \r\n. > >> > > > > > > > >> > >> > > > > > > > >> On Tuesday, March 1, 2016, Jacques Nadeau < > >> > [email protected] > >> > > > > > > > >> <javascript:;>> wrote: > >> > > > > > > > >> > >> > > > > > > > >> > It seems like a stretch to include DRILL-3623 right > >> before > >> > > the > >> > > > > > > > release. > >> > > > > > > > >> > This is a pretty fundamental change that seems like > it > >> > > should > >> > > > > soak > >> > > > > > > > for a > >> > > > > > > > >> > bit of time before we release. If we want to include, > >> I'd > >> > > > > suggest > >> > > > > > > > that we > >> > > > > > > > >> > disable the functionality by default and consider it > >> > > > > experimental. > >> > > > > > > > >> > > >> > > > > > > > >> > I'll propose a few other patches for inclusion > shortly. > >> > > > > > > > >> > > >> > > > > > > > >> > -- > >> > > > > > > > >> > Jacques Nadeau > >> > > > > > > > >> > CTO and Co-Founder, Dremio > >> > > > > > > > >> > > >> > > > > > > > >> > On Tue, Mar 1, 2016 at 5:04 PM, Parth Chandra < > >> > > > > [email protected] > >> > > > > > > > >> <javascript:;> > >> > > > > > > > >> > <javascript:;>> wrote: > >> > > > > > > > >> > > >> > > > > > > > >> > > Hello everyone, > >> > > > > > > > >> > > > >> > > > > > > > >> > > It's time to start looking into the 1.6 release. > >> > > > > > > > >> > > > >> > > > > > > > >> > > Can all the folks working on open issues let me > >> know > >> > if > >> > > > > there > >> > > > > > > are > >> > > > > > > > any > >> > > > > > > > >> > > JIRAs you would like to get into the release? > >> > > > > > > > >> > > > >> > > > > > > > >> > > I know of the following - > >> > > > > > > > >> > > > >> > > > > > > > >> > > *DRILL-4281 *- Drill should support inbound > >> > impersonation. > >> > > > > Pull > >> > > > > > > > request > >> > > > > > > > >> > > expected today. C++ client to be tested. > >> > > > > > > > >> > > > >> > > > > > > > >> > > *DRILL-4372* - Drill Operators and Functions should > >> > > > correctly > >> > > > > > > expose > >> > > > > > > > >> > their > >> > > > > > > > >> > > types within Calcite. Waiting for review. > >> > > > > > > > >> > > > >> > > > > > > > >> > > *DRILL-3623* - Limit 0 should avoid execution when > >> > > querying > >> > > > a > >> > > > > > > known > >> > > > > > > > >> > schema. > >> > > > > > > > >> > > Pull request expected today. Need to add > limitations > >> of > >> > > > > current > >> > > > > > > > impl to > >> > > > > > > > >> > the > >> > > > > > > > >> > > JIRA. Review needed. > >> > > > > > > > >> > > > >> > > > > > > > >> > > *DRILL-4313* - Improved client randomization. > Update > >> > JIRA > >> > > > with > >> > > > > > > > warnings > >> > > > > > > > >> > > about using the feature. Waiting for review. > >> > > > > > > > >> > > > >> > > > > > > > >> > > > >> > > > > > > > >> > > Thanks > >> > > > > > > > >> > > > >> > > > > > > > >> > > Parth > >> > > > > > > > >> > > > >> > > > > > > > >> > > >> > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > -- > >> > > > > > > > > Sent from my iThing > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > > >
