Jacques submitted a PR for fixing the failed baselines. I've merged them into automation master and confirmed the failed tests are all passing now. Thanks.
-Chun On Thu, Mar 3, 2016 at 10:48 PM, Jacques Nadeau <[email protected]> wrote: > I think we need to include DRILL-4467 > <https://issues.apache.org/jira/browse/DRILL-4467>. I think it is a one > line patch and it provides unpredictable plans at a minimum but may also > present invalid result. Still need to think through the second half. I've > seen this plan instability in some of my recent test runs (even without > Java 8) when running extended HBase tests. > > -- > Jacques Nadeau > CTO and Co-Founder, Dremio > > On Thu, Mar 3, 2016 at 10:02 PM, Parth Chandra <[email protected]> wrote: > > > Updated list (I'll follow up with the folks named here separately) - > > > > Committed for 1.6 - > > > > DRILL-4384 - Query profile is missing important information on WebUi - > > Merged > > DRILL-3488/pr 388 (Java 1.8 support) - Merged. > > DRILL-4410/pr 380 (listvector should initiatlize bits...) - Merged > > DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos, etc) - Merged > > DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration) - > > Waiting to be merged > > DRILL-4437 (and others)/pr 394 (Operator unit test framework). Waiting to > > be merged. > > > > DRILL-4281/pr 400 (Drill should support inbound impersonation) (Jacques > to > > review) > > DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly > expose > > their types within Calcite.) - Waiting for Aman to review. (Owners: > Hsuan, > > Jinfeng, Aman, Sudheesh) > > DRILL-4313/pr 396 (Improved client randomization. Update JIRA with > > warnings about using the feature ) (Sudheesh to review.) > > DRILL-4449/pr 389 (Wrong results when metadata cache is used..) (Aman to > > review) > > DRILL-4069/pr 352 Enable RPC thread offload by default (Owner: Sudheesh) > > > > Need review - > > DRILL-4375/pr 402 (Fix the maven release profile) > > DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite) > > DRILL-4332/pr 389 (Make vector comparison order stable in test framework) > > DRILL-4411/pr 381 (hash join over-memory condition) > > DRILL-4387/pr 379 (GroupScan should not use star column) > > DRILL-4184/pr 372 (support variable length decimal fields in parquet) > > DRILL-4120 - dir0 does not work when the directory structure contains > Avro > > files - Partial patch available. > > DRILL-4203/pr 341 (fix dates written into parquet files to conform to > > parquet format spec) > > > > Not included (yet) - > > DRILL-3149 - No patch available > > DRILL-4441 - IN operator does not work with Avro reader - No patch > > available > > DRILL-3745/pr 399 - Hive char support - New feature - Needs QA - Not > > included in 1.6 > > DRILL-3623 - Limit 0 should avoid execution when querying a known schema. > > (Need to add limitations of current impl). Intrusive change; should be > > included at beginning of release cycle. > > DRILL-4416/pr 385 (quote path separator) (Owner: Hanifi) - Causes leak. > > > > Others - > > DRILL-2517 - Already resolved. > > DRILL-3688/pr 382 (skip.header.line.count in hive). - Already merged. PR > > needs to be closed. > > > > > > On Thu, Mar 3, 2016 at 9:44 PM, Parth Chandra <[email protected]> wrote: > > > > > Right. My mistake. Thanks, Jacques, for reviewing. > > > > > > On Thu, Mar 3, 2016 at 9:08 PM, Zelaine Fong <[email protected]> > wrote: > > > > > >> DRILL-4281/pr 400 (Drill should support inbound impersonation) > (Sudheesh > > >> to > > >> review) > > >> > > >> Sudheesh is the fixer of DRILL-4281, so I don't think he can be the > > >> reviewer :). > > >> > > >> -- Zelaine > > >> > > >> On Thu, Mar 3, 2016 at 6:30 PM, Parth Chandra <[email protected]> > > wrote: > > >> > > >> > Here's an updated list with names of reviewers added. If anyone else > > is > > >> > reviewing the open PRs please let me know. Some PRs have owners > names > > >> that > > >> > I will follow up with. > > >> > Jason, I've included your JIRA in the list. > > >> > > > >> > > > >> > Committed for 1.6 - > > >> > > > >> > DRILL-4384 - Query profile is missing important information on > WebUi - > > >> > Merged > > >> > DRILL-3488/pr 388 (Java 1.8 support) - Merged. > > >> > DRILL-4410/pr 380 (listvector should initiatlize bits...) - Merged > > >> > DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos, etc) - > > Merged > > >> > DRILL-4465/pr 401 (Simplify Calcite parsing & planning integration) > - > > >> > Waiting to be merged > > >> > > > >> > DRILL-4281/pr 400 (Drill should support inbound impersonation) > > >> (Sudheesh to > > >> > review) > > >> > DRILL-4372/pr 377(?) (Drill Operators and Functions should correctly > > >> expose > > >> > their types within Calcite.) - Waiting for Aman to review. (Owners: > > >> Hsuan, > > >> > Jinfeng, Aman, Sudheesh) > > >> > DRILL-4313/pr 396 (Improved client randomization. Update JIRA with > > >> > warnings about using the feature ) (Sudheesh to review.) > > >> > DRILL-4437 (and others)/pr 394 (Operator unit test framework). > (Parth > > to > > >> > review) > > >> > DRILL-4449/pr 389 (Wrong results when metadata cache is used..) > (Aman > > to > > >> > review) > > >> > DRILL-4416/pr 385 (quote path separator) (Owner: Hanifi) > > >> > DRILL-4069/pr 352 Enable RPC thread offload by default (Owner: > > Sudheesh) > > >> > > > >> > Need review - > > >> > DRILL-4375/pr 402 (Fix the maven release profile) > > >> > DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite) > > >> > DRILL-4332/pr 389 (Make vector comparison order stable in test > > >> framework) > > >> > DRILL-4411/pr 381 (hash join over-memory condition) > > >> > DRILL-4387/pr 379 (GroupScan should not use star column) > > >> > DRILL-4184/pr 372 (support variable length decimal fields in > parquet) > > >> > DRILL-4120 - dir0 does not work when the directory structure > contains > > >> Avro > > >> > files - Partial patch available. > > >> > DRILL-4203/pr 341 (fix dates written into parquet files to conform > to > > >> > parquet format spec) > > >> > > > >> > Not included (yet) - > > >> > DRILL-3149 - No patch available > > >> > DRILL-4441 - IN operator does not work with Avro reader - No patch > > >> > available > > >> > DRILL-3745/pr 399 - Hive char support - New feature - Needs QA - Not > > >> > included in 1.6 > > >> > DRILL-3623 - Limit 0 should avoid execution when querying a known > > >> schema. > > >> > (Need to add limitations of current impl). Intrusive change; should > be > > >> > included at beginning of release cycle. > > >> > > > >> > Others - > > >> > DRILL-2517 - Already resolved. > > >> > DRILL-3688/pr 382 (skip.header.line.count in hive). - Already > merged. > > PR > > >> > needs to be closed. > > >> > > > >> > > > >> > > > >> > Thanks > > >> > > > >> > Parth > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > On Thu, Mar 3, 2016 at 3:21 PM, Jason Altekruse < > > >> [email protected]> > > >> > wrote: > > >> > > > >> > > I have updated the PR for the parquet date corruption issue that > > >> didn't > > >> > > make it into 1.5. > > >> > > > > >> > > https://github.com/apache/drill/pull/341 > > >> > > https://issues.apache.org/jira/browse/DRILL-4203 > > >> > > > > >> > > If this can get reviewed, I think it would be good to get into the > > >> > release. > > >> > > Any takers? > > >> > > > > >> > > On Wed, Mar 2, 2016 at 11:07 PM, Parth Chandra <[email protected] > > > > >> > wrote: > > >> > > > > >> > > > I've summarized the list of JIRs below. > > >> > > > The first set of pull requests is under review (or have some > > >> reviewer > > >> > > > assigned). > > >> > > > The second set contains pull requests that need review. We need > > >> > > committers > > >> > > > to review these. Please volunteer or these will not be able to > > make > > >> it > > >> > > into > > >> > > > the release. > > >> > > > The third set is Jira's that do not have a patch and/or should > not > > >> be > > >> > > > included because they require deeper scrutiny. > > >> > > > I'm hoping we can finalize the list of PRs that can be reviewed > by > > >> > Friday > > >> > > > morning and possibly *finalize the list of issues to be included > > by > > >> > > Friday > > >> > > > end of day* so please take some time to review the PRs. > > >> > > > Also note that the QA team has offered to do sanity testing once > > we > > >> > > decide > > >> > > > on the final commit to be included, before the release candidate > > is > > >> > > rolled > > >> > > > out, which helps with the release candidate moving forward > > smoothly. > > >> > > > > > >> > > > Here's the list - > > >> > > > > > >> > > > *Committed for 1.6 -* > > >> > > > DRILL-4281/pr 400 (Drill should support inbound impersonation) > > >> > > > DRILL-4372/pr 377(?) (Drill Operators and Functions should > > correctly > > >> > > expose > > >> > > > their types within Calcite.) - Waiting for Aman to review. > > >> > > > DRILL-4313/pr 396 (Improved client randomization. Update JIRA > > with > > >> > > > warnings about using the feature ) Sudheesh to review. > > >> > > > DRILL-3488/pr 388 (Java 1.8 support) Hanifi to review > > >> > > > DRILL-4437 (and others)/pr 394 (Operator unit test framework). > > >> Parth to > > >> > > > review > > >> > > > DRILL-4384 - Query profile is missing important information on > > >> WebUi - > > >> > > > Marked as resolved. Patch not applied? > > >> > > > > > >> > > > *Need review -* > > >> > > > DRILL-4465/pr 401 (Simplify Calcite parsing & planning > > integration) > > >> > > > DRILL-4375/pr 402 (Fix the maven release profile) > > >> > > > DRILL-4452/pr 395 (Update Avatica Driver to latest Calcite) > > >> > > > DRILL-4332/pr 389 (Make vector comparison order stable in test > > >> > framework) > > >> > > > DRILL-4449/pr 389 (Wrong results when metadata cache is used..) > > >> > > > DRILL-4416/pr 385 (quote path separator) > > >> > > > DRILL-4411/pr 381 (hash join over-memory condition) > > >> > > > DRILL-4410/pr 380 (listvector should initiatlize bits...) > > >> > > > DRILL-4387/pr 379 (GroupScan should not use star column) > > >> > > > DRILL-4383/pr 375 (Allow custom configs for S3, Kerberos, etc) > > >> > > > DRILL-4184/pr 372 (support variable length decimal fields in > > >> parquet) > > >> > > > DRILL-4069/pr 352 Enable RPC thread offload by default > > >> > > > DRILL-4120 - dir0 does not work when the directory structure > > >> contains > > >> > > Avro > > >> > > > files - Partial patch available. > > >> > > > > > >> > > > *Not included (yet) - * > > >> > > > DRILL-3149 - No patch available > > >> > > > DRILL-4441 - IN operator does not work with Avro reader - No > patch > > >> > > > available > > >> > > > DRILL-3745/pr 399 - Hive char support - New feature - Needs QA - > > Not > > >> > > > included in 1.6 > > >> > > > DRILL-3623 - Limit 0 should avoid execution when querying a > known > > >> > schema. > > >> > > > (Need to add limitations of current impl). Intrusive change; > > should > > >> be > > >> > > > included at beginning of release cycle. > > >> > > > > > >> > > > *Others -* > > >> > > > DRILL-2517 - Already resolved. > > >> > > > DRILL-3688/pr 382 (skip.header.line.count in hive). - Already > > >> merged. > > >> > PR > > >> > > > needs to be closed. > > >> > > > > > >> > > > > > >> > > > > > >> > > > On Wed, Mar 2, 2016 at 3:11 PM, Vicky Markman < > > >> [email protected]> > > >> > > > wrote: > > >> > > > > > >> > > > > You are welcome, Jacques. > > >> > > > > > > >> > > > > Vick*y *:) > > >> > > > > > > >> > > > > On Wed, Mar 2, 2016 at 3:06 PM, Jacques Nadeau < > > >> [email protected]> > > >> > > > wrote: > > >> > > > > > > >> > > > > > I just realized that we didn't merge the broken profile > patch > > >> > (thanks > > >> > > > > > Vicki). We should get it merged as well. > > >> > > > > > > > >> > > > > > DRILL-4384 > > >> > > > > > > > >> > > > > > -- > > >> > > > > > Jacques Nadeau > > >> > > > > > CTO and Co-Founder, Dremio > > >> > > > > > > > >> > > > > > On Wed, Mar 2, 2016 at 10:46 AM, Jason Altekruse < > > >> > > > > [email protected] > > >> > > > > > > > > >> > > > > > wrote: > > >> > > > > > > > >> > > > > > > I should have merged this sooner but we will need this > patch > > >> > that I > > >> > > > had > > >> > > > > > > applied to the 1.5 release branch. The change is small and > > >> fixes > > >> > a > > >> > > > > build > > >> > > > > > > problem that only appears when running the maven release > > >> profile. > > >> > > > > > > > > >> > > > > > > https://github.com/apache/drill/pull/402 > > >> > > > > > > > > >> > > > > > > On Wed, Mar 2, 2016 at 9:28 AM, Jinfeng Ni < > > >> > [email protected]> > > >> > > > > > wrote: > > >> > > > > > > > > >> > > > > > > > Hi John, > > >> > > > > > > > > > >> > > > > > > > I think patch for DRILL-2517 has been merged to the > apache > > >> > master > > >> > > > > > > > branch. Have you tried your query on the latest master > > >> branch? > > >> > > > > > > > > > >> > > > > > > > In DRILL-2517, I posted some performance number for 117k > > >> small > > >> > > > > parquet > > >> > > > > > > > files. The patch did show improvement. > > >> > > > > > > > > > >> > > > > > > > Before DRILL-3996 is resolved, for now if your query > > relies > > >> on > > >> > > > filter > > >> > > > > > > > pushdown logic to push partitioning filter first, then > the > > >> > patch > > >> > > > for > > >> > > > > > > > DRILL-2517 will not help. > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > On Wed, Mar 2, 2016 at 4:23 AM, John Omernik < > > >> [email protected] > > >> > > > > >> > > > > wrote: > > >> > > > > > > > > I'd like to request drill-2517 be added as a bandaid > for > > >> the > > >> > > > > planning > > >> > > > > > > > > issues when there are lots of directories of parquet > > >> files. > > >> > > > This > > >> > > > > > > issue > > >> > > > > > > > is > > >> > > > > > > > > really hurting drill adoption for me in my org. > > >> > > > > > > > > > > >> > > > > > > > > Thanks, John > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > On Tuesday, March 1, 2016, Edmon Begoli < > > >> [email protected]> > > >> > > > wrote: > > >> > > > > > > > > > > >> > > > > > > > >> May I please ask to give this issue the attention for > > >> 1.6: > > >> > > > > > > > >> > > >> > > > > > > > > >> > > > > > >> > https://issues.apache.org/jira/plugins/servlet/mobile#issue/DRILL-3149 > > >> > > > > > > > >> > > >> > > > > > > > >> I will try to suggest a patch. Given my time > > constraints > > >> I > > >> > > might > > >> > > > > not > > >> > > > > > > be > > >> > > > > > > > >> able to submit a complete, unit tested code but at I > > >> least I > > >> > > > will > > >> > > > > > try > > >> > > > > > > to > > >> > > > > > > > >> submit a snippet that will help with fixing it up (I > > >> think > > >> > we > > >> > > > just > > >> > > > > > > need > > >> > > > > > > > to > > >> > > > > > > > >> do a look ahead byte lookup to ensure that it is not > > >> \r\n. > > >> > > > > > > > >> > > >> > > > > > > > >> On Tuesday, March 1, 2016, Jacques Nadeau < > > >> > [email protected] > > >> > > > > > > > >> <javascript:;>> wrote: > > >> > > > > > > > >> > > >> > > > > > > > >> > It seems like a stretch to include DRILL-3623 right > > >> before > > >> > > the > > >> > > > > > > > release. > > >> > > > > > > > >> > This is a pretty fundamental change that seems like > > it > > >> > > should > > >> > > > > soak > > >> > > > > > > > for a > > >> > > > > > > > >> > bit of time before we release. If we want to > include, > > >> I'd > > >> > > > > suggest > > >> > > > > > > > that we > > >> > > > > > > > >> > disable the functionality by default and consider > it > > >> > > > > experimental. > > >> > > > > > > > >> > > > >> > > > > > > > >> > I'll propose a few other patches for inclusion > > shortly. > > >> > > > > > > > >> > > > >> > > > > > > > >> > -- > > >> > > > > > > > >> > Jacques Nadeau > > >> > > > > > > > >> > CTO and Co-Founder, Dremio > > >> > > > > > > > >> > > > >> > > > > > > > >> > On Tue, Mar 1, 2016 at 5:04 PM, Parth Chandra < > > >> > > > > [email protected] > > >> > > > > > > > >> <javascript:;> > > >> > > > > > > > >> > <javascript:;>> wrote: > > >> > > > > > > > >> > > > >> > > > > > > > >> > > Hello everyone, > > >> > > > > > > > >> > > > > >> > > > > > > > >> > > It's time to start looking into the 1.6 > release. > > >> > > > > > > > >> > > > > >> > > > > > > > >> > > Can all the folks working on open issues let me > > >> know > > >> > if > > >> > > > > there > > >> > > > > > > are > > >> > > > > > > > any > > >> > > > > > > > >> > > JIRAs you would like to get into the release? > > >> > > > > > > > >> > > > > >> > > > > > > > >> > > I know of the following - > > >> > > > > > > > >> > > > > >> > > > > > > > >> > > *DRILL-4281 *- Drill should support inbound > > >> > impersonation. > > >> > > > > Pull > > >> > > > > > > > request > > >> > > > > > > > >> > > expected today. C++ client to be tested. > > >> > > > > > > > >> > > > > >> > > > > > > > >> > > *DRILL-4372* - Drill Operators and Functions > should > > >> > > > correctly > > >> > > > > > > expose > > >> > > > > > > > >> > their > > >> > > > > > > > >> > > types within Calcite. Waiting for review. > > >> > > > > > > > >> > > > > >> > > > > > > > >> > > *DRILL-3623* - Limit 0 should avoid execution > when > > >> > > querying > > >> > > > a > > >> > > > > > > known > > >> > > > > > > > >> > schema. > > >> > > > > > > > >> > > Pull request expected today. Need to add > > limitations > > >> of > > >> > > > > current > > >> > > > > > > > impl to > > >> > > > > > > > >> > the > > >> > > > > > > > >> > > JIRA. Review needed. > > >> > > > > > > > >> > > > > >> > > > > > > > >> > > *DRILL-4313* - Improved client randomization. > > Update > > >> > JIRA > > >> > > > with > > >> > > > > > > > warnings > > >> > > > > > > > >> > > about using the feature. Waiting for review. > > >> > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > > >> > > Thanks > > >> > > > > > > > >> > > > > >> > > > > > > > >> > > Parth > > >> > > > > > > > >> > > > > >> > > > > > > > >> > > > >> > > > > > > > >> > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > -- > > >> > > > > > > > > Sent from my iThing > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > >
