[jira] [Created] (HIVE-16405) Flaky Test: org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData

[Sunitha Beeram (JIRA)]

Sunitha Beeram created HIVE-16405:
-

 Summary: Flaky Test: 
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData
 Key: HIVE-16405
 URL: https://issues.apache.org/jira/browse/HIVE-16405
 Project: Hive
  Issue Type: Sub-task
Reporter: Sunitha Beeram


Has failed for about 30 builds now. One instance: 
https://builds.apache.org/job/PreCommit-HIVE-Build/4577/testReport/org.apache.hive.jdbc/TestJdbcDriver2/testResultSetMetaData/

{noformat}
Error Message

expected:<[c1]> but was:<[_col0]>
Stacktrace

org.junit.ComparisonFailure: expected:<[c1]> but was:<[_col0]>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData(TestJdbcDriver2.java:1541)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58181: HIVE-15986 Support for 'is [NOT] distinct from'

[pengcheng xiong]

> On April 5, 2017, 3:37 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g
> > Lines 590-592 (patched)
> > 
> >
> > It will be good to not add extra tokens in grammar as it increases the 
> > size of state machine. How about:
> > 
> > 
> > KW_IS (a=KW_NOT)? KW_DISTINCT KW_FROM
> > -> {$a !=null}? ^(EQUAL_NS)
> > -> ^(KW_NOT EQUAL_NS)
> 
> Vineet Garg wrote:
> I agree and I tried not to add new tokens but I couldn't figure out how 
> to write grammar in such a way to avoid adding it. 
> 
> For is distinct from we want AST as follows
> KW_NOT
>   EQUAL_NS
> Expr1
> Expr2
>   
> isDistinctFrom rule is invoked from precedenceEqualOperator which is 
> suppose to return an AST for operator. This AST is further used by 
> precedenceEqualOperator's invoker to make an AST with returned AST as root 
> and with two expression as it's children. So if isDistinctFrom return this AST
>   KW_NOT
> EQUAL_NS
> 
>   invoker of precedenceEqualOperator will end up creating
>   KW_NOT
>EQUAL_NS
>Expr1
>Expr2
>
>   which is not what we want.
>   
>  Your above suggestion throws an exception while parsing FAILED: 
> RewriteEmptyStreamException token KW_NOT.
>  
>  I am not sure why
> 
> pengcheng xiong wrote:
> can u try
> KW_IS (a=KW_NOT)? KW_DISTINCT KW_FROM
> -> {$a !=null}? ^(EQUAL_NS)
> -> ^(KW_NOT ^EQUAL_NS)
> 
> Vineet Garg wrote:
> I get compilation error
> error(100): IdentifiersParser.g:592:17: syntax error: antlr: 
> MismatchedTokenException

sorry, how about this?


KW_IS (a=KW_NOT)? KW_DISTINCT KW_FROM
-> {$a !=null}? ^(EQUAL_NS)
-> ^(KW_NOT ^(EQUAL_NS))


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58181/#review171125
---


On April 6, 2017, 8:02 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58181/
> ---
> 
> (Updated April 6, 2017, 8:02 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and 
> Pengcheng Xu.
> 
> 
> Bugs: HIVE-15986
> https://issues.apache.org/jira/browse/HIVE-15986
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch adds support for 'is distinct from' and 'is not distinct from'.
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 7a70c9c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ccfb455 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
>  85450c9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g d98a663 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 8c4ee8a 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> f979c14 
>   ql/src/test/queries/clientpositive/is_distinct_from.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/is_distinct_from.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/show_functions.q.out 68e248a 
>   ql/src/test/results/clientpositive/udf_equal.q.out 52bd843 
> 
> 
> Diff: https://reviews.apache.org/r/58181/diff/4/
> 
> 
> Testing
> ---
> 
> Added new tests
> Pre-commit testing
> 
> 
> Thanks,
> 
> Vineet Garg
> 
>

[jira] [Created] (HIVE-16404) Renaming of public classes in Calcite 12 breeaking druid integration

[slim bouguerra (JIRA)]

slim bouguerra created HIVE-16404:
-

 Summary: Renaming of public classes in Calcite 12 breeaking druid 
integration
 Key: HIVE-16404
 URL: https://issues.apache.org/jira/browse/HIVE-16404
 Project: Hive
  Issue Type: Bug
  Components: Druid integration
Affects Versions: 2.2.0
Reporter: slim bouguerra
 Fix For: 3.0.0


Changes to names in the druid rules is backward incompatible with current 
implementation.
https://github.com/apache/calcite/commit/a89c62cd6d6cc181c90881afa0bf099746739a91



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58181: HIVE-15986 Support for 'is [NOT] distinct from'

[Vineet Garg]

> On April 5, 2017, 3:37 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g
> > Lines 590-592 (patched)
> > 
> >
> > It will be good to not add extra tokens in grammar as it increases the 
> > size of state machine. How about:
> > 
> > 
> > KW_IS (a=KW_NOT)? KW_DISTINCT KW_FROM
> > -> {$a !=null}? ^(EQUAL_NS)
> > -> ^(KW_NOT EQUAL_NS)
> 
> Vineet Garg wrote:
> I agree and I tried not to add new tokens but I couldn't figure out how 
> to write grammar in such a way to avoid adding it. 
> 
> For is distinct from we want AST as follows
> KW_NOT
>   EQUAL_NS
> Expr1
> Expr2
>   
> isDistinctFrom rule is invoked from precedenceEqualOperator which is 
> suppose to return an AST for operator. This AST is further used by 
> precedenceEqualOperator's invoker to make an AST with returned AST as root 
> and with two expression as it's children. So if isDistinctFrom return this AST
>   KW_NOT
> EQUAL_NS
> 
>   invoker of precedenceEqualOperator will end up creating
>   KW_NOT
>EQUAL_NS
>Expr1
>Expr2
>
>   which is not what we want.
>   
>  Your above suggestion throws an exception while parsing FAILED: 
> RewriteEmptyStreamException token KW_NOT.
>  
>  I am not sure why
> 
> pengcheng xiong wrote:
> can u try
> KW_IS (a=KW_NOT)? KW_DISTINCT KW_FROM
> -> {$a !=null}? ^(EQUAL_NS)
> -> ^(KW_NOT ^EQUAL_NS)

I get compilation error
error(100): IdentifiersParser.g:592:17: syntax error: antlr: 
MismatchedTokenException


- Vineet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58181/#review171125
---


On April 6, 2017, 8:02 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58181/
> ---
> 
> (Updated April 6, 2017, 8:02 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and 
> Pengcheng Xu.
> 
> 
> Bugs: HIVE-15986
> https://issues.apache.org/jira/browse/HIVE-15986
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch adds support for 'is distinct from' and 'is not distinct from'.
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 7a70c9c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ccfb455 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
>  85450c9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g d98a663 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 8c4ee8a 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> f979c14 
>   ql/src/test/queries/clientpositive/is_distinct_from.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/is_distinct_from.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/show_functions.q.out 68e248a 
>   ql/src/test/results/clientpositive/udf_equal.q.out 52bd843 
> 
> 
> Diff: https://reviews.apache.org/r/58181/diff/4/
> 
> 
> Testing
> ---
> 
> Added new tests
> Pre-commit testing
> 
> 
> Thanks,
> 
> Vineet Garg
> 
>

Re: Review Request 58181: HIVE-15986 Support for 'is [NOT] distinct from'

[Vineet Garg]

> On April 6, 2017, 8:18 p.m., pengcheng xiong wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g
> > Line 590 (original), 597 (patched)
> > 
> >
> > Can you also confirm the precedence of is distinct with other 
> > operators? Now, it is the same precedence as equal. try 1 is not distinct 
> > from 2 is not distinct from false, etc.

As you have already noted precedence of 'is distinct from' is same as '='.


> On April 6, 2017, 8:18 p.m., pengcheng xiong wrote:
> > ql/src/test/queries/clientpositive/is_distinct_from.q
> > Lines 1 (patched)
> > 
> >
> > Can we compare a column with a constant in is distinct from? or two 
> > columns? Or it should be limited to compare constants only?

Yes it is not only limited to constant,you can compare columns as well.


- Vineet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58181/#review171246
---


On April 6, 2017, 8:02 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58181/
> ---
> 
> (Updated April 6, 2017, 8:02 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and 
> Pengcheng Xu.
> 
> 
> Bugs: HIVE-15986
> https://issues.apache.org/jira/browse/HIVE-15986
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch adds support for 'is distinct from' and 'is not distinct from'.
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 7a70c9c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ccfb455 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
>  85450c9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g d98a663 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 8c4ee8a 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> f979c14 
>   ql/src/test/queries/clientpositive/is_distinct_from.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/is_distinct_from.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/show_functions.q.out 68e248a 
>   ql/src/test/results/clientpositive/udf_equal.q.out 52bd843 
> 
> 
> Diff: https://reviews.apache.org/r/58181/diff/4/
> 
> 
> Testing
> ---
> 
> Added new tests
> Pre-commit testing
> 
> 
> Thanks,
> 
> Vineet Garg
> 
>

[jira] [Created] (HIVE-16403) LLAP UI shows the wrong number of executors

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-16403:
---

 Summary: LLAP UI shows the wrong number of executors
 Key: HIVE-16403
 URL: https://issues.apache.org/jira/browse/HIVE-16403
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


Queued tasks are added twice.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58181: HIVE-15986 Support for 'is [NOT] distinct from'

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58181/#review171246
---




ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g
Line 590 (original), 597 (patched)


Can you also confirm the precedence of is distinct with other operators? 
Now, it is the same precedence as equal. try 1 is not distinct from 2 is not 
distinct from false, etc.



ql/src/test/queries/clientpositive/is_distinct_from.q
Lines 1 (patched)


Can we compare a column with a constant in is distinct from? or two 
columns? Or it should be limited to compare constants only?



ql/src/test/results/clientpositive/udf_equal.q.out
Lines 46 (patched)


OK, if it is a synonyms of <=>, then the precedence is correct.


- pengcheng xiong


On April 6, 2017, 8:02 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58181/
> ---
> 
> (Updated April 6, 2017, 8:02 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and 
> Pengcheng Xu.
> 
> 
> Bugs: HIVE-15986
> https://issues.apache.org/jira/browse/HIVE-15986
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch adds support for 'is distinct from' and 'is not distinct from'.
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 7a70c9c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ccfb455 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
>  85450c9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g d98a663 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 8c4ee8a 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> f979c14 
>   ql/src/test/queries/clientpositive/is_distinct_from.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/is_distinct_from.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/show_functions.q.out 68e248a 
>   ql/src/test/results/clientpositive/udf_equal.q.out 52bd843 
> 
> 
> Diff: https://reviews.apache.org/r/58181/diff/4/
> 
> 
> Testing
> ---
> 
> Added new tests
> Pre-commit testing
> 
> 
> Thanks,
> 
> Vineet Garg
> 
>

Re: Review Request 58181: HIVE-15986 Support for 'is [NOT] distinct from'

[pengcheng xiong]

> On April 5, 2017, 3:37 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g
> > Lines 590-592 (patched)
> > 
> >
> > It will be good to not add extra tokens in grammar as it increases the 
> > size of state machine. How about:
> > 
> > 
> > KW_IS (a=KW_NOT)? KW_DISTINCT KW_FROM
> > -> {$a !=null}? ^(EQUAL_NS)
> > -> ^(KW_NOT EQUAL_NS)
> 
> Vineet Garg wrote:
> I agree and I tried not to add new tokens but I couldn't figure out how 
> to write grammar in such a way to avoid adding it. 
> 
> For is distinct from we want AST as follows
> KW_NOT
>   EQUAL_NS
> Expr1
> Expr2
>   
> isDistinctFrom rule is invoked from precedenceEqualOperator which is 
> suppose to return an AST for operator. This AST is further used by 
> precedenceEqualOperator's invoker to make an AST with returned AST as root 
> and with two expression as it's children. So if isDistinctFrom return this AST
>   KW_NOT
> EQUAL_NS
> 
>   invoker of precedenceEqualOperator will end up creating
>   KW_NOT
>EQUAL_NS
>Expr1
>Expr2
>
>   which is not what we want.
>   
>  Your above suggestion throws an exception while parsing FAILED: 
> RewriteEmptyStreamException token KW_NOT.
>  
>  I am not sure why

can u try
KW_IS (a=KW_NOT)? KW_DISTINCT KW_FROM
-> {$a !=null}? ^(EQUAL_NS)
-> ^(KW_NOT ^EQUAL_NS)


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58181/#review171125
---


On April 6, 2017, 8:02 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58181/
> ---
> 
> (Updated April 6, 2017, 8:02 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and 
> Pengcheng Xu.
> 
> 
> Bugs: HIVE-15986
> https://issues.apache.org/jira/browse/HIVE-15986
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch adds support for 'is distinct from' and 'is not distinct from'.
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 7a70c9c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ccfb455 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
>  85450c9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g d98a663 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 8c4ee8a 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> f979c14 
>   ql/src/test/queries/clientpositive/is_distinct_from.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/is_distinct_from.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/show_functions.q.out 68e248a 
>   ql/src/test/results/clientpositive/udf_equal.q.out 52bd843 
> 
> 
> Diff: https://reviews.apache.org/r/58181/diff/4/
> 
> 
> Testing
> ---
> 
> Added new tests
> Pre-commit testing
> 
> 
> Thanks,
> 
> Vineet Garg
> 
>

[jira] [Created] (HIVE-16402) Upgrade to Hadoop 2.8.0

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-16402:
---

 Summary: Upgrade to Hadoop 2.8.0
 Key: HIVE-16402
 URL: https://issues.apache.org/jira/browse/HIVE-16402
 Project: Hive
  Issue Type: Bug
Reporter: Sahil Takiar
Assignee: Sahil Takiar


Hadoop 2.8.0 has been out since March, we should upgrade to it. Release notes 
for Hadoop 2.8.x are here: http://hadoop.apache.org/docs/r2.8.0/index.html

It has a number of useful features, improvements for S3 support, ADLS support, 
etc. along with a bunch of other fixes. This should also help us on our way to 
upgrading to Hadoop 3.x (HIVE-15016).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16401) Hook Change Manager to Truncate Table.

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-16401:
---

 Summary: Hook Change Manager to Truncate Table.
 Key: HIVE-16401
 URL: https://issues.apache.org/jira/browse/HIVE-16401
 Project: Hive
  Issue Type: Sub-task
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan


Need to consider the case where insert event happens before truncate table 
which needs to see their data files through change management.
The data files should be recycled to the cmroot path before trashing it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] Apache Hive 1.2.2 Release Candidate 0

[Vaibhav Gumashta]

With 3 binding +1 and one non-binding +1, the vote has passed. Thanks to
everyone for testing. I’ll go ahead and publish the artifacts.

—Vaibhav

On 4/6/17, 8:48 AM, "Sushanth Sowmyan"  wrote:

>+1 (binding)
>
>Verified md5 and asc
>KEYS obtained from hive match (from
>https://people.apache.org/keys/group/hive.asc) , and is publically
>searchable and signed.
>RAT test succeeds
>Source and binary tarballs look good
>Compiling works, some base unit tests succeed.
>Testing local mode works.
>
>On Wed, Apr 5, 2017 at 11:16 PM, Thejas Nair 
>wrote:
>> +1 (binding)
>> - Verified signature and checksum
>> - Build from source
>> - Ran simple queries in local mode with binary tar.gz
>> - Checked RELEASE_NOTES file. Traditionally this file has had the set of
>> patches fixed in previous releases as well ( ie, each new release was
>> adding entries to the top of the file). This time it has only the new
>>patch
>> release patches. The old approach helps to quickly verify if patch is in
>> the release. I think it would be good to fix that in branch. I think it
>>is
>> OK for this release.
>> - README.txt has old 1.2.1 version number in it. IMO, we should just
>>remove
>> the mention of version in that file. Not a release blocker.
>>
>>
>>
>>
>> On Wed, Apr 5, 2017 at 3:52 PM, Sergio Pena 
>> wrote:
>>
>>> +1 (no-binding)
>>>
>>> I unpacked the bin and src packages.
>>> Verified gpg and md5 signatures.
>>> Check license and release notes files.
>>> Run a few queries from hive-cli.
>>>
>>> - Sergio
>>>
>>> On Tue, Apr 4, 2017 at 11:12 AM, Ashutosh Chauhan
>>>
>>> wrote:
>>>
>>> > Verified md5 of src and binary tar balls.
>>> > Built from src.
>>> > Ran some simple queries like join, group by.
>>> > All looks good.
>>> >
>>> > +1
>>> >
>>> > Thanks,
>>> > Ashutosh
>>> >
>>> > On Mon, Apr 3, 2017 at 4:47 PM, Vaibhav Gumashta <
>>> > vgumas...@hortonworks.com>
>>> > wrote:
>>> >
>>> > > Thanks for pointing out Ashutosh. Link to my PGP key:
>>> > > http://pgp.mit.edu/pks/lookup?search=gumashta=index.
>>> > >
>>> > > I think it will take a day or so for the KEYS file to be updated
>>>(it is
>>> > > auto generated), but if you want to test the release in the
>>>meantime,
>>> > > please use the above link to access the signing key.
>>> > >
>>> > > Thanks,
>>> > > ‹Vaibhav
>>> > >
>>> > > On 4/3/17, 2:53 PM, "Ashutosh Chauhan" 
>>>wrote:
>>> > >
>>> > > >Hi Vaibhav,
>>> > > >
>>> > > >Can't locate your key at any of standard location. Can you point
>>>out
>>> > which
>>> > > >key you used to sign the release?
>>> > > >
>>> > > >Thanks,
>>> > > >Ashutosh
>>> > > >
>>> > > >On Mon, Apr 3, 2017 at 12:51 AM, Vaibhav Gumashta
>>> > > >>> > > >> wrote:
>>> > > >> Hi everyone,
>>> > > >>
>>> > > >> Apache Hive 1.2.2 Release Candidate 0 is available here:
>>> > > >>
>>> > > >> 
>>>https://dist.apache.org/repos/dist/dev/hive/apache-hive-1.2.2-rc0/
>>> > > >>
>>> > > >> Maven artifacts are available here:
>>> > > >>
>>> > > >> https://repository.apache.org/content/repositories/
>>> > orgapachehive-1072/
>>> > > >>
>>> > > >> Source tag for RC0 is at:
>>> > > >> https://github.com/apache/hive/releases/tag/release-1.2.2-rc0
>>> > > >>
>>> > > >> Voting will conclude in 72 hours.
>>> > > >>
>>> > > >> Hive PMC Members: Please test and vote.
>>> > > >>
>>> > > >> Thanks,
>>> > > >> -Vaibhav
>>> > > >>
>>> > > >>
>>> > >
>>> > >
>>> >
>>>
>

[jira] [Created] (HIVE-16400) Fix the MDC reference to use slf4j rather than log4j

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-16400:
---

 Summary: Fix the MDC reference to use slf4j rather than log4j
 Key: HIVE-16400
 URL: https://issues.apache.org/jira/browse/HIVE-16400
 Project: Hive
  Issue Type: Sub-task
  Components: Logging
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


The MDC reference in LogUtils is using Log4J version, but we should use slf4j 
version.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58086: HIVE-16334 Query lock contains the query string, which can cause OOM on ZooKeeper

[Chaoyu Tang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58086/#review171222
---


Ship it!




Ship It!

- Chaoyu Tang


On April 6, 2017, 4:12 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58086/
> ---
> 
> (Updated April 6, 2017, 4:12 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Chaoyu Tang, Marta Kuczora, Miklos 
> Csanady, and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-16334
> https://issues.apache.org/jira/browse/HIVE-16334
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The patch contains the following changes:
> - Added a new config variable which defines the maximum length of the query 
> string in the lock object
> - When constructing the HiveLockObject gets the configuration as a new 
> parameter
> - When constructing the HiveLockObject truncates the length of the query 
> string if it is necessary
> - Updates every occurance of the HiveLockObject creation
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6de6ed6 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 53ee9c8 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java fff03df 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java 
> a371a5a 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDummyTxnManager.java 
> de3b8ad 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestEmbeddedLockManager.java 
> 0afbc1c 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java 
> 19cb129 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
>  3f9926e 
> 
> 
> Diff: https://reviews.apache.org/r/58086/diff/4/
> 
> 
> Testing
> ---
> 
> Added new unit test to test the truncation
> Updated the existing unit tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 58086: HIVE-16334 Query lock contains the query string, which can cause OOM on ZooKeeper

[Peter Vary]

> On April 6, 2017, 2:44 p.m., Chaoyu Tang wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 1774 (patched)
> > 
> >
> > The data limit of a znode, which stores the zk lock infomration 
> > including the query string, is 1M. Will it be better to set it (or some 
> > smaller value) as the hive.lock.query.string.max.length default value 
> > instead of Integer.MAX_VALUE?

Thanks for the review!

Changed the default value to 100


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58086/#review171207
---


On April 6, 2017, 4:12 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58086/
> ---
> 
> (Updated April 6, 2017, 4:12 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Chaoyu Tang, Marta Kuczora, Miklos 
> Csanady, and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-16334
> https://issues.apache.org/jira/browse/HIVE-16334
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The patch contains the following changes:
> - Added a new config variable which defines the maximum length of the query 
> string in the lock object
> - When constructing the HiveLockObject gets the configuration as a new 
> parameter
> - When constructing the HiveLockObject truncates the length of the query 
> string if it is necessary
> - Updates every occurance of the HiveLockObject creation
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6de6ed6 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 53ee9c8 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java fff03df 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java 
> a371a5a 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDummyTxnManager.java 
> de3b8ad 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestEmbeddedLockManager.java 
> 0afbc1c 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java 
> 19cb129 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
>  3f9926e 
> 
> 
> Diff: https://reviews.apache.org/r/58086/diff/4/
> 
> 
> Testing
> ---
> 
> Added new unit test to test the truncation
> Updated the existing unit tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 58086: HIVE-16334 Query lock contains the query string, which can cause OOM on ZooKeeper

[Peter Vary]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58086/
---

(Updated April 6, 2017, 4:12 p.m.)


Review request for hive, Aihua Xu, Chaoyu Tang, Marta Kuczora, Miklos Csanady, 
and Barna Zsombor Klara.


Changes
---

Changed the default value from Integer.MAx_VALUE to 100, so it aligns 
better with the data limit of a znode, which is 1MB


Bugs: HIVE-16334
https://issues.apache.org/jira/browse/HIVE-16334


Repository: hive-git


Description
---

The patch contains the following changes:
- Added a new config variable which defines the maximum length of the query 
string in the lock object
- When constructing the HiveLockObject gets the configuration as a new parameter
- When constructing the HiveLockObject truncates the length of the query string 
if it is necessary
- Updates every occurance of the HiveLockObject creation


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6de6ed6 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 53ee9c8 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java fff03df 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java a371a5a 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDummyTxnManager.java 
de3b8ad 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestEmbeddedLockManager.java 
0afbc1c 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java 19cb129 
  
ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
 3f9926e 


Diff: https://reviews.apache.org/r/58086/diff/4/

Changes: https://reviews.apache.org/r/58086/diff/3-4/


Testing
---

Added new unit test to test the truncation
Updated the existing unit tests


Thanks,

Peter Vary

[jira] [Created] (HIVE-16399) created an index for tc_txnid in TXN_COMPONENTS

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-16399:
-

 Summary: created an index for tc_txnid in TXN_COMPONENTS
 Key: HIVE-16399
 URL: https://issues.apache.org/jira/browse/HIVE-16399
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 1.0.0
Reporter: Eugene Koifman
Assignee: Wei Zheng


w/o this TxnStore.cleanEmptyAbortedTxns() can be very slow



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] Apache Hive 1.2.2 Release Candidate 0

[Sushanth Sowmyan]

+1 (binding)

Verified md5 and asc
KEYS obtained from hive match (from
https://people.apache.org/keys/group/hive.asc) , and is publically
searchable and signed.
RAT test succeeds
Source and binary tarballs look good
Compiling works, some base unit tests succeed.
Testing local mode works.

On Wed, Apr 5, 2017 at 11:16 PM, Thejas Nair  wrote:
> +1 (binding)
> - Verified signature and checksum
> - Build from source
> - Ran simple queries in local mode with binary tar.gz
> - Checked RELEASE_NOTES file. Traditionally this file has had the set of
> patches fixed in previous releases as well ( ie, each new release was
> adding entries to the top of the file). This time it has only the new patch
> release patches. The old approach helps to quickly verify if patch is in
> the release. I think it would be good to fix that in branch. I think it is
> OK for this release.
> - README.txt has old 1.2.1 version number in it. IMO, we should just remove
> the mention of version in that file. Not a release blocker.
>
>
>
>
> On Wed, Apr 5, 2017 at 3:52 PM, Sergio Pena 
> wrote:
>
>> +1 (no-binding)
>>
>> I unpacked the bin and src packages.
>> Verified gpg and md5 signatures.
>> Check license and release notes files.
>> Run a few queries from hive-cli.
>>
>> - Sergio
>>
>> On Tue, Apr 4, 2017 at 11:12 AM, Ashutosh Chauhan 
>> wrote:
>>
>> > Verified md5 of src and binary tar balls.
>> > Built from src.
>> > Ran some simple queries like join, group by.
>> > All looks good.
>> >
>> > +1
>> >
>> > Thanks,
>> > Ashutosh
>> >
>> > On Mon, Apr 3, 2017 at 4:47 PM, Vaibhav Gumashta <
>> > vgumas...@hortonworks.com>
>> > wrote:
>> >
>> > > Thanks for pointing out Ashutosh. Link to my PGP key:
>> > > http://pgp.mit.edu/pks/lookup?search=gumashta=index.
>> > >
>> > > I think it will take a day or so for the KEYS file to be updated (it is
>> > > auto generated), but if you want to test the release in the meantime,
>> > > please use the above link to access the signing key.
>> > >
>> > > Thanks,
>> > > ‹Vaibhav
>> > >
>> > > On 4/3/17, 2:53 PM, "Ashutosh Chauhan"  wrote:
>> > >
>> > > >Hi Vaibhav,
>> > > >
>> > > >Can't locate your key at any of standard location. Can you point out
>> > which
>> > > >key you used to sign the release?
>> > > >
>> > > >Thanks,
>> > > >Ashutosh
>> > > >
>> > > >On Mon, Apr 3, 2017 at 12:51 AM, Vaibhav Gumashta
>> > > >> > > >> wrote:
>> > > >> Hi everyone,
>> > > >>
>> > > >> Apache Hive 1.2.2 Release Candidate 0 is available here:
>> > > >>
>> > > >> https://dist.apache.org/repos/dist/dev/hive/apache-hive-1.2.2-rc0/
>> > > >>
>> > > >> Maven artifacts are available here:
>> > > >>
>> > > >> https://repository.apache.org/content/repositories/
>> > orgapachehive-1072/
>> > > >>
>> > > >> Source tag for RC0 is at:
>> > > >> https://github.com/apache/hive/releases/tag/release-1.2.2-rc0
>> > > >>
>> > > >> Voting will conclude in 72 hours.
>> > > >>
>> > > >> Hive PMC Members: Please test and vote.
>> > > >>
>> > > >> Thanks,
>> > > >> -Vaibhav
>> > > >>
>> > > >>
>> > >
>> > >
>> >
>>

Re: ops on #hive IRC channel

[Edward Capriolo]

On Mon, Apr 3, 2017 at 7:40 PM, Edward Capriolo 
wrote:

> I think we should take this opportunity to move to apache-hive. I dont
> think anyone every used this channel for anything. There is many things on
> the internet named hive and naturally it will get spam-ish. We could also
> renew an effort to actually use IRC.
>
> On Fri, Mar 31, 2017 at 8:59 PM, Sergey Shelukhin 
> wrote:
>
>> Does anyone have subj?
>> It would be nice to set the channel to +s (so it doesn’t show up in the
>> channel list), and also ban the bot (or whoever) that keeps joining and
>> posting random links and blurbs (it’s always using the same ident, ~
>> Thunderbi@112.198.78.145, as far as I see in the recent logs).
>> When on IRC, I’d prefer to be able to idle in peace.
>>
>
>
Move to apache-hive?

[14:53:33]  Is anyone in here actually using Hive the
apache software?
[17:20:14] stony [~stony@112.198.102.205] has joined #hive
[17:20:51]  Hayop ka... Hayop ka talaga, hayop!
[17:54:11]  "Oh what aheaven is love, Oh what a hell..."
[17:54:18] stony [~stony@112.198.102.205] has quit IRC: Quit: stony
[18:08:53]  me
[18:09:11]  well, not using, as such, but I run it when making
patches on some test clusters
[18:12:16] stony [~stony@112.198.102.205] has joined #hive
[18:31:19] stony [~stony@112.198.102.205] has quit IRC: Remote host closed
the connection
[18:33:28] stony [~stony@112.198.102.205] has joined #hive
[18:48:19]  I am Stony, I give up...
[18:52:10]  Donald C. Sangil, location: Glan Padidu, Sarangani.
Philippines...
[18:54:35]  Also known as, Archangel Stony. Gabriel in the Holy
Bible...
[18:55:17]  Forgive me.

Re: Review Request 58086: HIVE-16334 Query lock contains the query string, which can cause OOM on ZooKeeper

[Chaoyu Tang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58086/#review171207
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 1774 (patched)


The data limit of a znode, which stores the zk lock infomration including 
the query string, is 1M. Will it be better to set it (or some smaller value) as 
the hive.lock.query.string.max.length default value instead of 
Integer.MAX_VALUE?


- Chaoyu Tang


On April 6, 2017, 6:04 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58086/
> ---
> 
> (Updated April 6, 2017, 6:04 a.m.)
> 
> 
> Review request for hive, Aihua Xu, Chaoyu Tang, Marta Kuczora, Miklos 
> Csanady, and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-16334
> https://issues.apache.org/jira/browse/HIVE-16334
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The patch contains the following changes:
> - Added a new config variable which defines the maximum length of the query 
> string in the lock object
> - When constructing the HiveLockObject gets the configuration as a new 
> parameter
> - When constructing the HiveLockObject truncates the length of the query 
> string if it is necessary
> - Updates every occurance of the HiveLockObject creation
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6de6ed6 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 53ee9c8 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java fff03df 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java 
> a371a5a 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDummyTxnManager.java 
> de3b8ad 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestEmbeddedLockManager.java 
> 0afbc1c 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java 
> 19cb129 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
>  3f9926e 
> 
> 
> Diff: https://reviews.apache.org/r/58086/diff/3/
> 
> 
> Testing
> ---
> 
> Added new unit test to test the truncation
> Updated the existing unit tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 58203: HIVE-16345 BeeLineDriver should be able to run qtest files which are using default database tables

[Peter Vary]

> On April 5, 2017, 9:34 p.m., Yongzhi Chen wrote:
> >

Thanks for the fast review Yongzhi!


> On April 5, 2017, 9:34 p.m., Yongzhi Chen wrote:
> > itests/util/src/main/java/org/apache/hive/beeline/qfile/QFile.java
> > Lines 130 (patched)
> > 
> >
> > How do you handle the case command has comment following ';' and new 
> > command start after ;  ? Do these cases matters?
> > For example:
> > show tables; --comment
> > 
> > show tables; select * from
> > src;
> > 
> > The beeline.Commands class has code similar to getCommands:
> > handleMultiLineCmd, logic in execute
> > Could you figure out a way to use the some of the code there?

Previously I was thinking about this, but dropped the idea because I found some 
differences between the BeeLine query parsing and CLI query parsing.

After your comment I reconsidered, and decided to use the BeeLine version. It 
makes more sense to use the same parsing than the actual BeeLine, and if there 
are differences handle the appropriately.

Thanks for pointing this out!


> On April 5, 2017, 9:34 p.m., Yongzhi Chen wrote:
> > itests/util/src/main/java/org/apache/hive/beeline/qfile/QFile.java
> > Lines 160 (patched)
> > 
> >
> > Is that possible the table belong to other database?
> > For example:
> > use foo;
> > select * from tableinfoo;

You are right, that this can cause a problem. To highlight it, I added a 
warning message.
Luckily this is only a problem, if in the new database there are tables which 
are in the source table list.

The warning message looks like this:

The query file 
/Users/petervary/dev/upstream/hive/ql/src/test/queries/clientpositive/escape_comments.q
 contains "use escape_comments_db;" command
The source table name rewrite is turned on, so this might cause problems when 
the used database contains tables named any of the following: [src_cbo, cbo_t1, 
lineitem, src, cbo_t2, cbo_t3, part, src_thrift, alltypesorc, srcbucket, 
src_json, srcpart, src_hbase, src_sequencefile, src1, srcbucket2]
To turn off the table name rewrite use -Dtest.rewrite.source.tables=false


> On April 5, 2017, 9:34 p.m., Yongzhi Chen wrote:
> > itests/util/src/main/java/org/apache/hive/beeline/qfile/QFileBeeLineClient.java
> > Line 92 (original), 90 (patched)
> > 
> >
> > Why we need to replace the tablename with default.tablename? Could you 
> > just add use default ?

When running multiple tests in parallel there is a possibility that two 
different test try to create tables with the same name.
So we archive the separation by using different databases, and we have put 
extra effort into running the tests with the default tables.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58203/#review171166
---


On April 6, 2017, 9:25 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58203/
> ---
> 
> (Updated April 6, 2017, 9:25 a.m.)
> 
> 
> Review request for hive, Aihua Xu, Zoltan Haindrich, Yongzhi Chen, and Barna 
> Zsombor Klara.
> 
> 
> Bugs: HIVE-16345
> https://issues.apache.org/jira/browse/HIVE-16345
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The goal of the change is to run qtest files which contain queries on tables 
> created by the init scripts.
> It adds the possibility to rewrite the src table references to default.src
> 
> This patch contains the following changes:
> - Added new parameter to the driver, to control weather the rewrite the table 
> names or not (test.rewrite.source.tables) - default is true
> - Made QTestUtil.getSrcTables() available for QFile class
> - Run the QFile not with "!run testfile.q", but reading the file, and 
> assembling the commands - enable us to parse the queries, and provide better 
> feedback about the failing queries
> - QFile rewrites the source tables, if it is required
> - Used 9 qtest files from the CliDriver, and added them to BeeLine tests
> - Added new filters, and removed redundant ones - I was able to remove every 
> QFile specific filter, and corresponding setter methods as well
> - Moved QFile classes to org.apache.hive.beeline package, so it can use 
> package private methods from BeeLine, and Commands
> - Refactored needsContinuation method in BeeLine, so it can be called from a 
> static context as well
> 
> And one important change is:
> - In Utilities.setMapRedWork, change the INPUT_NAME value in the conf to a 
> mapreduce task specific value. This one 

Re: Review Request 58203: HIVE-16345 BeeLineDriver should be able to run qtest files which are using default database tables

[Peter Vary]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58203/
---

(Updated April 6, 2017, 9:25 a.m.)


Review request for hive, Aihua Xu, Zoltan Haindrich, Yongzhi Chen, and Barna 
Zsombor Klara.


Changes
---

Instead of the needsContinuation method refactor moved the getCommands to 
BeeLine, so we can reuse the whole file parsing algorithm in the tests.
Modified the test classes to use the BeeLine.getCommands method
Added a warning message, when use database command is used in the test script


Bugs: HIVE-16345
https://issues.apache.org/jira/browse/HIVE-16345


Repository: hive-git


Description
---

The goal of the change is to run qtest files which contain queries on tables 
created by the init scripts.
It adds the possibility to rewrite the src table references to default.src

This patch contains the following changes:
- Added new parameter to the driver, to control weather the rewrite the table 
names or not (test.rewrite.source.tables) - default is true
- Made QTestUtil.getSrcTables() available for QFile class
- Run the QFile not with "!run testfile.q", but reading the file, and 
assembling the commands - enable us to parse the queries, and provide better 
feedback about the failing queries
- QFile rewrites the source tables, if it is required
- Used 9 qtest files from the CliDriver, and added them to BeeLine tests
- Added new filters, and removed redundant ones - I was able to remove every 
QFile specific filter, and corresponding setter methods as well
- Moved QFile classes to org.apache.hive.beeline package, so it can use package 
private methods from BeeLine, and Commands
- Refactored needsContinuation method in BeeLine, so it can be called from a 
static context as well

And one important change is:
- In Utilities.setMapRedWork, change the INPUT_NAME value in the conf to a 
mapreduce task specific value. This one is used by the IOContextMap to cache 
the IOContext objects. Using the same value for every mapred task prevented 
them to run in the same JVM. The test were running sequencially, but failed 
randomly in parallel


Diffs (updated)
-

  beeline/src/java/org/apache/hive/beeline/BeeLine.java 11526a7 
  beeline/src/java/org/apache/hive/beeline/Commands.java 2578728 
  itests/src/test/resources/testconfiguration.properties 7a70c9c 
  
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreBeeLineDriver.java
 0d63f5d 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 2abf252 
  itests/util/src/main/java/org/apache/hive/beeline/qfile/QFile.java ae5a349 
  
itests/util/src/main/java/org/apache/hive/beeline/qfile/QFileBeeLineClient.java 
760fde6 
  itests/util/src/main/java/org/apache/hive/beeline/qfile/package-info.java 
fcd50ec 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 79955e9 
  ql/src/test/results/clientpositive/beeline/drop_with_concurrency.q.out 
385f9b7 
  ql/src/test/results/clientpositive/beeline/escape_comments.q.out abc0fee 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_10.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_11.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_12.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_13.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_16.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_3.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/58203/diff/2/

Changes: https://reviews.apache.org/r/58203/diff/1-2/


Testing
---

Run the test multiple times with the various combinations of the following 
parameters:
- test.rewrite.source.tables - runs with true, or without it, fails when set to 
false
- junit.parallel.threads - runs with 1, or without this parameter


Thanks,

Peter Vary

[jira] [Created] (HIVE-16398) [Hive on Tez] Jar Localization issues

[Ravi Teja Chilukuri (JIRA)]

Ravi Teja Chilukuri created HIVE-16398:
--

 Summary: [Hive on Tez] Jar Localization issues
 Key: HIVE-16398
 URL: https://issues.apache.org/jira/browse/HIVE-16398
 Project: Hive
  Issue Type: Bug
  Components: Tez
Affects Versions: 2.1.0
Reporter: Ravi Teja Chilukuri
Priority: Minor


The jar localization fails with the following error in these cases

1) If a jar is already present in the classpath and is added again via aux.jars 
or using add jar command by the user.
2) If the jar is missing- Ideally it should ignore or either throw a 
error/warning stating jar not found.
3) When a  "*" is present in some of the classpath entries, it will try to 
replicate the same * path on hdfs and fails.  



Error:
Total jobs = 1
Launching Job 1 out of 1
Exception in thread "Thread-17" java.lang.RuntimeException: 
java.io.IOException: Previous writer likely failed to write 
hdfs://clstr1/tmp/hive/adhoc-hive-user/_tez_session_dir/edd821a7-d4f5-4980-b693-6b9c17ac4257/hadoop-4mc-1.1.0.jar.
 Failing because I am unlikely to write too.
at 
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:617)
at 
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:526)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:78)
Caused by: java.io.IOException: Previous writer likely failed to write 
hdfs://clstr1/tmp/hive/adhoc-hive-user/_tez_session_dir/edd821a7-d4f5-4980-b693-6b9c17ac4257/hadoop-4mc-1.1.0.jar.
 Failing because I am unlikely to write too.
at 
org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1020)
at 
org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:900)
at 
org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:843)
at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:451)
at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:247)
at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:222)
at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:189)
at 
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:612)
... 2 more
FAILED: Execution Error, return code -1 from 
org.apache.hadoop.hive.ql.exec.tez.TezTask



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16397) alter table concatenate should check the state of folder before replacing it

[Xintong Bian (JIRA)]

Xintong Bian created HIVE-16397:
---

 Summary: alter table concatenate should check the state of folder 
before replacing it
 Key: HIVE-16397
 URL: https://issues.apache.org/jira/browse/HIVE-16397
 Project: Hive
  Issue Type: Improvement
Reporter: Xintong Bian


alter table concatenate command will remove all files containing in a directory 
after the success of the merge job, while in an external table, new data can 
arrive after the launch of the merge job, but they are also deleted in the 
replacing phase. 
We should check the last modified time of the directory, in case it is modified 
after the launch of merge job, abort the replace



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] Apache Hive 1.2.2 Release Candidate 0

[Thejas Nair]

+1 (binding)
- Verified signature and checksum
- Build from source
- Ran simple queries in local mode with binary tar.gz
- Checked RELEASE_NOTES file. Traditionally this file has had the set of
patches fixed in previous releases as well ( ie, each new release was
adding entries to the top of the file). This time it has only the new patch
release patches. The old approach helps to quickly verify if patch is in
the release. I think it would be good to fix that in branch. I think it is
OK for this release.
- README.txt has old 1.2.1 version number in it. IMO, we should just remove
the mention of version in that file. Not a release blocker.




On Wed, Apr 5, 2017 at 3:52 PM, Sergio Pena 
wrote:

> +1 (no-binding)
>
> I unpacked the bin and src packages.
> Verified gpg and md5 signatures.
> Check license and release notes files.
> Run a few queries from hive-cli.
>
> - Sergio
>
> On Tue, Apr 4, 2017 at 11:12 AM, Ashutosh Chauhan 
> wrote:
>
> > Verified md5 of src and binary tar balls.
> > Built from src.
> > Ran some simple queries like join, group by.
> > All looks good.
> >
> > +1
> >
> > Thanks,
> > Ashutosh
> >
> > On Mon, Apr 3, 2017 at 4:47 PM, Vaibhav Gumashta <
> > vgumas...@hortonworks.com>
> > wrote:
> >
> > > Thanks for pointing out Ashutosh. Link to my PGP key:
> > > http://pgp.mit.edu/pks/lookup?search=gumashta=index.
> > >
> > > I think it will take a day or so for the KEYS file to be updated (it is
> > > auto generated), but if you want to test the release in the meantime,
> > > please use the above link to access the signing key.
> > >
> > > Thanks,
> > > ‹Vaibhav
> > >
> > > On 4/3/17, 2:53 PM, "Ashutosh Chauhan"  wrote:
> > >
> > > >Hi Vaibhav,
> > > >
> > > >Can't locate your key at any of standard location. Can you point out
> > which
> > > >key you used to sign the release?
> > > >
> > > >Thanks,
> > > >Ashutosh
> > > >
> > > >On Mon, Apr 3, 2017 at 12:51 AM, Vaibhav Gumashta
> > > > > > >> wrote:
> > > >> Hi everyone,
> > > >>
> > > >> Apache Hive 1.2.2 Release Candidate 0 is available here:
> > > >>
> > > >> https://dist.apache.org/repos/dist/dev/hive/apache-hive-1.2.2-rc0/
> > > >>
> > > >> Maven artifacts are available here:
> > > >>
> > > >> https://repository.apache.org/content/repositories/
> > orgapachehive-1072/
> > > >>
> > > >> Source tag for RC0 is at:
> > > >> https://github.com/apache/hive/releases/tag/release-1.2.2-rc0
> > > >>
> > > >> Voting will conclude in 72 hours.
> > > >>
> > > >> Hive PMC Members: Please test and vote.
> > > >>
> > > >> Thanks,
> > > >> -Vaibhav
> > > >>
> > > >>
> > >
> > >
> >
>

Re: Review Request 58086: HIVE-16334 Query lock contains the query string, which can cause OOM on ZooKeeper

[Peter Vary]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58086/
---

(Updated April 6, 2017, 6:04 a.m.)


Review request for hive, Aihua Xu, Chaoyu Tang, Marta Kuczora, Miklos Csanady, 
and Barna Zsombor Klara.


Changes
---

Resetting the default value in configuration before testing the truncation, to 
avoid future flakyness


Bugs: HIVE-16334
https://issues.apache.org/jira/browse/HIVE-16334


Repository: hive-git


Description
---

The patch contains the following changes:
- Added a new config variable which defines the maximum length of the query 
string in the lock object
- When constructing the HiveLockObject gets the configuration as a new parameter
- When constructing the HiveLockObject truncates the length of the query string 
if it is necessary
- Updates every occurance of the HiveLockObject creation


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6de6ed6 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 53ee9c8 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java fff03df 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java a371a5a 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDummyTxnManager.java 
de3b8ad 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestEmbeddedLockManager.java 
0afbc1c 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java 19cb129 
  
ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
 3f9926e 


Diff: https://reviews.apache.org/r/58086/diff/3/

Changes: https://reviews.apache.org/r/58086/diff/2-3/


Testing
---

Added new unit test to test the truncation
Updated the existing unit tests


Thanks,

Peter Vary