from:"Alexander Pivovarov"

str_to_map UDF description fix

2017-01-31 Thread Alexander Pivovarov

Hello Everyone

Can anyone look at str_to_map UDF description fix?

https://issues.apache.org/jira/browse/HIVE-15746

Thank you
Alex

[jira] [Created] (HIVE-15746) Fix default delimiter2 in str_to_map UDF or in method description

2017-01-27 Thread Alexander Pivovarov (JIRA)

Alexander Pivovarov created HIVE-15746:
--

 Summary: Fix default delimiter2 in str_to_map UDF or in method 
description
 Key: HIVE-15746
 URL: https://issues.apache.org/jira/browse/HIVE-15746
 Project: Hive
  Issue Type: Bug
  Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
Priority: Trivial


According to UDF wiki and to GenericUDFStringToMap.java class comments default 
delimiter 2 should be '='.

But in the code default_del2 = ":"
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFStringToMap.java#L53

We need to fix code or fix the method description and UDF wiki

Let me know what you think?

{code}
str_to_map("a=1,b=2")
vs
str_to_map("a:1,b:2")
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Hadoop-1 is dead, long live Hadoop

2015-10-26 Thread Alexander Pivovarov

Awesome!

Looks like cloud.databricks.com run Spark-1.5 clusters with hive-1.2.1 and
hadoop-1 (lol). I guess they need to move to hadoop-2
> org.apache.hadoop.util.VersionInfo.getVersion
res8: String = 1.2.2-SNAPSHOT

On Mon, Oct 26, 2015 at 3:02 PM, Alan Gates  wrote:

> I've committed HIVE-11378 to master, which means Hadoop-1 is no longer
> supported in Hive's master branch.  When building it is no longer necessary
> to say -Phadoop-2.  No profile is needed for regular builds.
>
> Alan.
>

Re: Review Request 37706: HIVE-11593 Add aes_encrypt and aes_decrypt UDFs

2015-08-26 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37706/
---

(Updated Aug. 27, 2015, 1:53 a.m.)


Review request for hive and Jason Dere.


Changes
---

move Cipher.getInstance to initialize


Bugs: HIVE-11593
https://issues.apache.org/jira/browse/HIVE-11593


Repository: hive-git


Description
---

HIVE-11593 Add aes_encrypt and aes_decrypt UDFs


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
4c1c53eff84f7e855fcd344aba16012fc1b31d1a 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAesBase.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAesDecrypt.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAesEncrypt.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFParamUtils.java 
cdbc6eae96c850ed46806e79c6f661c0f406d0e8 
  
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFAesDecrypt.java 
PRE-CREATION 
  
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFAesEncrypt.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_aes_decrypt.q PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_aes_encrypt.q PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 
540079bc8787547cafe747f5855f41f5a829177c 
  ql/src/test/results/clientpositive/udf_aes_decrypt.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/udf_aes_encrypt.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/37706/diff/


Testing
---


Thanks,

Alexander Pivovarov

Review Request 37706: HIVE-11593 Add aes_encrypt and aes_decrypt UDFs

2015-08-23 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37706/
---

Review request for hive and Jason Dere.


Bugs: HIVE-11593
https://issues.apache.org/jira/browse/HIVE-11593


Repository: hive-git


Description
---

HIVE-11593 Add aes_encrypt and aes_decrypt UDFs


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
4c1c53eff84f7e855fcd344aba16012fc1b31d1a 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAesBase.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAesDecrypt.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAesEncrypt.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFParamUtils.java 
cdbc6eae96c850ed46806e79c6f661c0f406d0e8 
  
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFAesDecrypt.java 
PRE-CREATION 
  
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFAesEncrypt.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_aes_decrypt.q PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_aes_encrypt.q PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 
540079bc8787547cafe747f5855f41f5a829177c 
  ql/src/test/results/clientpositive/udf_aes_decrypt.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/udf_aes_encrypt.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/37706/diff/


Testing
---


Thanks,

Alexander Pivovarov

[jira] [Created] (HIVE-11593) Add aes_encrypt and aes_decrypt UDFs

2015-08-18 Thread Alexander Pivovarov (JIRA)

Alexander Pivovarov created HIVE-11593:
--

 Summary: Add aes_encrypt and aes_decrypt UDFs
 Key: HIVE-11593
 URL: https://issues.apache.org/jira/browse/HIVE-11593
 Project: Hive
  Issue Type: New Feature
  Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov


AES (Advanced Encryption Standard) algorithm.

Oracle JRE supports AES-128 out of the box
AES-192 and AES-256 are supported if Cryptography Extension (JCE) Unlimited 
Strength Jurisdiction Policy Files installed




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 35902: HIVE-11103 Add banker's rounding BROUND UDF

2015-08-10 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35902/
---

(Updated Aug. 10, 2015, 7:14 a.m.)


Review request for hive and Jason Dere.


Changes
---

add non-default constructor to FuncRoundWithNumDigitsDecimalToDecimal


Bugs: HIVE-11103
https://issues.apache.org/jira/browse/HIVE-11103


Repository: hive-git


Description
---

HIVE-11103 Add banker's rounding BROUND UDF


Diffs (updated)
-

  ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java 
54f3783d94a139c0ccee77dc91657cfe59f81ba9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
fb06d446a45cbf22c2e8b65e7cb28a9005049627 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/BRoundWithNumDigitsDoubleToDouble.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/DecimalUtil.java 
ef800596deed612b525ed3371b196f275ad88e09 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FuncBRoundWithNumDigitsDecimalToDecimal.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FuncRoundWithNumDigitsDecimalToDecimal.java
 9f3e8a3fcacb17990c6644a67cf587ae9948adad 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/MathExpr.java 
aef923e2c362a8d15b8dcc3467aef01a862c205c 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
82c3e5032f5bf81cde4b52c91f71e0525728812c 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBRound.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRound.java 
963e4a87c417798f95bb1490a4275339a61e869c 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/RoundUtils.java 
0b389a5783fa2cf6643919c411ee57a7ed873d84 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorizationContext.java 
98a8c3eb61f97fdaa44fe11c81c884146bac1d45 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFBRound.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_bround.q PRE-CREATION 
  ql/src/test/queries/clientpositive/vector_bround.q PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 
5de4ffcd1ace477af026b83fb7bfb8068fc192b3 
  ql/src/test/results/clientpositive/udf_bround.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/vector_bround.q.out PRE-CREATION 
  storage-api/src/java/org/apache/hadoop/hive/common/type/HiveDecimal.java 
7d7fb2841bc985c62ecd1781bbde5380647b81ca 

Diff: https://reviews.apache.org/r/35902/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: Review Request 35902: HIVE-11103 Add banker's rounding BROUND UDF

2015-08-10 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35902/
---

(Updated Aug. 10, 2015, 7:05 a.m.)


Review request for hive and Jason Dere.


Changes
---

added non-default constructor to BRoundWithNumDigitsDoubleToDouble, added 
BRound to TestVectorizationContext, fixed vector_bround.q.out


Bugs: HIVE-11103
https://issues.apache.org/jira/browse/HIVE-11103


Repository: hive-git


Description
---

HIVE-11103 Add banker's rounding BROUND UDF


Diffs (updated)
-

  ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java 
54f3783d94a139c0ccee77dc91657cfe59f81ba9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
fb06d446a45cbf22c2e8b65e7cb28a9005049627 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/BRoundWithNumDigitsDoubleToDouble.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/DecimalUtil.java 
ef800596deed612b525ed3371b196f275ad88e09 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FuncBRoundWithNumDigitsDecimalToDecimal.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FuncRoundWithNumDigitsDecimalToDecimal.java
 9f3e8a3fcacb17990c6644a67cf587ae9948adad 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/MathExpr.java 
aef923e2c362a8d15b8dcc3467aef01a862c205c 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
82c3e5032f5bf81cde4b52c91f71e0525728812c 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBRound.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRound.java 
963e4a87c417798f95bb1490a4275339a61e869c 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/RoundUtils.java 
0b389a5783fa2cf6643919c411ee57a7ed873d84 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorizationContext.java 
98a8c3eb61f97fdaa44fe11c81c884146bac1d45 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFBRound.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_bround.q PRE-CREATION 
  ql/src/test/queries/clientpositive/vector_bround.q PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 
5de4ffcd1ace477af026b83fb7bfb8068fc192b3 
  ql/src/test/results/clientpositive/udf_bround.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/vector_bround.q.out PRE-CREATION 
  storage-api/src/java/org/apache/hadoop/hive/common/type/HiveDecimal.java 
7d7fb2841bc985c62ecd1781bbde5380647b81ca 

Diff: https://reviews.apache.org/r/35902/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: Review Request 35902: HIVE-11103 Add banker's rounding BROUND UDF

2015-07-27 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35902/
---

(Updated July 27, 2015, 6:12 a.m.)


Review request for hive and Jason Dere.


Changes
---

added vector_bround.q, added testMathExprBround


Bugs: HIVE-11103
https://issues.apache.org/jira/browse/HIVE-11103


Repository: hive-git


Description
---

HIVE-11103 Add banker's rounding BROUND UDF


Diffs (updated)
-

  ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java 
6485a2ac5f12dbdba7bdf4d17ba18ad054c6f73b 
  common/src/java/org/apache/hadoop/hive/common/type/HiveDecimal.java 
a8215f29aed3a0399ec274cc311a3c92e0cca55b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
fabc21e2092561cbf98c35a406e4ee40e71fe1de 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/BRoundWithNumDigitsDoubleToDouble.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/DecimalUtil.java 
ef800596deed612b525ed3371b196f275ad88e09 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FuncBRoundWithNumDigitsDecimalToDecimal.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FuncRoundWithNumDigitsDecimalToDecimal.java
 9f3e8a3fcacb17990c6644a67cf587ae9948adad 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/MathExpr.java 
aef923e2c362a8d15b8dcc3467aef01a862c205c 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBRound.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRound.java 
963e4a87c417798f95bb1490a4275339a61e869c 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/RoundUtils.java 
0b389a5783fa2cf6643919c411ee57a7ed873d84 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFBRound.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_bround.q PRE-CREATION 
  ql/src/test/queries/clientpositive/vector_bround.q PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 
5de4ffcd1ace477af026b83fb7bfb8068fc192b3 
  ql/src/test/results/clientpositive/udf_bround.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/vector_bround.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/35902/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: Review Request 35902: HIVE-11103 Add banker's rounding BROUND UDF

2015-07-27 Thread Alexander Pivovarov



 On July 2, 2015, 9:10 p.m., Jason Dere wrote:
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/MathExpr.java,
   line 38
  https://reviews.apache.org/r/35902/diff/1/?file=992839#file992839line38
 
  Can you add a JUnit test somewhere to test that the behavior of 
  MathExpr.bround(x) == RoundUtils.bround(x, 0)?

added TestGenericUDFBRound.testMathExprBround


 On July 2, 2015, 9:10 p.m., Jason Dere wrote:
  ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java, line 756
  https://reviews.apache.org/r/35902/diff/1/?file=992832#file992832line756
 
  If you're going to add a vectorized version of the function, can you 
  add a q-file test to test the vectorized bround()?

added vector_bround.q


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35902/#review90283
---


On July 27, 2015, 6:12 a.m., Alexander Pivovarov wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/35902/
 ---
 
 (Updated July 27, 2015, 6:12 a.m.)
 
 
 Review request for hive and Jason Dere.
 
 
 Bugs: HIVE-11103
 https://issues.apache.org/jira/browse/HIVE-11103
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 HIVE-11103 Add banker's rounding BROUND UDF
 
 
 Diffs
 -
 
   ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java 
 6485a2ac5f12dbdba7bdf4d17ba18ad054c6f73b 
   common/src/java/org/apache/hadoop/hive/common/type/HiveDecimal.java 
 a8215f29aed3a0399ec274cc311a3c92e0cca55b 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
 fabc21e2092561cbf98c35a406e4ee40e71fe1de 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/BRoundWithNumDigitsDoubleToDouble.java
  PRE-CREATION 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/DecimalUtil.java
  ef800596deed612b525ed3371b196f275ad88e09 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FuncBRoundWithNumDigitsDecimalToDecimal.java
  PRE-CREATION 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FuncRoundWithNumDigitsDecimalToDecimal.java
  9f3e8a3fcacb17990c6644a67cf587ae9948adad 
   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/MathExpr.java 
 aef923e2c362a8d15b8dcc3467aef01a862c205c 
   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBRound.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRound.java 
 963e4a87c417798f95bb1490a4275339a61e869c 
   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/RoundUtils.java 
 0b389a5783fa2cf6643919c411ee57a7ed873d84 
   ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFBRound.java 
 PRE-CREATION 
   ql/src/test/queries/clientpositive/udf_bround.q PRE-CREATION 
   ql/src/test/queries/clientpositive/vector_bround.q PRE-CREATION 
   ql/src/test/results/clientpositive/show_functions.q.out 
 5de4ffcd1ace477af026b83fb7bfb8068fc192b3 
   ql/src/test/results/clientpositive/udf_bround.q.out PRE-CREATION 
   ql/src/test/results/clientpositive/vector_bround.q.out PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/35902/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Alexander Pivovarov

Re: Review Request 35963: HIVE-7150 FileInputStream is not closed in HiveConnection class

2015-06-27 Thread Alexander Pivovarov



 On June 27, 2015, 1:56 p.m., Xuefu Zhang wrote:
  Ship It!

Thank you Xuefu! Could you leave +1 in the Jira comments


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35963/#review89624
---


On June 27, 2015, 3:23 a.m., Alexander Pivovarov wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/35963/
 ---
 
 (Updated June 27, 2015, 3:23 a.m.)
 
 
 Review request for hive, Ashutosh Chauhan and Thejas Nair.
 
 
 Bugs: HIVE-7150
 https://issues.apache.org/jira/browse/HIVE-7150
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 HIVE-7150 FileInputStream is not closed in HiveConnection class
 
 
 Diffs
 -
 
   jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 
 277f6d44d3b3472218526a2933380420676729a3 
   jdbc/src/java/org/apache/hive/jdbc/Utils.java 
 0e4693b34544fcc4845a7ed0b16dc5299a06455a 
 
 Diff: https://reviews.apache.org/r/35963/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Alexander Pivovarov

Re: Review Request 35968: 1. Added preliminary UDF code for cosine similarity. 2. Added unit tests and integration tests. 3. Registered the UDF in the FunctionRegistry class.

2015-06-27 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35968/#review89635
---



.reviewboardrc 
https://reviews.apache.org/r/35968/#comment142287

this file should not be in the patch



ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
https://reviews.apache.org/r/35968/#comment142288

Try to minimize the changes in existing classes.
I do not think we shoul replace list of imports with *



ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCosineSimilarity.java (line 4)
https://reviews.apache.org/r/35968/#comment142297

I can not find import org.apache.commons.math3 in other classes. I'm not 
sure hive-exec has explicit dependency on commons-math3 jar



ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCosineSimilarity.java (line 12)
https://reviews.apache.org/r/35968/#comment142289

do not use * for import



ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCosineSimilarity.java (line 26)
https://reviews.apache.org/r/35968/#comment142290

it should be full stop at the end. remove trailing space



ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCosineSimilarity.java (line 28)
https://reviews.apache.org/r/35968/#comment142291

I do not think we need new line in the middle of the sentence.



ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCosineSimilarity.java (line 29)
https://reviews.apache.org/r/35968/#comment142292

Can you add Example?



ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCosineSimilarity.java (line 32)
https://reviews.apache.org/r/35968/#comment142293

why final?



ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCosineSimilarity.java (line 39)
https://reviews.apache.org/r/35968/#comment142294

add check for null and return null. In most cases Hive UDFs do not throw 
exception if args are null. UDF should just return null.



ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCosineSimilarity.java (line 47)
https://reviews.apache.org/r/35968/#comment142295

Use class field FloatWritable and use set() method instead of creating new 
FloatWritable on evry wor



ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCosineSimilarity.java (line 50)
https://reviews.apache.org/r/35968/#comment142296

why private? I recommend to use protected to have an ability to extend your 
UDF in future and create another UDF with slightly different behavious



ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCosineSimilarity.java (line 60)
https://reviews.apache.org/r/35968/#comment142298

Why you cast value to float in the middle of the calculation.
I recommend to use double internally and convert final result of 
calculation to float



ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCosineSimilarity.java (line 63)
https://reviews.apache.org/r/35968/#comment142301

Probably it's better to make it protected



ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCosineSimilarity.java (line 72)
https://reviews.apache.org/r/35968/#comment142300

Hive UDF should return null in case input data is null or invalid



ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCosineSimilarity.java (line 81)
https://reviews.apache.org/r/35968/#comment142306

It can be top level class or at least public static



ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCosineSimilarity.java (line 88)
https://reviews.apache.org/r/35968/#comment142304

UDF should not throw exceptions in evaluate method



ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCosineSimilarity.java (line 93)
https://reviews.apache.org/r/35968/#comment142307

forgot generic. Probably it should be HashSetString



ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCosineSimilarity.java (line 109)
https://reviews.apache.org/r/35968/#comment142303

UDF should not throw exceptions in evaluate method



ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCosineSimilarity.java (line 122)
https://reviews.apache.org/r/35968/#comment142302

UDF should not throw exceptions in evaluate method


- Alexander Pivovarov


On June 27, 2015, 8:32 p.m., Nishant Kelkar wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/35968/
 ---
 
 (Updated June 27, 2015, 8:32 p.m.)
 
 
 Review request for hive and Alexander Pivovarov.
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 1. Added preliminary UDF code for cosine similarity. 2. Added unit tests and 
 integration tests. 3. Registered the UDF in the FunctionRegistry class.
 
 
 Diffs
 -
 
   .reviewboardrc abc33f91a44b76573cbba334c33417307c63956f 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
 fabc21e2092561cbf98c35a406e4ee40e71fe1de 
   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCosineSimilarity.java 
 PRE-CREATION 
   ql/src/test/org/apache/hadoop/hive/ql/udf

Re: Review Request 35963: HIVE-7150 FileInputStream is not closed in HiveConnection class

2015-06-27 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35963/
---

(Updated June 27, 2015, 8:17 p.m.)


Review request for hive, Ashutosh Chauhan and Thejas Nair.


Changes
---

using try-with-resources


Bugs: HIVE-7150
https://issues.apache.org/jira/browse/HIVE-7150


Repository: hive-git


Description
---

HIVE-7150 FileInputStream is not closed in HiveConnection class


Diffs (updated)
-

  jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 
277f6d44d3b3472218526a2933380420676729a3 

Diff: https://reviews.apache.org/r/35963/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: Review Request 35964: HIVE-7180 BufferedReader is not closed in MetaStoreSchemaInfo

2015-06-27 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35964/
---

(Updated June 27, 2015, 8:09 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

using java 7 try-with-resources


Bugs: HIVE-7180
https://issues.apache.org/jira/browse/HIVE-7180


Repository: hive-git


Description
---

HIVE-7180 BufferedReader is not closed in MetaStoreSchemaInfo


Diffs (updated)
-

  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java 
a5eed6d3c0ff4a229f3209175b8dfcf8bde5e79e 

Diff: https://reviews.apache.org/r/35964/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: Review Request 35963: HIVE-7150 FileInputStream is not closed in HiveConnection class

2015-06-27 Thread Alexander Pivovarov



 On June 28, 2015, 3:30 a.m., Xuefu Zhang wrote:
  jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java, line 394
  https://reviews.apache.org/r/35963/diff/2/?file=993962#file993962line394
 
  This is nice, but will break java 6 build. Though, I'm not sure if jdk6 
  support is dropped from hive yet.

HIVE-9521 Drop support for Java6 was Resolved on Feb 3 in 1.2.0


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35963/#review89652
---


On June 27, 2015, 8:17 p.m., Alexander Pivovarov wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/35963/
 ---
 
 (Updated June 27, 2015, 8:17 p.m.)
 
 
 Review request for hive, Ashutosh Chauhan and Thejas Nair.
 
 
 Bugs: HIVE-7150
 https://issues.apache.org/jira/browse/HIVE-7150
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 HIVE-7150 FileInputStream is not closed in HiveConnection class
 
 
 Diffs
 -
 
   jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 
 277f6d44d3b3472218526a2933380420676729a3 
 
 Diff: https://reviews.apache.org/r/35963/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Alexander Pivovarov

Re: Review Request 35964: HIVE-7180 BufferedReader is not closed in MetaStoreSchemaInfo

2015-06-27 Thread Alexander Pivovarov



 On June 28, 2015, 3:32 a.m., Xuefu Zhang wrote:
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java,
   line 67
  https://reviews.apache.org/r/35964/diff/2/?file=993961#file993961line67
 
  This is nice, but will break java 6 build. Though, I'm not sure if jdk6 
  support is dropped from hive yet.

HIVE-9521 Drop support for Java6 was Resolved on Feb 3 in 1.2.0


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35964/#review89653
---


On June 27, 2015, 8:09 p.m., Alexander Pivovarov wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/35964/
 ---
 
 (Updated June 27, 2015, 8:09 p.m.)
 
 
 Review request for hive and Ashutosh Chauhan.
 
 
 Bugs: HIVE-7180
 https://issues.apache.org/jira/browse/HIVE-7180
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 HIVE-7180 BufferedReader is not closed in MetaStoreSchemaInfo
 
 
 Diffs
 -
 
   
 metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java 
 a5eed6d3c0ff4a229f3209175b8dfcf8bde5e79e 
 
 Diff: https://reviews.apache.org/r/35964/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Alexander Pivovarov

Re: [ANNOUNCE] New Hive Committers - Jesus Camacho Rodriguez and Chinna Rao Lalam

2015-06-26 Thread Alexander Pivovarov

Congrats!
On Jun 26, 2015 1:36 PM, Hari Subramaniyan hsubramani...@hortonworks.com
wrote:

 Congrats Jesús and Chinna!

 From: Chao Sun c...@cloudera.com
 Sent: Friday, June 26, 2015 12:06 PM
 To: dev@hive.apache.org
 Cc: jcama...@apache.org; Lalam, Chinna R
 Subject: Re: [ANNOUNCE] New Hive Committers - Jesus Camacho Rodriguez and
 Chinna Rao Lalam

 Good job! Congratulations!

 On Fri, Jun 26, 2015 at 12:02 PM, Jimmy Xiang jxi...@cloudera.com wrote:

  Congrats!!

  On Fri, Jun 26, 2015 at 12:00 PM, Xuefu Zhang xzh...@cloudera.com
 wrote:

   Congratulations, Chinna and Jesus! Impressive!

   --Xuefu

   On Fri, Jun 26, 2015 at 11:57 AM, Carl Steinbach c...@apache.org
 wrote:

On behalf of the Apache Hive PMC I am pleased to announce that Jesus
Camacho Rodriguez and Chinna Rao Lalam have been voted in as
  committers.

Please join me in congratulating Jesus and Chinna!

Thanks.

- Carl

Review Request 35963: HIVE-7150 FileInputStream is not closed in HiveConnection class

2015-06-26 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35963/
---

Review request for hive, Ashutosh Chauhan and Thejas Nair.


Bugs: HIVE-7150
https://issues.apache.org/jira/browse/HIVE-7150


Repository: hive-git


Description
---

HIVE-7150 FileInputStream is not closed in HiveConnection class


Diffs
-

  jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 
277f6d44d3b3472218526a2933380420676729a3 
  jdbc/src/java/org/apache/hive/jdbc/Utils.java 
0e4693b34544fcc4845a7ed0b16dc5299a06455a 

Diff: https://reviews.apache.org/r/35963/diff/


Testing
---


Thanks,

Alexander Pivovarov

Review Request 35964: HIVE-7180 BufferedReader is not closed in MetaStoreSchemaInfo

2015-06-26 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35964/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-7180
https://issues.apache.org/jira/browse/HIVE-7180


Repository: hive-git


Description
---

HIVE-7180 BufferedReader is not closed in MetaStoreSchemaInfo


Diffs
-

  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java 
a5eed6d3c0ff4a229f3209175b8dfcf8bde5e79e 

Diff: https://reviews.apache.org/r/35964/diff/


Testing
---


Thanks,

Alexander Pivovarov

[jira] [Created] (HIVE-11116) Can not select data from table which points to remote hdfs location

2015-06-25 Thread Alexander Pivovarov (JIRA)

Alexander Pivovarov created HIVE-6:
--

 Summary: Can not select data from table which points to remote 
hdfs location
 Key: HIVE-6
 URL: https://issues.apache.org/jira/browse/HIVE-6
 Project: Hive
  Issue Type: Bug
  Components: Encryption
Reporter: Alexander Pivovarov


I tried to create new table which points to remote hdfs location and select 
data from it.

It works for hive-0.14 and hive-1.0  but it does not work starting from hive-1.1

to reproduce the issue
1. create folder on remote hdfs
{code}
hadoop fs -mkdir -p hdfs://remote-nn/tmp/et1
{code}

2. create table 
{code}
CREATE TABLE et1 (
  a string
) stored as textfile
LOCATION 'hdfs://remote-nn/tmp/et1';
{code}

3. run select
{code}
select * from et1 limit 10;
{code}

4. Should get the following error
{code}
select * from et1;
15/06/25 13:43:44 [main]: ERROR parse.CalcitePlanner: 
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to determine if 
hdfs://remote_nn/tmp/et1is encrypted: java.lang.IllegalArgumentException: Wrong 
FS: hdfs://remote_nn/tmp/et1, expected: hdfs://localhost:8020
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.isPathEncrypted(SemanticAnalyzer.java:1763)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStagingDirectoryPathname(SemanticAnalyzer.java:1875)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1689)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1427)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:10132)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10147)
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:190)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:421)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:307)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1112)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1160)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1039)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.IllegalArgumentException: Wrong FS: 
hdfs://remote_nn/tmp/et1, expected: hdfs://localhost:8020
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:193)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.getEZForPath(DistributedFileSystem.java:1906)
at 
org.apache.hadoop.hdfs.client.HdfsAdmin.getEncryptionZoneForPath(HdfsAdmin.java:262)
at 
org.apache.hadoop.hive.shims.Hadoop23Shims$HdfsEncryptionShim.isPathEncrypted(Hadoop23Shims.java:1097)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.isPathEncrypted(SemanticAnalyzer.java:1759)
... 25 more

FAILED: SemanticException Unable to determine if hdfs://remote_nn/tmp/et1is 
encrypted: java.lang.IllegalArgumentException: Wrong FS: 
hdfs://remote_nn/tmp/et1, expected: hdfs://localhost:8020
15/06/25 13:43:44 [main]: ERROR ql.Driver: FAILED: SemanticException Unable to 
determine if hdfs://remote_nn/tmp/et1is encrypted: 
java.lang.IllegalArgumentException: Wrong FS: hdfs://remote_nn/tmp/et1, 
expected: hdfs://localhost:8020
org.apache.hadoop.hive.ql.parse.SemanticException: Unable to determine if 
hdfs://remote_nn/tmp/et1is encrypted: java.lang.IllegalArgumentException: Wrong 
FS: hdfs://remote_nn/tmp/et1, expected: hdfs://localhost:8020
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1743)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1427

Review Request 35902: HIVE-11103 Add banker's rounding BROUND UDF

2015-06-25 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35902/
---

Review request for hive and Jason Dere.


Bugs: HIVE-11103
https://issues.apache.org/jira/browse/HIVE-11103


Repository: hive-git


Description
---

HIVE-11103 Add banker's rounding BROUND UDF


Diffs
-

  ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java 
6485a2ac5f12dbdba7bdf4d17ba18ad054c6f73b 
  common/src/java/org/apache/hadoop/hive/common/type/HiveDecimal.java 
a8215f29aed3a0399ec274cc311a3c92e0cca55b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
fabc21e2092561cbf98c35a406e4ee40e71fe1de 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/BRoundWithNumDigitsDoubleToDouble.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/DecimalUtil.java 
ef800596deed612b525ed3371b196f275ad88e09 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FuncBRoundWithNumDigitsDecimalToDecimal.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FuncRoundWithNumDigitsDecimalToDecimal.java
 9f3e8a3fcacb17990c6644a67cf587ae9948adad 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/MathExpr.java 
aef923e2c362a8d15b8dcc3467aef01a862c205c 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBRound.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRound.java 
963e4a87c417798f95bb1490a4275339a61e869c 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/RoundUtils.java 
0b389a5783fa2cf6643919c411ee57a7ed873d84 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFBRound.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_bround.q PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 
5de4ffcd1ace477af026b83fb7bfb8068fc192b3 
  ql/src/test/results/clientpositive/udf_bround.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/35902/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: How FuncRoundDoubleToDouble.java is generated?

2015-06-25 Thread Alexander Pivovarov

hive-ant GenVectorCode

On Thu, Jun 25, 2015 at 5:20 PM, Alexander Pivovarov apivova...@gmail.com
wrote:

 Hi Everyone

 Can you please explain me how FuncRoundDoubleToDouble is generated?
 I tried to find its template and can not find it...

 I need to generate similar class for BROUND UDF  (HIVE-11103)

 Thank you
 Alex

How FuncRoundDoubleToDouble.java is generated?

2015-06-25 Thread Alexander Pivovarov

Hi Everyone

Can you please explain me how FuncRoundDoubleToDouble is generated?
I tried to find its template and can not find it...

I need to generate similar class for BROUND UDF  (HIVE-11103)

Thank you
Alex

[jira] [Created] (HIVE-11103) Add banker's rounding BROUND UDF

2015-06-24 Thread Alexander Pivovarov (JIRA)

Alexander Pivovarov created HIVE-11103:
--

 Summary: Add banker's rounding BROUND UDF
 Key: HIVE-11103
 URL: https://issues.apache.org/jira/browse/HIVE-11103
 Project: Hive
  Issue Type: New Feature
  Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov


Banker's rounding: the value is rounded to the nearest even number. Also known 
as Gaussian rounding, and, in German, mathematische Rundung.

Example
{code}
  2 digits2 digits
UnroundedStandard roundingGaussian rounding
  54.1754  54.18  54.18
 343.2050 343.21 343.20
+106.2038+106.20+106.20 
=======
 503.5842 503.59 503.58
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 34586: HIVE-10704

2015-06-11 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34586/#review87672
---

Ship it!


Ship It!

- Alexander Pivovarov


On May 27, 2015, 6:33 a.m., Mostafa Mokhtar wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34586/
 ---
 
 (Updated May 27, 2015, 6:33 a.m.)
 
 
 Review request for hive.
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 fix biggest small table selection when table sizes are 0
 fallback to dividing memory equally if any tables have invalid size
 
 
 Diffs
 -
 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 536b92c 
 
 Diff: https://reviews.apache.org/r/34586/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Mostafa Mokhtar

Re: [ANNOUNCE] New Hive PMC Members - Chao Sun and Gopal Vijayaraghavan

2015-06-10 Thread Alexander Pivovarov

Congratulations to both of you!!!

On Wed, Jun 10, 2015 at 3:26 PM, Gopal Vijayaraghavan gop...@apache.org
wrote:

 Hi,

 Thanks everyone and Congratulations to Chao!

 Cheers,
 Gopal

 On 6/10/15, 2:20 PM, Carl Steinbach c...@apache.org wrote:

 I am pleased to announce that Chao Sun and Gopal Vijayaraghavan have been
 elected to the Hive Project Management Committee. Please join me in
 congratulating Chao and Gopal!
 
 Thanks.
 
 - Carl

Re: Review Request 34393: HIVE-10427 - collect_list() and collect_set() should accept struct types as argument

2015-06-04 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34393/#review86692
---

Ship it!


Ship It!

- Alexander Pivovarov


On June 1, 2015, 4:19 p.m., Chao Sun wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34393/
 ---
 
 (Updated June 1, 2015, 4:19 p.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-10427
 https://issues.apache.org/jira/browse/HIVE-10427
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Currently for collect_list() and collect_set(), only primitive types are 
 supported. This patch adds support for struct, list and map types as well.
 
 It turned out I that all I need is loosen the type checking.
 
 
 Diffs
 -
 
   data/files/customers.txt PRE-CREATION 
   data/files/nested_orders.txt PRE-CREATION 
   data/files/orders.txt PRE-CREATION 
   
 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java 
 536c4a7 
   
 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java 
 6dc424a 
   
 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
  efcc8f5 
   ql/src/test/queries/clientnegative/udaf_collect_set_unsupported.q 
 PRE-CREATION 
   ql/src/test/queries/clientpositive/udaf_collect_set_2.q PRE-CREATION 
   ql/src/test/results/clientnegative/udaf_collect_set_unsupported.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/udaf_collect_set_2.q.out PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/34393/diff/
 
 
 Testing
 ---
 
 All but one test (which seems unrelated) are passing.
 I also added a test: udaf_collect_list_set_2.q
 
 
 Thanks,
 
 Chao Sun

Re: Creating branch-1

2015-06-02 Thread Alexander Pivovarov

Hi Alan

Is branch-1 on github already?

https://github.com/apache/hive/tree/branch-1

On Tue, Jun 2, 2015 at 10:37 AM, Alan Gates alanfga...@gmail.com wrote:

 Ok, I've created branch-1.  I've also updated HowToContribute with
 information on what patches go where now.  Proofreaders and feedback on
 clarity are welcomed.  (At the same time I updated the page to contain git
 instructions instead of SVN, so feedback on that is good too.)

 Alan.

   Alan Gates alanfga...@gmail.com
  June 1, 2015 at 16:19
  Based on our discussion and vote last week I'm working on creating
 branch-1.   I plan to make the branch tomorrow.  If anyone has a large
 commit they don't want to have to commit twice and they are close to
 committing it let me know so I can make sure it gets in before I branch.

 I'll also be updating
 https://cwiki.apache.org/confluence/display/Hive/HowToContribute to
 clarify how to handle feature and bug fix patches on master and branch-1.

 Also, we will need to make sure patches can be tested against master and
 branch-1.  If I understand correctly the test system today will run a patch
 against a branch instead of master if the patch is named with the branch
 name.  There are a couple of issues with this.  One, people will often want
 to submit two versions of patches and have them both tested (one against
 master and one against branch-1) rather than one or the other.  The second
 is we will want a way for one patch to be tested against both when
 appropriate.  The first case could be handled by the system picking up both
 branch-1 and master patches and running them automatically.  The second
 could be handled by hints in the comments so the system needs to run both.
 I'm open to other suggestions as well.  Can someone familiar with the
 testing code point to where I'd look to see what it would take to make this
 work?

 Alan.

Re: hive.optimize.index.filter + ORC + TIMESTAMP throws NPE or IAE depending on hive version

2015-06-02 Thread Alexander Pivovarov

Hi Gopal

Thank you for your reply.
Can you send your examples again?
What you sent earlier is showed as ³1²  ³9² and ³11²  ³9²
x  Œ9¹;

On Mon, Jun 1, 2015 at 8:26 PM, Gopal Vijayaraghavan gop...@apache.org
wrote:

 Hi,

 That¹s expected behaviour since you are comparing a Timestamp to a string.

 Timestamp = String

 is being skipped because the SARGs need to be the same type to offer
 non-equality comparisons accurately.

 https://issues.apache.org/jira/browse/HIVE-10286


 I logged the bug after I hit bugs with PPD for that case when using ORC
 APIs from outside Hive (i.e ³1²  ³9² and ³11²  ³9²).

 That was a mistake anyone could¹ve made while hand-creating SARGs, but I
 wanted to make it better for the next person who might miss it and bail
 out without PPD when the arguments don¹t match PredicateLeaf.Type.


 You can try the same with something where hive does the right thing with a
 Filter expression

 hive create temporary table xx(x int) stored as orc;

 hive insert into xx values (1),(9),(11);
 hive select * from xy where x  Œ9¹;

 Cheers,
 Gopal

 On 6/1/15, 7:21 PM, Alexander Pivovarov apivova...@gmail.com wrote:

 if hive.optimize.index.filter is enabled then it causes the following the
 following stacktraces
 
 --
 create table ts (ts timestamp);
 insert into table ts values('2015-01-01 00:00:00');
 
 set hive.optimize.index.filter=true;
 select * from ts where ts = '2015-01-01 00:00:00';
 --
 
 
 -- HIVE-1.3.0 
 OK
 15/06/01 19:07:08 [main]: INFO ql.Driver: OK
 15/06/01 19:07:08 [main]: INFO log.PerfLogger: PERFLOG
 method=releaseLocks
 from=org.apache.hadoop.hive.ql.Driver
 15/06/01 19:07:08 [main]: INFO log.PerfLogger: /PERFLOG
 method=releaseLocks start=1433210828865 end=1433210828865 duration=0
 from=org.apache.hadoop.hive.ql.Driver
 15/06/01 19:07:08 [main]: INFO log.PerfLogger: /PERFLOG method=Driver.run
 start=1433210828758 end=1433210828865 duration=107
 from=org.apache.hadoop.hive.ql.Driver
 15/06/01 19:07:08 [main]: INFO log.PerfLogger: PERFLOG
 method=OrcGetSplits
 from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl
 15/06/01 19:07:08 [main]: INFO orc.OrcInputFormat: FooterCacheHitRatio:
 0/0
 15/06/01 19:07:08 [main]: INFO log.PerfLogger: /PERFLOG
 method=OrcGetSplits start=1433210828870 end=1433210828876 duration=6
 from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl
 15/06/01 19:07:08 [main]: INFO orc.OrcInputFormat: ORC pushdown predicate:
 leaf-0 = (LESS_THAN ts 2015-01-01 00:00:00)
 expr = (not leaf-0)
 15/06/01 19:07:08 [main]: INFO orc.OrcRawRecordMerger: min key = null, max
 key = null
 15/06/01 19:07:08 [main]: INFO orc.ReaderImpl: Reading ORC rows from
 hdfs://localhost/apps/apivovarov/warehouse/ts/00_0 with {include:
 [true, true], offset: 0, length: 9223372036854775807, sarg: leaf-0 =
 (LESS_THAN ts 2015-01-01 00:00:00)
 expr = (not leaf-0), columns: ['null', 'ts']}
 15/06/01 19:07:08 [main]: WARN orc.RecordReaderImpl: Exception when
 evaluating predicate. Skipping ORC PPD. Exception:
 java.lang.IllegalArgumentException: ORC SARGS could not convert from
 String
 to TIMESTAMP
 at
 org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.getBaseObjectForComparis
 on(RecordReaderImpl.java:659)
 at
 org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.evaluatePredicateRange(R
 ecordReaderImpl.java:373)
 at
 org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.evaluatePredicateProto(R
 ecordReaderImpl.java:338)
 at
 org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$SargApplier.pickRowGroup
 s(RecordReaderImpl.java:711)
 at
 org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.pickRowGroups(RecordRead
 erImpl.java:752)
 at
 org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderI
 mpl.java:778)
 at
 org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordRead
 erImpl.java:987)
 at
 org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordR
 eaderImpl.java:1020)
 at
 org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.init(RecordReaderImpl.
 java:205)
 at
 org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:53
 9)
 at
 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.init(OrcR
 awRecordMerger.java:183)
 at
 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.in
 it(OrcRawRecordMerger.java:226)
 at
 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.init(OrcRawRecordMer
 ger.java:437)
 at
 org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.j
 ava:1219)
 at
 org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFo
 rmat.java:1117)
 at
 org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getReco
 rdReader(FetchOperator.java:673)
 at
 org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator
 .java:323

Re: Review Request 34522: HIVE-10748 Replace StringBuffer with StringBuilder where possible

2015-06-01 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34522/
---

(Updated June 1, 2015, 7:20 p.m.)


Review request for hive, Ashutosh Chauhan and Sergio Pena.


Changes
---

rebased to the latest


Bugs: HIVE-10748
https://issues.apache.org/jira/browse/HIVE-10748


Repository: hive-git


Description
---

HIVE-10748 Replace StringBuffer with StringBuilder where possible


Diffs (updated)
-

  
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParser.java
 6d6bbc2ee2bca67645356089046a39a3b6969df0 
  common/src/test/org/apache/hadoop/hive/common/type/TestHiveBaseChar.java 
012c28b1a0024b7292a97076f42de1097dae6b2a 
  common/src/test/org/apache/hadoop/hive/common/type/TestHiveVarchar.java 
309d0427da3f17a85d16da0e0dca46ad29a1c48e 
  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatException.java 
265d08dec6d3e260adfadfe7f629592ebeb5039d 
  hcatalog/core/src/test/java/org/apache/hive/hcatalog/data/TestJsonSerDe.java 
2947c4333b925e0beabd8a85b188419a4d71a2e3 
  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/DelimitedInputWriter.java
 eae91cbd79ebb47e59263e8e47b8acdb457d576d 
  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
 3c2548635b95509da8cbdf474149c01da0662bbb 
  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 329e5da5c4675ad3d5f57fbdbddfc5ea168a6dbe 
  jdbc/src/java/org/apache/hive/jdbc/HivePreparedStatement.java 
8a0671fc28c4e8326df068f7de5cf278c863e362 
  
metastore/src/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
 52147bcbd0bd214b62e52d4ed2a6775e04a94143 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 
ada79bd0a235eff06aa48c5550ff622f8e2f774d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
5d588390bfa00a956f4094310819204371f81122 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
a9d2dbf1f7ddccaf71ce06a14e9681ab559186bb 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java 
4423cd1a9960c68b74788f41e386bea105cee4eb 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedBatchUtil.java 
4a16b4c196c7080b1ec64d8ffdc25f359698b4d6 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
c5f03d94672a80849400e51a238bcec1db56659d 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
acd9bf5017ca23578616a5bd9b902d2c2abed1ef 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
f7e1dbce4ef1c985b8f2987df413aed0ab087051 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
43882e7cd9dfd0380035faff78120ce977e21226 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkReduceSinkMapJoinProc.java
 e477f04d83715f5488e72bddd8527728faeb6789 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java 
7108a47676a6a8e2765f098c1799d08e587db58e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
d609732bf91bbeed68fa604f66893bf7734c7c56 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 
9e197331bffb8db4b02aa5d5d842d68d55f7001a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
8dff2fcee46a4d366bef559576348e9ea8ef6336 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 
cb0b68075ca4101df0b5ad2699afc45f1d038d4a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 
aa291b9b1f704c682c82d85675c5de17f3965403 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
b8e18eafb67307c9b974194de28482fa8a7c6f2a 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
847d75199d6d614bd17ea852a4e3e87bf6911be7 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 
f26225a72c34252c8fdf615bd34b59532376c5de 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
 f3f7d95ef90f3e4f1beacecb4d681030bd69a231 
  serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 
19fe952f5e84755d1e7a8b752997c084dab339b9 
  service/src/java/org/apache/hive/service/auth/HttpAuthUtils.java 
3ef55779a6bde85193ca63ec9683cf9f67d6a39d 

Diff: https://reviews.apache.org/r/34522/diff/


Testing
---


Thanks,

Alexander Pivovarov

hive.ppd.remove.duplicatefilters description is incorrect. What is the correct one?

2015-06-01 Thread Alexander Pivovarov

I noticed that conf/hive-default.xml.template has the following description

  property
namehive.ppd.remove.duplicatefilters/name
valuetrue/value
descriptionWhether to push predicates down into storage handlers.
Ignored when hive.optimize.ppd is false./description
  /property

Most probably the description was taken from hive.optimize.ppd.storage

So, what is the correct description for hive.ppd.remove.duplicatefilters?

hive.optimize.index.filter + ORC + TIMESTAMP throws NPE or IAE depending on hive version

2015-06-01 Thread Alexander Pivovarov

if hive.optimize.index.filter is enabled then it causes the following the
following stacktraces

--
create table ts (ts timestamp);
insert into table ts values('2015-01-01 00:00:00');

set hive.optimize.index.filter=true;
select * from ts where ts = '2015-01-01 00:00:00';
--


-- HIVE-1.3.0 
OK
15/06/01 19:07:08 [main]: INFO ql.Driver: OK
15/06/01 19:07:08 [main]: INFO log.PerfLogger: PERFLOG method=releaseLocks
from=org.apache.hadoop.hive.ql.Driver
15/06/01 19:07:08 [main]: INFO log.PerfLogger: /PERFLOG
method=releaseLocks start=1433210828865 end=1433210828865 duration=0
from=org.apache.hadoop.hive.ql.Driver
15/06/01 19:07:08 [main]: INFO log.PerfLogger: /PERFLOG method=Driver.run
start=1433210828758 end=1433210828865 duration=107
from=org.apache.hadoop.hive.ql.Driver
15/06/01 19:07:08 [main]: INFO log.PerfLogger: PERFLOG method=OrcGetSplits
from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl
15/06/01 19:07:08 [main]: INFO orc.OrcInputFormat: FooterCacheHitRatio: 0/0
15/06/01 19:07:08 [main]: INFO log.PerfLogger: /PERFLOG
method=OrcGetSplits start=1433210828870 end=1433210828876 duration=6
from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl
15/06/01 19:07:08 [main]: INFO orc.OrcInputFormat: ORC pushdown predicate:
leaf-0 = (LESS_THAN ts 2015-01-01 00:00:00)
expr = (not leaf-0)
15/06/01 19:07:08 [main]: INFO orc.OrcRawRecordMerger: min key = null, max
key = null
15/06/01 19:07:08 [main]: INFO orc.ReaderImpl: Reading ORC rows from
hdfs://localhost/apps/apivovarov/warehouse/ts/00_0 with {include:
[true, true], offset: 0, length: 9223372036854775807, sarg: leaf-0 =
(LESS_THAN ts 2015-01-01 00:00:00)
expr = (not leaf-0), columns: ['null', 'ts']}
15/06/01 19:07:08 [main]: WARN orc.RecordReaderImpl: Exception when
evaluating predicate. Skipping ORC PPD. Exception:
java.lang.IllegalArgumentException: ORC SARGS could not convert from String
to TIMESTAMP
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.getBaseObjectForComparison(RecordReaderImpl.java:659)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.evaluatePredicateRange(RecordReaderImpl.java:373)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.evaluatePredicateProto(RecordReaderImpl.java:338)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$SargApplier.pickRowGroups(RecordReaderImpl.java:711)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.pickRowGroups(RecordReaderImpl.java:752)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:778)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:987)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1020)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.init(RecordReaderImpl.java:205)
at
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539)
at
org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.init(OrcRawRecordMerger.java:183)
at
org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.init(OrcRawRecordMerger.java:226)
at
org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.init(OrcRawRecordMerger.java:437)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1219)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1117)
at
org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:673)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:323)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:445)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:414)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1671)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

15/06/01 19:07:08 [main]: INFO orc.OrcInputFormat: ORC pushdown predicate:
leaf-0 =

Re: Review Request 34798: HIVE-10788 - Change sort_array to support non-primitive types

2015-05-29 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34798/#review85852
---

Ship it!


Ship It!

- Alexander Pivovarov


On May 29, 2015, 5:52 a.m., Chao Sun wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34798/
 ---
 
 (Updated May 29, 2015, 5:52 a.m.)
 
 
 Review request for hive and Alexander Pivovarov.
 
 
 Bugs: HIVE-10788
 https://issues.apache.org/jira/browse/HIVE-10788
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Currently sort_array only support primitive types. As we already support 
 comparison between non-primitive types, it makes sense to remove this 
 restriction.
 
 
 Diffs
 -
 
   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java 
 2d6d58c 
   
 ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFSortArray.java
  PRE-CREATION 
   ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q 034de06 
   ql/src/test/queries/clientpositive/udf_sort_array.q 313bcf8 
   ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out 075f49b 
   ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out c068ecd 
   ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out abf7124 
   ql/src/test/results/clientpositive/udf_sort_array.q.out 9631c2d 
 
 Diff: https://reviews.apache.org/r/34798/diff/
 
 
 Testing
 ---
 
 I added a few tests to udf_sort_array.q, and also added unit tests.
 
 
 Thanks,
 
 Chao Sun

Re: Review Request 34586: HIVE-10704

2015-05-29 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34586/#review85853
---



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java
https://reviews.apache.org/r/34586/#comment137688

The comment above says - if any table has bad size estimate
But why you check totalSize = 0 then?
Should you iterate over all small tables and check that they all have good 
size estimate.

What if you have table sizes (100, -4, 0)
totalSize is 96. But table #2 size is -4, which is bad size.

To make code clear I recommend to add new boolean variable 
isAnyTableHasBadSize and set its value it in the place where you calc 
totalSize, biggest and maxSize


- Alexander Pivovarov


On May 27, 2015, 6:33 a.m., Mostafa Mokhtar wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34586/
 ---
 
 (Updated May 27, 2015, 6:33 a.m.)
 
 
 Review request for hive.
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 fix biggest small table selection when table sizes are 0
 fallback to dividing memory equally if any tables have invalid size
 
 
 Diffs
 -
 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 536b92c 
 
 Diff: https://reviews.apache.org/r/34586/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Mostafa Mokhtar

Can anyone review HIVE-686 substring_index UDF?

2015-05-29 Thread Alexander Pivovarov

https://issues.apache.org/jira/browse/HIVE-686

https://reviews.apache.org/r/34696/

Re: Slow JIRA

2015-05-28 Thread Alexander Pivovarov

Yes, it's. Need to wait 1-2 min before ticked is opened.

On Thu, May 28, 2015 at 1:45 PM, Xuefu Zhang xzh...@cloudera.com wrote:

 Anyone else found JIRA site is extremely slow? Loading a single JIRA takes
 a long time, and not to mention doing a search.

 Has this been reported yet? I'm having the problem since yesterday.

 Thanks,
 Xuefu

Relativity simple query returns incorrect result

2015-05-28 Thread Alexander Pivovarov

Hi Everyone

Couple days ago I stepped on one strange issue - WHERE col is not null
does not work for one query.

Details are here https://issues.apache.org/jira/browse/HIVE-10841

What you think? How critical is it?

Re: Relativity simple query returns incorrect result

2015-05-28 Thread Alexander Pivovarov

I just added

2.1 explain select ... output for hive-1.3.0 MR

2.2. explain select... output for hive-0.13.1 Tez

On Thu, May 28, 2015 at 11:08 AM, Gopal Vijayaraghavan gop...@apache.org
wrote:

 Hi,

 I saw that, but haven¹t got any time to run an EXPLAIN yet.

 Filter push-down cases are generally all down to hive.optimize.ppd=true; -
 everything becomes clearer when you look at an explain.

 If I had to guess, I¹d say that the new JOIN syntax won¹t have this bug.

 select Š from a,b,c where a.id = b.id, c.x = b.x;

 Cheers,
 Gopal

 On 5/28/15, 10:57 AM, Alexander Pivovarov apivova...@gmail.com wrote:

 Hi Everyone
 
 Couple days ago I stepped on one strange issue - WHERE col is not null
 does not work for one query.
 
 Details are here https://issues.apache.org/jira/browse/HIVE-10841
 
 What you think? How critical is it?

Re: Review Request 34696: HIVE-686 add UDF substring_index

2015-05-27 Thread Alexander Pivovarov



 On May 27, 2015, 4:42 a.m., Swarnim Kulkarni wrote:
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSubstringIndex.java,
   line 45
  https://reviews.apache.org/r/34696/diff/1/?file=972489#file972489line45
 
  Worth mentinoning in your example what the expected output would look 
  like?
 
 Alexander Pivovarov wrote:
 Not sure I got the issue...
 
 --- desc output
 hive desc function extended substring_index;
 OK
 ...
 Example:
   SELECT substring_index('www.apache.org', '.', 2);
  'www.apache'
 
 
 -- actual select
 hive SELECT substring_index('www.apache.org', '.', 2);
 OK
 www.apache
 
 Swarnim Kulkarni wrote:
 My point was just that why not also include a sample result what the 
 users could expect to see after this command is executed. Might improve the 
 readability a bit.

it's included. The result is 'www.apache' - right adter \n symbol


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34696/#review85318
---


On May 27, 2015, 3:35 a.m., Alexander Pivovarov wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34696/
 ---
 
 (Updated May 27, 2015, 3:35 a.m.)
 
 
 Review request for hive, Hao Cheng, Jason Dere, namit jain, and Thejas Nair.
 
 
 Bugs: HIVE-686
 https://issues.apache.org/jira/browse/HIVE-686
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 HIVE-686 add UDF substring_index
 
 
 Diffs
 -
 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
 94a3b1787e2b3571eb7a8102c28f7334ae3fa829 
   
 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSubstringIndex.java
  PRE-CREATION 
   
 ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFSubstringIndex.java
  PRE-CREATION 
   ql/src/test/queries/clientpositive/udf_substring_index.q PRE-CREATION 
   ql/src/test/results/clientpositive/show_functions.q.out 
 16820ca887320da13a42bebe0876f29eec373c8f 
   ql/src/test/results/clientpositive/udf_substring_index.q.out PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/34696/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Alexander Pivovarov

[jira] [Created] (HIVE-10841) [WHERE col is not null] does not work for large queries

2015-05-27 Thread Alexander Pivovarov (JIRA)

Alexander Pivovarov created HIVE-10841:
--

 Summary: [WHERE col is not null] does not work for large queries
 Key: HIVE-10841
 URL: https://issues.apache.org/jira/browse/HIVE-10841
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Reporter: Alexander Pivovarov


The result from the following SELCT query is 3 rows but it should be 1 row.
I checked it in MySQL - it returned 1 row.

To reproduce the issue in Hive
1. prepare tables
{code}
drop table if exists L;
drop table if exists LA;
drop table if exists FR;
drop table if exists A;
drop table if exists PI;
drop table if exists acct;

create table L as select 4436 id;
create table LA as select 4436 loan_id, 4748 aid, 4415 pi_id;
create table FR as select 4436 loan_id;
create table A as select 4748 id;
create table PI as select 4415 id;

create table acct as select 4748 aid, 10 acc_n, 122 brn;
insert into table acct values(4748, null, null);
insert into table acct values(4748, null, null);
{code}

2. run SELECT query
{code}
select
  acct.ACC_N,
  acct.brn
FROM L
JOIN LA ON L.id = LA.loan_id
JOIN FR ON L.id = FR.loan_id
JOIN A ON LA.aid = A.id
JOIN PI ON PI.id = LA.pi_id
JOIN acct ON A.id = acct.aid
WHERE
  L.id = 4436
  and acct.brn is not null;
{code}

the result is 3 rows
{code}
10  122
NULLNULL
NULLNULL
{code}

but it should be 1 row

{code}
10  122
{code}

3. workaround is to put acct.brn is not null to join condition
{code}
select
  acct.ACC_N,
  acct.brn
FROM L
JOIN LA ON L.id = LA.loan_id
JOIN FR ON L.id = FR.loan_id
JOIN A ON LA.aid = A.id
JOIN PI ON PI.id = LA.pi_id
JOIN acct ON A.id = acct.aid and acct.brn is not null
WHERE
  L.id = 4436;
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Stable releases from branch-1 and experimental releases from master

2015-05-27 Thread Alexander Pivovarov

+1
On May 27, 2015 10:45 AM, Vikram Dixit K vikram.di...@gmail.com wrote:

 +1 for all the reasons outlined.

 On Tue, May 26, 2015 at 6:13 PM, Thejas Nair thejas.n...@gmail.com
 wrote:
  +1
  - This is great for users who want to take longer to upgrade from
  hadoop-1 and care mainly for bug fixes and incremental features,
  rather than radical new features.
  - The ability to release initial 2.x releases marked as alpha/beta
  also helps to get users to try it out, and also lets them choose what
  is right for them.
  - This also lets developers focus on major new features without the
  burden of maintaining hadoop-1 compatibility.
 
  On Tue, May 26, 2015 at 11:41 AM, Alan Gates alanfga...@gmail.com
 wrote:
  We have discussed this for several weeks now.  Some concerns have been
  raised which I have tried to address.  I think it is time to vote on it
 as
  our release plan.  To be specific, I propose:
 
  Hive makes a branch-1 from the current master.  This would be used for
 1.3
  and future 1.x releases.  This branch would not deprecate existing
  functionality.  Any new features in this branch would also need to be
 put on
  master.  An upgrade path for users will be maintained from one 1.x
 release
  to the next, as well as from the latest 1.x release to the latest 2.x
  release.
 
  Going forward releases numbered 2.x will be made from master.  The
 purpose
  of these releases will be to enable users to get access to new features
  being developed in Hive and allow developers to get feedback.  It is
  expected that for a while these releases will not be production ready
 and
  will be clearly so labeled.  Some legacy features, such as Hadoop 1 and
  MapReduce, will no longer be supported in the master.  Any critical bug
  fixes (security, incorrect results, crashes) fixed in master will also
 be
  ported to branch-1 for at least a year.  This time period may be
 extended in
  the future based on the stability and adoption of 2.x releases.
 
  Based on Hive's bylaws this release plan vote will be open for 3 days
 and
  all active committers have binding votes.
 
  Here's my +1.
 
  Alan.



 --
 Nothing better than when appreciated for hard work.
 -Mark

Re: Review Request 34696: HIVE-686 add UDF substring_index

2015-05-26 Thread Alexander Pivovarov



 On May 27, 2015, 4:42 a.m., Swarnim Kulkarni wrote:
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSubstringIndex.java,
   line 45
  https://reviews.apache.org/r/34696/diff/1/?file=972489#file972489line45
 
  Worth mentinoning in your example what the expected output would look 
  like?

Not sure I got the issue...

--- desc output
hive desc function extended substring_index;
OK
...
Example:
  SELECT substring_index('www.apache.org', '.', 2);
 'www.apache'


-- actual select
hive SELECT substring_index('www.apache.org', '.', 2);
OK
www.apache


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34696/#review85318
---


On May 27, 2015, 3:35 a.m., Alexander Pivovarov wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34696/
 ---
 
 (Updated May 27, 2015, 3:35 a.m.)
 
 
 Review request for hive, Hao Cheng, Jason Dere, namit jain, and Thejas Nair.
 
 
 Bugs: HIVE-686
 https://issues.apache.org/jira/browse/HIVE-686
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 HIVE-686 add UDF substring_index
 
 
 Diffs
 -
 
   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
 94a3b1787e2b3571eb7a8102c28f7334ae3fa829 
   
 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSubstringIndex.java
  PRE-CREATION 
   
 ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFSubstringIndex.java
  PRE-CREATION 
   ql/src/test/queries/clientpositive/udf_substring_index.q PRE-CREATION 
   ql/src/test/results/clientpositive/show_functions.q.out 
 16820ca887320da13a42bebe0876f29eec373c8f 
   ql/src/test/results/clientpositive/udf_substring_index.q.out PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/34696/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Alexander Pivovarov

Review Request 34696: HIVE-686 add UDF substring_index

2015-05-26 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34696/
---

Review request for hive, Hao Cheng, Jason Dere, namit jain, and Thejas Nair.


Bugs: HIVE-686
https://issues.apache.org/jira/browse/HIVE-686


Repository: hive-git


Description
---

HIVE-686 add UDF substring_index


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
94a3b1787e2b3571eb7a8102c28f7334ae3fa829 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSubstringIndex.java 
PRE-CREATION 
  
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFSubstringIndex.java
 PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_substring_index.q PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 
16820ca887320da13a42bebe0876f29eec373c8f 
  ql/src/test/results/clientpositive/udf_substring_index.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/34696/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: Review Request 34455: HIVE-10550 Dynamic RDD caching optimization for HoS.[Spark Branch]

2015-05-22 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34455/#review84876
---



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ReduceTran.java
https://reviews.apache.org/r/34455/#comment136299

use 2 spaces for indent



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ReduceTran.java
https://reviews.apache.org/r/34455/#comment136300

use 2 spaces for indent


- Alexander Pivovarov


On May 22, 2015, 6:18 a.m., chengxiang li wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34455/
 ---
 
 (Updated May 22, 2015, 6:18 a.m.)
 
 
 Review request for hive, Chao Sun, Jimmy Xiang, and Xuefu Zhang.
 
 
 Bugs: HIVE-10550
 https://issues.apache.org/jira/browse/HIVE-10550
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 see jira description
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 43c53fc 
   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/CacheTran.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapTran.java 2170243 
   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ReduceTran.java e60dfac 
   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java ee5c78a 
   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 
 3f240f5 
   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java 
 e6c845c 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkRddCachingResolver.java
  PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 
 19aae70 
   ql/src/java/org/apache/hadoop/hive/ql/plan/SparkWork.java bb5dd79 
 
 Diff: https://reviews.apache.org/r/34455/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 chengxiang li

Re: [DISCUSS] Supporting Hadoop-1 and experimental features

2015-05-22 Thread Alexander Pivovarov

Looks like we discussing 3 options:

1. Support hadoop 1, 2 and 3 in master branch.

2. Support hadoop 1 in branch-1, hadoop 2 in branch-2, hadoop 3 in branch-3

3. Support hadoop 2 and 3 in master

I DO not think option 2 is good solution because it is much more difficuilt
to manage 3 active prod branches rather than one master branch.

I think we should go with options 1 or 3.

+1 on Xuefu and Edward opinion
On May 22, 2015 9:09 AM, Sergey Shelukhin ser...@hortonworks.com wrote:

 I think branch-2 doesn’t need to be framed as particularly adventurous
 (other than due to general increase of the amount of work done in Hive by
 community).
 All the new features that normally go on trunk/master will go to branch-2.
 branch-2 is just trunk as it is now, in fact there will be no branch-2,
 just master :) The difference is the dropped functionality, not added one.
 So you shouldn’t lose stability if you retain the same process as now by
 just staying on versions off master.

 Perhaps, as is usually the case in Apache projects, developing features on
 older branches would be discouraged. Right now, all features usually go on
 trunk/master, and are then back ported as needed and practical; so you
 wouldn’t (in Apache) make a feature on Hive 0.14 to be released in 0.14.N,
 and not back port to master.

 On 15/5/22, 00:49, Chris Drome cdr...@yahoo-inc.com.INVALID wrote:

 I understand the motivation and benefits of creating a branch-2 where
 more disruptive work can go on without affecting branch-1. While not
 necessarily against this approach, from Yahoo's standpoint, I do have
 some questions (concerns).
 Upgrading to a new version of Hive requires a significant commitment of
 time and resources to stabilize and certify a build for deployment to our
 clusters. Given the size of our clusters and scale of datasets, we have
 to be particularly careful about adopting new functionality. However, at
 the same time we are interested in new testing and making available new
 features and functionality. That said, we would have to rely on branch-1
 for the immediate future.
 One concern is that branch-1 would be left to stagnate, at which point
 there would be no option but for users to move to branch-2 as branch-1
 would be effectively end-of-lifed. I'm not sure how long this would take,
 but it would eventually happen as a direct result of the very reason for
 creating branch-2.
 A related concern is how disruptive the code changes will be in branch-2.
 I imagine that changes in early in branch-2 will be easy to backport to
 branch-1, while this effort will become more difficult, if not
 impractical, as time goes. If the code bases diverge too much then this
 could lead to more pressure for users of branch-1 to add features just to
 branch-1, which has been mentioned as undesirable. By the same token,
 backporting any code in branch-2 will require an increasing amount of
 effort, which contributors to branch-2 may not be interested in
 committing to.
 These questions affect us directly because, while we require a certain
 amount of stability, we also like to pull in new functionality that will
 be of value to our users. For example, our current 0.13 release is
 probably closer to 0.14 at this point. Given the lifespan of a release,
 it is often more palatable to backport features and bugfixes than to jump
 to a new version.
 
 The good thing about this proposal is the opportunity to evaluate and
 clean up alot of the old code.
 Thanks,
 chris
 
 
 
  On Monday, May 18, 2015 11:48 AM, Sergey Shelukhin
 ser...@hortonworks.com wrote:
 
 
  Note: by “cannot” I mean “are unwilling to”; upgrade paths exist, but
 some
 people are set in their ways or have practical considerations and don’t
 care for new shiny stuff.
 
 On 15/5/18, 11:46, Sergey Shelukhin ser...@hortonworks.com wrote:
 
 I think we need some path for deprecating old Hadoop versions, the same
 way we deprecate old Java version support or old RDBMS version support.
 At some point the cost of supporting Hadoop 1 exceeds the benefit. Same
 goes for stuff like MR; supporting it, esp. for perf work, becomes a
 burden, and it’s outdated with 2 alternatives, one of which has been
 around for 2 releases.
 The branches are a graceful way to get rid of the legacy burden.
 
 Alternatively, when sweeping changes are made, we can do what Hbase did
 (which is not pretty imho), where 0.94 version had ~30 dot releases
 because people cannot upgrade to 0.96 “singularity” release.
 
 
 I posit that people who run Hadoop 1 and MR at this day and age (and more
 so as time passes) are people who either don’t care about perf and new
 features, only stability; so, stability-focused branch would be perfect
 to
 support them.
 
 
 On 15/5/18, 10:04, Edward Capriolo edlinuxg...@gmail.com wrote:
 
 Up until recently Hive supported numerous versions of Hadoop code base
 with
 a simple shim layer. I would rather we stick to the shim layer. I think
 this was easily the best part about

Re: [DISCUSS] Supporting Hadoop-1 and experimental features

2015-05-22 Thread Alexander Pivovarov

Alan, your email client is not compatible with gmail viewer. For some
reason your reply contains the whole thread of the discussion
On May 22, 2015 10:58 AM, Alan Gates alanfga...@gmail.com wrote:

 I don't think anyone is advocating for option 2, as that would be
 disastrous.  Option 3 is closest to what I'm proposing, though again
 dropping support for Hadoop 1 is only a part of it.

 Alan.

   Alexander Pivovarov apivova...@gmail.com
  May 22, 2015 at 10:03
 Looks like we discussing 3 options:

 1. Support hadoop 1, 2 and 3 in master branch.

 2. Support hadoop 1 in branch-1, hadoop 2 in branch-2, hadoop 3 in branch-3

 3. Support hadoop 2 and 3 in master

 I DO not think option 2 is good solution because it is much more difficuilt
 to manage 3 active prod branches rather than one master branch.

 I think we should go with options 1 or 3.

 +1 on Xuefu and Edward opinion

   Sergey Shelukhin ser...@hortonworks.com
  May 22, 2015 at 9:08
 I think branch-2 doesn’t need to be framed as particularly adventurous
 (other than due to general increase of the amount of work done in Hive by
 community).
 All the new features that normally go on trunk/master will go to branch-2.
 branch-2 is just trunk as it is now, in fact there will be no branch-2,
 just master :) The difference is the dropped functionality, not added one.
 So you shouldn’t lose stability if you retain the same process as now by
 just staying on versions off master.

 Perhaps, as is usually the case in Apache projects, developing features on
 older branches would be discouraged. Right now, all features usually go on
 trunk/master, and are then back ported as needed and practical; so you
 wouldn’t (in Apache) make a feature on Hive 0.14 to be released in 0.14.N,
 and not back port to master.


   Chris Drome cdr...@yahoo-inc.com.INVALID
  May 22, 2015 at 0:49
 I understand the motivation and benefits of creating a branch-2 where more
 disruptive work can go on without affecting branch-1. While not necessarily
 against this approach, from Yahoo's standpoint, I do have some questions
 (concerns).
 Upgrading to a new version of Hive requires a significant commitment of
 time and resources to stabilize and certify a build for deployment to our
 clusters. Given the size of our clusters and scale of datasets, we have to
 be particularly careful about adopting new functionality. However, at the
 same time we are interested in new testing and making available new
 features and functionality. That said, we would have to rely on branch-1
 for the immediate future.
 One concern is that branch-1 would be left to stagnate, at which point
 there would be no option but for users to move to branch-2 as branch-1
 would be effectively end-of-lifed. I'm not sure how long this would take,
 but it would eventually happen as a direct result of the very reason for
 creating branch-2.
 A related concern is how disruptive the code changes will be in branch-2.
 I imagine that changes in early in branch-2 will be easy to backport to
 branch-1, while this effort will become more difficult, if not impractical,
 as time goes. If the code bases diverge too much then this could lead to
 more pressure for users of branch-1 to add features just to branch-1, which
 has been mentioned as undesirable. By the same token, backporting any code
 in branch-2 will require an increasing amount of effort, which contributors
 to branch-2 may not be interested in committing to.
 These questions affect us directly because, while we require a certain
 amount of stability, we also like to pull in new functionality that will be
 of value to our users. For example, our current 0.13 release is probably
 closer to 0.14 at this point. Given the lifespan of a release, it is often
 more palatable to backport features and bugfixes than to jump to a new
 version.

 The good thing about this proposal is the opportunity to evaluate and
 clean up alot of the old code.
 Thanks,
 chris



 On Monday, May 18, 2015 11:48 AM, Sergey Shelukhin
 ser...@hortonworks.com ser...@hortonworks.com wrote:


 Note: by “cannot” I mean “are unwilling to”; upgrade paths exist, but some
 people are set in their ways or have practical considerations and don’t
 care for new shiny stuff.





   Sergey Shelukhin ser...@hortonworks.com
  May 18, 2015 at 11:47
 Note: by “cannot” I mean “are unwilling to”; upgrade paths exist, but some
 people are set in their ways or have practical considerations and don’t
 care for new shiny stuff.


   Sergey Shelukhin ser...@hortonworks.com
  May 18, 2015 at 11:46
 I think we need some path for deprecating old Hadoop versions, the same
 way we deprecate old Java version support or old RDBMS version support.
 At some point the cost of supporting Hadoop 1 exceeds the benefit. Same
 goes for stuff like MR; supporting it, esp. for perf work, becomes a
 burden, and it’s outdated with 2 alternatives, one of which has been
 around for 2 releases.
 The branches are a graceful way to get rid

Re: Review Request 34393: HIVE-10427 - collect_list() and collect_set() should accept struct types as argument

2015-05-21 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34393/#review84747
---



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
https://reviews.apache.org/r/34393/#comment136093

Can you replace this if block with
checkArgsSize(arguments, min, max) ?



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
https://reviews.apache.org/r/34393/#comment136095

can you remove unused imports?
import 
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector.Category;


- Alexander Pivovarov


On May 21, 2015, 5:30 p.m., Chao Sun wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34393/
 ---
 
 (Updated May 21, 2015, 5:30 p.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-10427
 https://issues.apache.org/jira/browse/HIVE-10427
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Currently for collect_list() and collect_set(), only primitive types are 
 supported. This patch adds support for struct, list and map types as well.
 
 It turned out I that all I need is loosen the type checking.
 
 
 Diffs
 -
 
   data/files/customers.txt PRE-CREATION 
   data/files/nested_orders.txt PRE-CREATION 
   data/files/orders.txt PRE-CREATION 
   
 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java 
 536c4a7 
   
 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java 
 6dc424a 
   
 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
  efcc8f5 
   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java 
 2d6d58c 
   ql/src/test/queries/clientnegative/udaf_collect_set_unsupported.q 
 PRE-CREATION 
   ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q 034de06 
   ql/src/test/queries/clientpositive/udaf_collect_set_2.q PRE-CREATION 
   ql/src/test/results/clientnegative/udaf_collect_set_unsupported.q.out 
 PRE-CREATION 
   ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out c068ecd 
   ql/src/test/results/clientpositive/udaf_collect_set_2.q.out PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/34393/diff/
 
 
 Testing
 ---
 
 All but one test (which seems unrelated) are passing.
 I also added a test: udaf_collect_list_set_2.q
 
 
 Thanks,
 
 Chao Sun

Re: Review Request 34393: HIVE-10427 - collect_list() and collect_set() should accept struct types as argument

2015-05-21 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34393/#review84749
---



ql/src/test/queries/clientpositive/udaf_collect_set_2.q
https://reviews.apache.org/r/34393/#comment136097

Is it necessary?


- Alexander Pivovarov


On May 21, 2015, 5:30 p.m., Chao Sun wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34393/
 ---
 
 (Updated May 21, 2015, 5:30 p.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-10427
 https://issues.apache.org/jira/browse/HIVE-10427
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Currently for collect_list() and collect_set(), only primitive types are 
 supported. This patch adds support for struct, list and map types as well.
 
 It turned out I that all I need is loosen the type checking.
 
 
 Diffs
 -
 
   data/files/customers.txt PRE-CREATION 
   data/files/nested_orders.txt PRE-CREATION 
   data/files/orders.txt PRE-CREATION 
   
 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java 
 536c4a7 
   
 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java 
 6dc424a 
   
 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
  efcc8f5 
   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java 
 2d6d58c 
   ql/src/test/queries/clientnegative/udaf_collect_set_unsupported.q 
 PRE-CREATION 
   ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q 034de06 
   ql/src/test/queries/clientpositive/udaf_collect_set_2.q PRE-CREATION 
   ql/src/test/results/clientnegative/udaf_collect_set_unsupported.q.out 
 PRE-CREATION 
   ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out c068ecd 
   ql/src/test/results/clientpositive/udaf_collect_set_2.q.out PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/34393/diff/
 
 
 Testing
 ---
 
 All but one test (which seems unrelated) are passing.
 I also added a test: udaf_collect_list_set_2.q
 
 
 Thanks,
 
 Chao Sun

Review Request 34463: HIVE-10717 Fix failed qtest encryption_insert_partition_static test in Jenkin

2015-05-20 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34463/
---

Review request for hive.


Bugs: HIVE-10717
https://issues.apache.org/jira/browse/HIVE-10717


Repository: hive-git


Description
---

HIVE-10717 Fix failed qtest encryption_insert_partition_static test in Jenkin


Diffs
-

  ql/src/test/queries/clientpositive/encryption_insert_partition_static.q 
10fb96582fb4952e194540e9243a0df97f971495 
  
ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_static.q.out
 96f8b6abdb12f44a1c39d9e9180dbc84b5aa5bee 

Diff: https://reviews.apache.org/r/34463/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: Review Request 34463: HIVE-10717 Fix failed qtest encryption_insert_partition_static test in Jenkin

2015-05-20 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34463/
---

(Updated May 20, 2015, 6:02 p.m.)


Review request for hive.


Changes
---

rebased and rerun the test


Bugs: HIVE-10717
https://issues.apache.org/jira/browse/HIVE-10717


Repository: hive-git


Description
---

HIVE-10717 Fix failed qtest encryption_insert_partition_static test in Jenkin


Diffs (updated)
-

  ql/src/test/queries/clientpositive/encryption_insert_partition_static.q 
9bff0bd34b5ef0cf43f1e6738a6f1bbf5a6f372e 
  
ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_static.q.out
 8966608d7ec74c66dafc563aa2984efe86cab333 

Diff: https://reviews.apache.org/r/34463/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: Review Request 34473: HIVE-10749 Implement Insert statement for parquet

2015-05-20 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34473/#review84574
---



ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java
https://reviews.apache.org/r/34473/#comment135898

missing space after comma and before tableProperties



ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetStructObjectInspector.java
https://reviews.apache.org/r/34473/#comment135901

you can use
final ArrayListObject list = new 
ArrayListObject(Collections.nCopies(fields.size(), null));
instead


- Alexander Pivovarov


On May 20, 2015, 2:54 p.m., cheng xu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34473/
 ---
 
 (Updated May 20, 2015, 2:54 p.m.)
 
 
 Review request for hive, Alan Gates and Sergio Pena.
 
 
 Bugs: HIVE-10749
 https://issues.apache.org/jira/browse/HIVE-10749
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Implement the insert statement for parquet format.
 
 
 Diffs
 -
 
   
 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java
  000eb38 
   
 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java
  8380117 
   
 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java
  4e1820c 
   
 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/acid/ParquetRawRecordMerger.java
  PRE-CREATION 
   
 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/acid/ParquetRecordUpdater.java
  PRE-CREATION 
   
 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/HiveSchemaConverter.java
  43c772f 
   
 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/ParquetRecordReaderWrapper.java
  0a5edbb 
   
 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetStructObjectInspector.java
  PRE-CREATION 
   
 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/ParquetRecordWriterWrapper.java
  0d32e49 
   
 ql/src/test/org/apache/hadoop/hive/ql/io/parquet/AbstractTestParquetDirect.java
  5f7f597 
   
 ql/src/test/org/apache/hadoop/hive/ql/io/parquet/acid/TestParquetRecordUpdater.java
  PRE-CREATION 
   ql/src/test/queries/clientpositive/acid_parquet_insert.q PRE-CREATION 
   ql/src/test/results/clientpositive/acid_parquet_insert.q.out PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/34473/diff/
 
 
 Testing
 ---
 
 Newly added qtest and UT passed locally
 
 
 Thanks,
 
 cheng xu

Review Request 34514: HIVE-9875 replace new SimpleDateFormat with DateUtils.getDateFormat

2015-05-20 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34514/
---

Review request for hive.


Bugs: HIVE-9875
https://issues.apache.org/jira/browse/HIVE-9875


Repository: hive-git


Description
---

HIVE-9875 replace new SimpleDateFormat with DateUtils.getDateFormat


Diffs
-

  common/src/java/org/apache/hive/common/util/DateUtils.java 
c749bcb5e67c1ca23bb55242c362c349993ca185 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
7b48b8b87a0c54f482c32e460930978b691bcdb5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 
6a6593c3ac4890d7e169b5af6608aee66679933a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 
a5c1463be4e9e8200c5f79783b4426b83e963d66 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
a9d2dbf1f7ddccaf71ce06a14e9681ab559186bb 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
 1f5d91df7c4dd13fadc4488472fdd7bf541e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/MetaDataExportListener.java 
1739fd2381f995b677899812bf7bea2fea78df5e 
  serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 
a2a6c79f937aed561b062ad9275bb9eae2346c8b 
  serde/src/test/org/apache/hadoop/hive/serde2/io/TestDateWritable.java 
fd95ccf467224b0aad37d8348a1267f2102a7082 
  serde/src/test/org/apache/hadoop/hive/serde2/io/TestTimestampWritable.java 
5fee0197d4b3c9c7c395ee252167b22f6fda9f6e 

Diff: https://reviews.apache.org/r/34514/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: [ANNOUNCE] New Hive Committer - Chaoyu Tang

2015-05-20 Thread Alexander Pivovarov

Congrats Chaoyu!

On Wed, May 20, 2015 at 3:29 PM, Carl Steinbach c...@apache.org wrote:

 The Apache Hive PMC has voted to make Chaoyu Tang a committer on the Apache
 Hive Project.

 Please join me in congratulating Chaoyu!

 Thanks.

 - Carl

Re: Review Request 34522: HIVE-10748 Replace StringBuffer with StringBuilder where possible

2015-05-20 Thread Alexander Pivovarov



 On May 21, 2015, 5:06 a.m., Swarnim Kulkarni wrote:
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java, line 
  1833
  https://reviews.apache.org/r/34522/diff/1/?file=966061#file966061line1833
 
  Out of curiosity, any specific reason for this change?

I changed aggregationKey type from StringBuffer to Appendable on line 1818 
(it's public API. I can not change aggregationKey to StringBuilder. Appendable 
is an interface implemented by both classes.)

Appendable.append() signature has throws IOException.
StringBuffer.append() does not have it.

This is why line 1829 aggregationKey.append() throw s IOException now which I 
wrap with RuntimeException


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34522/#review84657
---


On May 21, 2015, 2:22 a.m., Alexander Pivovarov wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34522/
 ---
 
 (Updated May 21, 2015, 2:22 a.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-10748
 https://issues.apache.org/jira/browse/HIVE-10748
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 HIVE-10748 Replace StringBuffer with StringBuilder where possible
 
 
 Diffs
 -
 
   
 common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParser.java
  6d6bbc2ee2bca67645356089046a39a3b6969df0 
   common/src/test/org/apache/hadoop/hive/common/type/TestHiveBaseChar.java 
 012c28b1a0024b7292a97076f42de1097dae6b2a 
   common/src/test/org/apache/hadoop/hive/common/type/TestHiveVarchar.java 
 309d0427da3f17a85d16da0e0dca46ad29a1c48e 
   
 hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatException.java
  265d08dec6d3e260adfadfe7f629592ebeb5039d 
   
 hcatalog/core/src/test/java/org/apache/hive/hcatalog/data/TestJsonSerDe.java 
 2947c4333b925e0beabd8a85b188419a4d71a2e3 
   
 hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/DelimitedInputWriter.java
  eae91cbd79ebb47e59263e8e47b8acdb457d576d 
   
 hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
  3c2548635b95509da8cbdf474149c01da0662bbb 
   
 hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
  329e5da5c4675ad3d5f57fbdbddfc5ea168a6dbe 
   jdbc/src/java/org/apache/hive/jdbc/HivePreparedStatement.java 
 8a0671fc28c4e8326df068f7de5cf278c863e362 
   
 metastore/src/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
  52147bcbd0bd214b62e52d4ed2a6775e04a94143 
   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 
 835015f249684820a9f0eb453d3316a98af52e00 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
 7b48b8b87a0c54f482c32e460930978b691bcdb5 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
 a9d2dbf1f7ddccaf71ce06a14e9681ab559186bb 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java 
 4423cd1a9960c68b74788f41e386bea105cee4eb 
   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedBatchUtil.java 
 4a16b4c196c7080b1ec64d8ffdc25f359698b4d6 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
 c5f03d94672a80849400e51a238bcec1db56659d 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
 acd9bf5017ca23578616a5bd9b902d2c2abed1ef 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
 f7e1dbce4ef1c985b8f2987df413aed0ab087051 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
 0de74882f3b92aa979c1960ac64023d3c750b876 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkReduceSinkMapJoinProc.java
  e477f04d83715f5488e72bddd8527728faeb6789 
   ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java 
 7108a47676a6a8e2765f098c1799d08e587db58e 
   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
 086d9a2b1740a8dc8560667c19826b7dff6cb75b 
   
 ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java
  9e197331bffb8db4b02aa5d5d842d68d55f7001a 
   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
 8dff2fcee46a4d366bef559576348e9ea8ef6336 
   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 
 87a25480740df061e0918228d71dd9ec8e08a275 
   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 
 aa291b9b1f704c682c82d85675c5de17f3965403 
   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
 b8e18eafb67307c9b974194de28482fa8a7c6f2a 
   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
 847d75199d6d614bd17ea852a4e3e87bf6911be7 
   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 
 f26225a72c34252c8fdf615bd34b59532376c5de 
   
 serde/src/java/org/apache/hadoop

Review Request 34522: HIVE-10748 Replace StringBuffer with StringBuilder where possible

2015-05-20 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34522/
---

Review request for hive.


Bugs: HIVE-10748
https://issues.apache.org/jira/browse/HIVE-10748


Repository: hive-git


Description
---

HIVE-10748 Replace StringBuffer with StringBuilder where possible


Diffs
-

  
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParser.java
 6d6bbc2ee2bca67645356089046a39a3b6969df0 
  common/src/test/org/apache/hadoop/hive/common/type/TestHiveBaseChar.java 
012c28b1a0024b7292a97076f42de1097dae6b2a 
  common/src/test/org/apache/hadoop/hive/common/type/TestHiveVarchar.java 
309d0427da3f17a85d16da0e0dca46ad29a1c48e 
  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatException.java 
265d08dec6d3e260adfadfe7f629592ebeb5039d 
  hcatalog/core/src/test/java/org/apache/hive/hcatalog/data/TestJsonSerDe.java 
2947c4333b925e0beabd8a85b188419a4d71a2e3 
  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/DelimitedInputWriter.java
 eae91cbd79ebb47e59263e8e47b8acdb457d576d 
  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
 3c2548635b95509da8cbdf474149c01da0662bbb 
  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 329e5da5c4675ad3d5f57fbdbddfc5ea168a6dbe 
  jdbc/src/java/org/apache/hive/jdbc/HivePreparedStatement.java 
8a0671fc28c4e8326df068f7de5cf278c863e362 
  
metastore/src/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
 52147bcbd0bd214b62e52d4ed2a6775e04a94143 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 
835015f249684820a9f0eb453d3316a98af52e00 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
7b48b8b87a0c54f482c32e460930978b691bcdb5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
a9d2dbf1f7ddccaf71ce06a14e9681ab559186bb 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java 
4423cd1a9960c68b74788f41e386bea105cee4eb 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedBatchUtil.java 
4a16b4c196c7080b1ec64d8ffdc25f359698b4d6 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
c5f03d94672a80849400e51a238bcec1db56659d 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
acd9bf5017ca23578616a5bd9b902d2c2abed1ef 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
f7e1dbce4ef1c985b8f2987df413aed0ab087051 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
0de74882f3b92aa979c1960ac64023d3c750b876 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkReduceSinkMapJoinProc.java
 e477f04d83715f5488e72bddd8527728faeb6789 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java 
7108a47676a6a8e2765f098c1799d08e587db58e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
086d9a2b1740a8dc8560667c19826b7dff6cb75b 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 
9e197331bffb8db4b02aa5d5d842d68d55f7001a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
8dff2fcee46a4d366bef559576348e9ea8ef6336 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 
87a25480740df061e0918228d71dd9ec8e08a275 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 
aa291b9b1f704c682c82d85675c5de17f3965403 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
b8e18eafb67307c9b974194de28482fa8a7c6f2a 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
847d75199d6d614bd17ea852a4e3e87bf6911be7 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 
f26225a72c34252c8fdf615bd34b59532376c5de 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
 f3f7d95ef90f3e4f1beacecb4d681030bd69a231 
  serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 
19fe952f5e84755d1e7a8b752997c084dab339b9 
  service/src/java/org/apache/hive/service/auth/HttpAuthUtils.java 
3ef55779a6bde85193ca63ec9683cf9f67d6a39d 

Diff: https://reviews.apache.org/r/34522/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: Review Request 34522: HIVE-10748 Replace StringBuffer with StringBuilder where possible

2015-05-20 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34522/#review84659
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
https://reviews.apache.org/r/34522/#comment136012

I changed aggregationKey type from StringBuffer to Appendable on line 1818 
(it's public API. I can not change aggregationKey to StringBuilder. Appendable 
is an interface implemented by both classes.)

Appendable.append() signature has throws IOException.
StringBuffer.append() does not have it.

This is why line 1829 aggregationKey.append() throw s IOException now which 
I wrap with RuntimeException


- Alexander Pivovarov


On May 21, 2015, 2:22 a.m., Alexander Pivovarov wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34522/
 ---
 
 (Updated May 21, 2015, 2:22 a.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-10748
 https://issues.apache.org/jira/browse/HIVE-10748
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 HIVE-10748 Replace StringBuffer with StringBuilder where possible
 
 
 Diffs
 -
 
   
 common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParser.java
  6d6bbc2ee2bca67645356089046a39a3b6969df0 
   common/src/test/org/apache/hadoop/hive/common/type/TestHiveBaseChar.java 
 012c28b1a0024b7292a97076f42de1097dae6b2a 
   common/src/test/org/apache/hadoop/hive/common/type/TestHiveVarchar.java 
 309d0427da3f17a85d16da0e0dca46ad29a1c48e 
   
 hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatException.java
  265d08dec6d3e260adfadfe7f629592ebeb5039d 
   
 hcatalog/core/src/test/java/org/apache/hive/hcatalog/data/TestJsonSerDe.java 
 2947c4333b925e0beabd8a85b188419a4d71a2e3 
   
 hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/DelimitedInputWriter.java
  eae91cbd79ebb47e59263e8e47b8acdb457d576d 
   
 hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java
  3c2548635b95509da8cbdf474149c01da0662bbb 
   
 hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
  329e5da5c4675ad3d5f57fbdbddfc5ea168a6dbe 
   jdbc/src/java/org/apache/hive/jdbc/HivePreparedStatement.java 
 8a0671fc28c4e8326df068f7de5cf278c863e362 
   
 metastore/src/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
  52147bcbd0bd214b62e52d4ed2a6775e04a94143 
   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java 
 835015f249684820a9f0eb453d3316a98af52e00 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
 7b48b8b87a0c54f482c32e460930978b691bcdb5 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
 a9d2dbf1f7ddccaf71ce06a14e9681ab559186bb 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java 
 4423cd1a9960c68b74788f41e386bea105cee4eb 
   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedBatchUtil.java 
 4a16b4c196c7080b1ec64d8ffdc25f359698b4d6 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
 c5f03d94672a80849400e51a238bcec1db56659d 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
 acd9bf5017ca23578616a5bd9b902d2c2abed1ef 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
 f7e1dbce4ef1c985b8f2987df413aed0ab087051 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
 0de74882f3b92aa979c1960ac64023d3c750b876 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkReduceSinkMapJoinProc.java
  e477f04d83715f5488e72bddd8527728faeb6789 
   ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java 
 7108a47676a6a8e2765f098c1799d08e587db58e 
   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
 086d9a2b1740a8dc8560667c19826b7dff6cb75b 
   
 ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java
  9e197331bffb8db4b02aa5d5d842d68d55f7001a 
   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 
 8dff2fcee46a4d366bef559576348e9ea8ef6336 
   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 
 87a25480740df061e0918228d71dd9ec8e08a275 
   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 
 aa291b9b1f704c682c82d85675c5de17f3965403 
   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
 b8e18eafb67307c9b974194de28482fa8a7c6f2a 
   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
 847d75199d6d614bd17ea852a4e3e87bf6911be7 
   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 
 f26225a72c34252c8fdf615bd34b59532376c5de 
   
 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
  f3f7d95ef90f3e4f1beacecb4d681030bd69a231 
   serde/src/test/org/apache

[jira] [Created] (HIVE-10748) Replace StringBuffer with StringBuilder where possible

2015-05-19 Thread Alexander Pivovarov (JIRA)

Alexander Pivovarov created HIVE-10748:
--

 Summary: Replace StringBuffer with StringBuilder where possible
 Key: HIVE-10748
 URL: https://issues.apache.org/jira/browse/HIVE-10748
 Project: Hive
  Issue Type: Improvement
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
Priority: Minor


I found 40 places in Hive where new StringBuffer( is used.

Where possible, it is recommended that StringBuilder be used in preference to 
StringBuffer as it will be faster under most implementations

https://docs.oracle.com/javase/7/docs/api/java/lang/StringBuilder.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

GenericUDF.getConstantLongValue

2015-05-18 Thread Alexander Pivovarov

Hello Everyone

There is a bug in GenericUDF.getConstantLongValue.

There are 2 patches available:
1. fix the bug https://issues.apache.org/jira/browse/HIVE-10580

2. delete the method because it's not used
https://issues.apache.org/jira/browse/HIVE-10710

Can any committer +1 on one or another solution. I'm fine with any solution.

Thank you
Alex

Re: Review Request 33968: HIVE-10644 create SHA2 UDF

2015-05-18 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33968/
---

(Updated May 18, 2015, 10:24 p.m.)


Review request for hive and Jason Dere.


Changes
---

added GenericUDFParamUtils


Bugs: HIVE-10644
https://issues.apache.org/jira/browse/HIVE-10644


Repository: hive-git


Description
---

HIVE-10644 create SHA2 UDF


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
02a604ff0a4ed92dfd94b199e8b539f636b66f77 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFParamUtils.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSha2.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFSha2.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_sha2.q PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 
a422760400c62d026324dd667e4a632bfbe01b82 
  ql/src/test/results/clientpositive/udf_sha2.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/33968/diff/


Testing
---


Thanks,

Alexander Pivovarov

[jira] [Created] (HIVE-10710) Delete GenericUDF.getConstantLongValue

2015-05-14 Thread Alexander Pivovarov (JIRA)

Alexander Pivovarov created HIVE-10710:
--

 Summary: Delete GenericUDF.getConstantLongValue
 Key: HIVE-10710
 URL: https://issues.apache.org/jira/browse/HIVE-10710
 Project: Hive
  Issue Type: Improvement
  Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
Priority: Trivial


GenericUDF.getConstantLongValue has a bug.
Instead of fixing the bug it was suggested to delete the method because it is 
not used in hive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 34223: HIVE-10710 Delete GenericUDF.getConstantLongValue

2015-05-14 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34223/
---

Review request for hive, Ashutosh Chauhan and Jason Dere.


Bugs: HIVE-10710
https://issues.apache.org/jira/browse/HIVE-10710


Repository: hive-git


Description
---

HIVE-10710 Delete GenericUDF.getConstantLongValue


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 
b043bdc882af7c0b83787526a5a55c9dc29c6681 

Diff: https://reviews.apache.org/r/34223/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: JIRA notifications

2015-05-14 Thread Alexander Pivovarov

You can use the following command to create new review. It takes about 3-5
sec
$ rbt post -g yes

To update the review you can run.
$ rbt post -u -g yes

On Thu, May 14, 2015 at 10:48 AM, Prasanth Jayachandran 
pjayachand...@hortonworks.com wrote:

 @Swarnim..
 Generating patch with git diff needs to include the full index for it to
 be uploaded to review board. “git diff —full-index”.
 https://code.google.com/p/reviewboard/issues/detail?id=3115

 - Prasanth

  On May 14, 2015, at 9:14 AM, Thejas Nair thejas.n...@gmail.com wrote:
 
  Now that we have moved to git, you can try using github pull request
 instead.
  It also  integrates with jira.
  More git instructions - http://accumulo.apache.org/git.html
 
 
  On Thu, May 14, 2015 at 8:01 AM, kulkarni.swar...@gmail.com
  kulkarni.swar...@gmail.com wrote:
  Also not sure if it's related but seems like RB has been pretty sluggish
  lately too for me. It takes forever for a patch to submitted and a
 review
  request created(the latest one is still running for past 30 minutes
 with no
  output)
 
  On Wed, May 13, 2015 at 4:26 PM, Lefty Leverenz 
 leftylever...@gmail.com
  wrote:
 
  By the way, we still need to add iss...@hive.apache.org to the
  website's Mailing
  Lists http://hive.apache.org/mailing_lists.html page -- see
 HIVE-10124
  https://issues.apache.org/jira/browse/HIVE-10124.
 
  -- Lefty
 
  On Wed, May 13, 2015 at 2:16 PM, Lefty Leverenz 
 leftylever...@gmail.com
  wrote:
 
  But some notifications and comments aren't making it onto any Hive
  mailing
  list -- see INFRA-9221 
 https://issues.apache.org/jira/browse/INFRA-9221
  (please
  add your own comments and examples).  This means the mail archives
 don't
  have a complete record of JIRA activity.
 
  -- Lefty
 
  On Wed, May 13, 2015 at 10:03 AM, Thejas Nair thejas.n...@gmail.com
  wrote:
 
  comments now added go to iss...@hive.apache.org .
  emails for JIRAs created should still go to dev@
 
 
  On Wed, May 13, 2015 at 9:25 AM, kulkarni.swar...@gmail.com
  kulkarni.swar...@gmail.com wrote:
  I noticed that I haven't been getting notifications(or they are
 really
  delayed) on any of the new JIRAs created/ comments added. Anyone
 else
  noticing similar issues as well?
 
  --
  Swarnim
 
 
 
 
 
 
 
  --
  Swarnim

fixed couple q tests which failed in recent builds. Need committer review

2015-05-14 Thread Alexander Pivovarov

HIVE-10665 https://issues.apache.org/jira/browse/HIVE-10665
udaf_percentile_approx_23.q

HIVE-10706 https://issues.apache.org/jira/browse/HIVE-10706
vectorized_timestamp_funcs.q

Re: Review Request 34059: HIVE-10673 Dynamically partitioned hash join for Tez

2015-05-14 Thread Alexander Pivovarov



 On May 12, 2015, 6:26 a.m., Alexander Pivovarov wrote:
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java, 
  line 107
  https://reviews.apache.org/r/34059/diff/1/?file=955672#file955672line107
 
  ReduceSinkOperator uses Object.hashCode() and equals() methods.
  HashSet algo relies on hashCode/equals methods
 
 Jason Dere wrote:
 So that means equals() only works if it is the exact same 
 ReduceSinkOperator object. This should be ok for our usage, if we are 
 referring to the same ReduceSinkOperator, we should be using that exact same 
 object.

Do you want to use IdentityHashMap then?
This class implements the Map interface with a hash table, using 
reference-equality in place of object-equality when comparing keys (and 
values). In other words, in an IdentityHashMap, two keys k1 and k2 are 
considered equal if and only if (k1==k2)


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34059/#review83362
---


On May 11, 2015, 9:48 p.m., Jason Dere wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34059/
 ---
 
 (Updated May 11, 2015, 9:48 p.m.)
 
 
 Review request for hive, Matt McCline and Vikram Dixit Kumaraswamy.
 
 
 Bugs: HIVE-10673
 https://issues.apache.org/jira/browse/HIVE-10673
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Reduce-side hash join (using MapJoinOperator), where the Tez inputs to the 
 reducer are unsorted.
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java eff4d30 
   itests/src/test/resources/testconfiguration.properties eeb46cc 
   ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b1352f3 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java d7f1b42 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesAdapter.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValue.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValues.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
 545d7c6 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java 
 cdabe3a 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 
 15c747e 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinCommonOperator.java
  a9082eb 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
 d42b643 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 
 4d84f0f 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
 f7e1dbc 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java adc31ae 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 241e9d7 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java 6db8220 
   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java a342738 
   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java fb3c4a3 
   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cee9100 
   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_1.q PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_2.q PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_vector_dynpart_hashjoin_1.q 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_1.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_2.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_vector_dynpart_hashjoin_1.q.out 
 PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/34059/diff/
 
 
 Testing
 ---
 
 q-file tests added
 
 
 Thanks,
 
 Jason Dere

Re: Review Request 34059: HIVE-10673 Dynamically partitioned hash join for Tez

2015-05-14 Thread Alexander Pivovarov



 On May 12, 2015, 6:26 a.m., Alexander Pivovarov wrote:
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java, 
  line 107
  https://reviews.apache.org/r/34059/diff/1/?file=955672#file955672line107
 
  ReduceSinkOperator uses Object.hashCode() and equals() methods.
  HashSet algo relies on hashCode/equals methods
 
 Jason Dere wrote:
 So that means equals() only works if it is the exact same 
 ReduceSinkOperator object. This should be ok for our usage, if we are 
 referring to the same ReduceSinkOperator, we should be using that exact same 
 object.
 
 Alexander Pivovarov wrote:
 Do you want to use IdentityHashMap then?
 This class implements the Map interface with a hash table, using 
 reference-equality in place of object-equality when comparing keys (and 
 values). In other words, in an IdentityHashMap, two keys k1 and k2 are 
 considered equal if and only if (k1==k2)
 
 Jason Dere wrote:
 We're using a Set here as opposed to a Map. I'll change to use 
 Sets.newIdentityHashSet() from Guava.

IdentityHashMap contains private KeySet class already
to get its instance you can call keySet() method
e.g.
IdentityHashMapInteger, Object rsMap = new IdentityHashMapInteger, 
Object();
rsMap.put(1, null);
rsMap.put(2, null);
rsMap.put(3, null);
SetInteger rsSet = rsMap.keySet();
System.out.println(rsSet);
[3, 1, 2]


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34059/#review83362
---


On May 15, 2015, 1:02 a.m., Jason Dere wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34059/
 ---
 
 (Updated May 15, 2015, 1:02 a.m.)
 
 
 Review request for hive, Matt McCline and Vikram Dixit Kumaraswamy.
 
 
 Bugs: HIVE-10673
 https://issues.apache.org/jira/browse/HIVE-10673
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Reduce-side hash join (using MapJoinOperator), where the Tez inputs to the 
 reducer are unsorted.
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java eff4d30 
   itests/src/test/resources/testconfiguration.properties f9c9351 
   ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b1352f3 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java d7f1b42 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesAdapter.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValue.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValues.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
 545d7c6 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java 
 cdabe3a 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 
 e9bd44a 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinCommonOperator.java
  a9082eb 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
 d42b643 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 
 4d84f0f 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
 f7e1dbc 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java adc31ae 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 241e9d7 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java 6db8220 
   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java a342738 
   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java fb3c4a3 
   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cee9100 
   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_1.q PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_2.q PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_vector_dynpart_hashjoin_1.q 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_1.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_2.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_vector_dynpart_hashjoin_1.q.out 
 PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/34059/diff/
 
 
 Testing
 ---
 
 q-file tests added
 
 
 Thanks,
 
 Jason Dere

Review Request 34197: HIVE-10706 Make vectorized_timestamp_funcs test more stable

2015-05-14 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34197/
---

Review request for hive and Jason Dere.


Bugs: HIVE-10706
https://issues.apache.org/jira/browse/HIVE-10706


Repository: hive-git


Description
---

HIVE-10706 Make vectorized_timestamp_funcs test more stable


Diffs
-

  ql/src/test/queries/clientpositive/vectorized_timestamp_funcs.q 
8a2d5aaf5fb0396e551bdefdde507d1e9902919b 
  ql/src/test/results/clientpositive/spark/vectorized_timestamp_funcs.q.out 
304458215b4dcbc4d49321ba5f14ca5a87f2ec26 
  ql/src/test/results/clientpositive/tez/vectorized_timestamp_funcs.q.out 
fa3ed21232004d710b33cadac66680eabaca2c8a 
  ql/src/test/results/clientpositive/vectorized_timestamp_funcs.q.out 
31a96c68b22bd5332fb71b52982de71710df65fa 

Diff: https://reviews.apache.org/r/34197/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: Review Request 33881: HIVE-10623 Implement hive cli options using beeline functionality

2015-05-14 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33881/#review83724
---



beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java
https://reviews.apache.org/r/33881/#comment134770

You can use IOUtils.closeQuietly(bw)
I do not think we need to log close buffer error


- Alexander Pivovarov


On May 14, 2015, 5:51 a.m., cheng xu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/33881/
 ---
 
 (Updated May 14, 2015, 5:51 a.m.)
 
 
 Review request for hive and Xuefu Zhang.
 
 
 Bugs: HIVE-10623
 https://issues.apache.org/jira/browse/HIVE-10623
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Changes:
 1. Support the hive cli options including database, e, !, H, f.
 2. Add error handler for using f and e together
 3. Add error handler for invalid option
 
 
 Diffs
 -
 
   beeline/src/java/org/apache/hive/beeline/BeeLine.java 0da15f6 
   beeline/src/java/org/apache/hive/beeline/cli/CliOptionsProcessor.java 
 PRE-CREATION 
   beeline/src/java/org/apache/hive/beeline/cli/HiveCli.java PRE-CREATION 
   beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java PRE-CREATION 
   beeline/src/test/resources/hive-site.xml PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/33881/diff/
 
 
 Testing
 ---
 
 Newly add unit test passed locally.
 
 
 Thanks,
 
 cheng xu

Re: Review Request 33798: HIVE-10590 fix potential NPE in HiveMetaStore.equals

2015-05-13 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33798/#review83662
---

Ship it!


Ship It!

- Alexander Pivovarov


On May 4, 2015, 1:23 a.m., Alexander Pivovarov wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/33798/
 ---
 
 (Updated May 4, 2015, 1:23 a.m.)
 
 
 Review request for hive and Ashutosh Chauhan.
 
 
 Bugs: HIVE-10590
 https://issues.apache.org/jira/browse/HIVE-10590
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 HIVE-10590 fix potential NPE in HiveMetaStore.equals
 
 
 Diffs
 -
 
   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
 3f267ff0eb20560c36a19b74353f9d6749c8b333 
 
 Diff: https://reviews.apache.org/r/33798/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Alexander Pivovarov

Re: Review Request 33878: HIVE-10619 Fix ConcurrentHashMap.get in MetadataListStructObjectInspector.getInstance (52)

2015-05-13 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33878/#review83657
---

Ship it!


Ship It!

- Alexander Pivovarov


On May 6, 2015, 4 a.m., Alexander Pivovarov wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/33878/
 ---
 
 (Updated May 6, 2015, 4 a.m.)
 
 
 Review request for hive, Dhruba Borthakur and Szehon Ho.
 
 
 Bugs: HIVE-10619
 https://issues.apache.org/jira/browse/HIVE-10619
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 HIVE-10619 Fix ConcurrentHashMap.get in 
 MetadataListStructObjectInspector.getInstance (52)
 
 
 Diffs
 -
 
   
 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/MetadataListStructObjectInspector.java
  e68325f8548d2115f9fedd54cc8adefd4d5e76f8 
 
 Diff: https://reviews.apache.org/r/33878/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Alexander Pivovarov

Re: Review Request 33788: HIVE-10580 Fix impossible cast in GenericUDF.getConstantLongValue

2015-05-13 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33788/#review83677
---

Ship it!


Ship It!

- Alexander Pivovarov


On May 3, 2015, 5:26 a.m., Alexander Pivovarov wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/33788/
 ---
 
 (Updated May 3, 2015, 5:26 a.m.)
 
 
 Review request for hive and Jason Dere.
 
 
 Bugs: HIVE-10580
 https://issues.apache.org/jira/browse/HIVE-10580
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 HIVE-10580 Fix impossible cast in GenericUDF.getConstantLongValue
 
 
 Diffs
 -
 
   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 
 b043bdc882af7c0b83787526a5a55c9dc29c6681 
 
 Diff: https://reviews.apache.org/r/33788/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Alexander Pivovarov

Review Request 34183: HIVE-10665 Continue to make udaf_percentile_approx_23.q test more stable

2015-05-13 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34183/
---

Review request for hive and Jason Dere.


Bugs: HIVE-10665
https://issues.apache.org/jira/browse/HIVE-10665


Repository: hive-git


Description
---

HIVE-10665 Continue to make udaf_percentile_approx_23.q test more stable


Diffs
-

  ql/src/test/queries/clientpositive/udaf_percentile_approx_23.q 
b345b37251ff2ab440f0505b172c5f6481d8521f 
  ql/src/test/results/clientpositive/udaf_percentile_approx_23.q.out 
022a884834cb3077b63b98386adbcd879df39fe7 

Diff: https://reviews.apache.org/r/34183/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: Review Request 32807: HIVE-2327 Optimize REGEX UDFs with constant parameter information

2015-05-13 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32807/
---

(Updated May 14, 2015, 2:42 a.m.)


Review request for hive, Ashutosh Chauhan and Jason Dere.


Changes
---

updated q.out files


Bugs: HIVE-2327
https://issues.apache.org/jira/browse/HIVE-2327


Repository: hive-git


Description
---

HIVE-2327 Optimize REGEX UDFs with constant parameter information


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
02a604ff0a4ed92dfd94b199e8b539f636b66f77 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
096239e23c9c713e468c75a9a4fe3f01264300f9 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFRegExp.java 
76e1d2e8a11991c4b5e1acca0a534c46a2c7cbec 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRegExp.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFRegexp.java 
PRE-CREATION 
  ql/src/test/results/clientpositive/spark/vectorization_short_regress.q.out 
a4b8e0562d1ed3d0aad0bb9d6487a4dec4d578d9 
  ql/src/test/results/clientpositive/tez/vectorization_short_regress.q.out 
a3c723d1969daa33c0496ef6ab2e2f56e059776d 
  ql/src/test/results/clientpositive/vectorization_short_regress.q.out 
b9ab174914604baf40f1e5d5162a5fde5cd4cb0c 

Diff: https://reviews.apache.org/r/32807/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: Review Request 33881: HIVE-10623 Implement hive cli options using beeline functionality

2015-05-13 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33881/#review83634
---



beeline/src/java/org/apache/hive/beeline/cli/CliOptionsProcessor.java
https://reviews.apache.org/r/33881/#comment134641

remove spaces pls



beeline/src/java/org/apache/hive/beeline/cli/CliOptionsProcessor.java
https://reviews.apache.org/r/33881/#comment134642

line with spaces


- Alexander Pivovarov


On May 11, 2015, 3:22 p.m., cheng xu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/33881/
 ---
 
 (Updated May 11, 2015, 3:22 p.m.)
 
 
 Review request for hive and Xuefu Zhang.
 
 
 Bugs: HIVE-10623
 https://issues.apache.org/jira/browse/HIVE-10623
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Changes:
 1. Support the hive cli options including database, e, !, H, f.
 2. Add error handler for using f and e together
 3. Add error handler for invalid option
 
 
 Diffs
 -
 
   beeline/src/java/org/apache/hive/beeline/BeeLine.java b3d89cb 
   beeline/src/java/org/apache/hive/beeline/cli/CliOptionsProcessor.java 
 PRE-CREATION 
   beeline/src/java/org/apache/hive/beeline/cli/HiveCli.java PRE-CREATION 
   beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/33881/diff/
 
 
 Testing
 ---
 
 Newly add unit test passed locally.
 
 
 Thanks,
 
 cheng xu

Re: Review Request 33881: HIVE-10623 Implement hive cli options using beeline functionality

2015-05-13 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33881/#review83635
---



beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java
https://reviews.apache.org/r/33881/#comment134645

UPPERCASE variables are constants in java.  static final



beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java
https://reviews.apache.org/r/33881/#comment134643

move bw.close() to finally block


- Alexander Pivovarov


On May 11, 2015, 3:22 p.m., cheng xu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/33881/
 ---
 
 (Updated May 11, 2015, 3:22 p.m.)
 
 
 Review request for hive and Xuefu Zhang.
 
 
 Bugs: HIVE-10623
 https://issues.apache.org/jira/browse/HIVE-10623
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Changes:
 1. Support the hive cli options including database, e, !, H, f.
 2. Add error handler for using f and e together
 3. Add error handler for invalid option
 
 
 Diffs
 -
 
   beeline/src/java/org/apache/hive/beeline/BeeLine.java b3d89cb 
   beeline/src/java/org/apache/hive/beeline/cli/CliOptionsProcessor.java 
 PRE-CREATION 
   beeline/src/java/org/apache/hive/beeline/cli/HiveCli.java PRE-CREATION 
   beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/33881/diff/
 
 
 Testing
 ---
 
 Newly add unit test passed locally.
 
 
 Thanks,
 
 cheng xu

Re: Review Request 33881: HIVE-10623 Implement hive cli options using beeline functionality

2015-05-13 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33881/#review83632
---



beeline/src/java/org/apache/hive/beeline/BeeLine.java
https://reviews.apache.org/r/33881/#comment134640

use String.equalsIgnoreCase instead


- Alexander Pivovarov


On May 11, 2015, 3:22 p.m., cheng xu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/33881/
 ---
 
 (Updated May 11, 2015, 3:22 p.m.)
 
 
 Review request for hive and Xuefu Zhang.
 
 
 Bugs: HIVE-10623
 https://issues.apache.org/jira/browse/HIVE-10623
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Changes:
 1. Support the hive cli options including database, e, !, H, f.
 2. Add error handler for using f and e together
 3. Add error handler for invalid option
 
 
 Diffs
 -
 
   beeline/src/java/org/apache/hive/beeline/BeeLine.java b3d89cb 
   beeline/src/java/org/apache/hive/beeline/cli/CliOptionsProcessor.java 
 PRE-CREATION 
   beeline/src/java/org/apache/hive/beeline/cli/HiveCli.java PRE-CREATION 
   beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/33881/diff/
 
 
 Testing
 ---
 
 Newly add unit test passed locally.
 
 
 Thanks,
 
 cheng xu

[jira] [Created] (HIVE-10706) Make vectorized_timestamp_funcs test more stable

2015-05-13 Thread Alexander Pivovarov (JIRA)

Alexander Pivovarov created HIVE-10706:
--

 Summary: Make vectorized_timestamp_funcs test more stable
 Key: HIVE-10706
 URL: https://issues.apache.org/jira/browse/HIVE-10706
 Project: Hive
  Issue Type: Bug
  Components: UDF, Vectorization
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
Priority: Minor


TestCliDriver.testCliDriver_vectorized_timestamp_funcs failed recently
The problem is Double to Double numbers comparison without delta.
{code}
Running: diff -a 
/home/hiveptest/54.196.24.219-hiveptest-1/apache-github-source-source/itests/qtest/../../itests/qtest/target/qfile-results/clientpositive/vectorized_timestamp_funcs.q.out
 
/home/hiveptest/54.196.24.219-hiveptest-1/apache-github-source-source/itests/qtest/../../ql/src/test/results/clientpositive/vectorized_timestamp_funcs.q.out
729c729
 1123143.857003
---
 1123143.856998
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 33790: HIVE-10582 variable typo in HiveOpConverter (714) and SemanticAnalyzer (7496)

2015-05-13 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33790/#review83566
---

Ship it!


Ship It!

- Alexander Pivovarov


On May 3, 2015, 5:48 a.m., Alexander Pivovarov wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/33790/
 ---
 
 (Updated May 3, 2015, 5:48 a.m.)
 
 
 Review request for hive and Ashutosh Chauhan.
 
 
 Bugs: HIVE-10582
 https://issues.apache.org/jira/browse/HIVE-10582
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 HIVE-10582 variable typo in HiveOpConverter (714) and SemanticAnalyzer (7496)
 
 
 Diffs
 -
 
   
 ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
  b8ae604f054b8ccc99a4e5dc36b3b8870355a3d4 
   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
 dec0e38f4e63a59e08a91eb97d5b7c51626d9f51 
 
 Diff: https://reviews.apache.org/r/33790/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Alexander Pivovarov

Re: Review Request 34059: HIVE-10673 Dynamically partitioned hash join for Tez

2015-05-12 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34059/#review83359
---



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValue.java
https://reviews.apache.org/r/34059/#comment134334

booleans in java are false by default



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValue.java
https://reviews.apache.org/r/34059/#comment134335

Objects are null by default in Java



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValue.java
https://reviews.apache.org/r/34059/#comment134336

It is not necessary but I do not see a reason why the visibility of this 
method should be reduced. Should it be public as all others?


- Alexander Pivovarov


On May 11, 2015, 9:48 p.m., Jason Dere wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34059/
 ---
 
 (Updated May 11, 2015, 9:48 p.m.)
 
 
 Review request for hive, Matt McCline and Vikram Dixit Kumaraswamy.
 
 
 Bugs: HIVE-10673
 https://issues.apache.org/jira/browse/HIVE-10673
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Reduce-side hash join (using MapJoinOperator), where the Tez inputs to the 
 reducer are unsorted.
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java eff4d30 
   itests/src/test/resources/testconfiguration.properties eeb46cc 
   ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b1352f3 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java d7f1b42 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesAdapter.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValue.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValues.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
 545d7c6 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java 
 cdabe3a 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 
 15c747e 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinCommonOperator.java
  a9082eb 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
 d42b643 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 
 4d84f0f 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
 f7e1dbc 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java adc31ae 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 241e9d7 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java 6db8220 
   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java a342738 
   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java fb3c4a3 
   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cee9100 
   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_1.q PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_2.q PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_vector_dynpart_hashjoin_1.q 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_1.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_2.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_vector_dynpart_hashjoin_1.q.out 
 PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/34059/diff/
 
 
 Testing
 ---
 
 q-file tests added
 
 
 Thanks,
 
 Jason Dere

Re: Review Request 34059: HIVE-10673 Dynamically partitioned hash join for Tez

2015-05-12 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34059/#review83362
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
https://reviews.apache.org/r/34059/#comment134342

usually static Log should be private because superclass static methods 
should use their own static Log to avoid confusion.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
https://reviews.apache.org/r/34059/#comment134340

Can you use Map.Entry to avoid unnecesary lookup 3 lines below?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java
https://reviews.apache.org/r/34059/#comment134343

ReduceSinkOperator uses Object.hashCode() and equals() methods.
HashSet algo relies on hashCode/equals methods


- Alexander Pivovarov


On May 11, 2015, 9:48 p.m., Jason Dere wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34059/
 ---
 
 (Updated May 11, 2015, 9:48 p.m.)
 
 
 Review request for hive, Matt McCline and Vikram Dixit Kumaraswamy.
 
 
 Bugs: HIVE-10673
 https://issues.apache.org/jira/browse/HIVE-10673
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Reduce-side hash join (using MapJoinOperator), where the Tez inputs to the 
 reducer are unsorted.
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java eff4d30 
   itests/src/test/resources/testconfiguration.properties eeb46cc 
   ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b1352f3 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java d7f1b42 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesAdapter.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValue.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValues.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
 545d7c6 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java 
 cdabe3a 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 
 15c747e 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinCommonOperator.java
  a9082eb 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
 d42b643 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 
 4d84f0f 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
 f7e1dbc 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java adc31ae 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 241e9d7 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java 6db8220 
   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java a342738 
   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java fb3c4a3 
   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cee9100 
   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_1.q PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_2.q PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_vector_dynpart_hashjoin_1.q 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_1.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_2.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_vector_dynpart_hashjoin_1.q.out 
 PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/34059/diff/
 
 
 Testing
 ---
 
 q-file tests added
 
 
 Thanks,
 
 Jason Dere

Re: Review Request 34059: HIVE-10673 Dynamically partitioned hash join for Tez

2015-05-12 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34059/#review83367
---



ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java
https://reviews.apache.org/r/34059/#comment134344

trailing space



ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java
https://reviews.apache.org/r/34059/#comment134347

Why calling getEntry(key) two times consequently? 
containsKey() and get() call getEntry internally

Just call get(rs) one time, check thet result is not null and remove the 
second get(rs)


- Alexander Pivovarov


On May 11, 2015, 9:48 p.m., Jason Dere wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34059/
 ---
 
 (Updated May 11, 2015, 9:48 p.m.)
 
 
 Review request for hive, Matt McCline and Vikram Dixit Kumaraswamy.
 
 
 Bugs: HIVE-10673
 https://issues.apache.org/jira/browse/HIVE-10673
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Reduce-side hash join (using MapJoinOperator), where the Tez inputs to the 
 reducer are unsorted.
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java eff4d30 
   itests/src/test/resources/testconfiguration.properties eeb46cc 
   ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b1352f3 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java d7f1b42 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesAdapter.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValue.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValues.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
 545d7c6 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java 
 cdabe3a 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 
 15c747e 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinCommonOperator.java
  a9082eb 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
 d42b643 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 
 4d84f0f 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
 f7e1dbc 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java adc31ae 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 241e9d7 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java 6db8220 
   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java a342738 
   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java fb3c4a3 
   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cee9100 
   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_1.q PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_2.q PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_vector_dynpart_hashjoin_1.q 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_1.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_2.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_vector_dynpart_hashjoin_1.q.out 
 PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/34059/diff/
 
 
 Testing
 ---
 
 q-file tests added
 
 
 Thanks,
 
 Jason Dere

Re: Review Request 34059: HIVE-10673 Dynamically partitioned hash join for Tez

2015-05-12 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34059/#review83371
---



ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java
https://reviews.apache.org/r/34059/#comment134348

trailing space



ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java
https://reviews.apache.org/r/34059/#comment134349

Java will set it to 0 in constructor anyway.



ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
https://reviews.apache.org/r/34059/#comment134350

Remove this line and add String type declaration 3 lines below. Do not 
confuse GC.



ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java
https://reviews.apache.org/r/34059/#comment134351

it will be false by default


- Alexander Pivovarov


On May 11, 2015, 9:48 p.m., Jason Dere wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34059/
 ---
 
 (Updated May 11, 2015, 9:48 p.m.)
 
 
 Review request for hive, Matt McCline and Vikram Dixit Kumaraswamy.
 
 
 Bugs: HIVE-10673
 https://issues.apache.org/jira/browse/HIVE-10673
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Reduce-side hash join (using MapJoinOperator), where the Tez inputs to the 
 reducer are unsorted.
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java eff4d30 
   itests/src/test/resources/testconfiguration.properties eeb46cc 
   ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b1352f3 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java d7f1b42 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesAdapter.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValue.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValues.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
 545d7c6 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java 
 cdabe3a 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 
 15c747e 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinCommonOperator.java
  a9082eb 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
 d42b643 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 
 4d84f0f 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
 f7e1dbc 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java adc31ae 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 241e9d7 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java 6db8220 
   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java a342738 
   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java fb3c4a3 
   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cee9100 
   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_1.q PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_2.q PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_vector_dynpart_hashjoin_1.q 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_1.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_2.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_vector_dynpart_hashjoin_1.q.out 
 PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/34059/diff/
 
 
 Testing
 ---
 
 q-file tests added
 
 
 Thanks,
 
 Jason Dere

Re: Review Request 33992: HIVE-10657 Remove copyBytes operation from MD5 UDF

2015-05-12 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33992/
---

(Updated May 12, 2015, 9:57 p.m.)


Review request for hive and Jason Dere.


Bugs: HIVE-10657
https://issues.apache.org/jira/browse/HIVE-10657


Repository: hive-git


Description
---

HIVE-10657 Remove copyBytes operation from MD5 UDF


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFMd5.java 
62c16c23375eec96def5553404945dd963459850 

Diff: https://reviews.apache.org/r/33992/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: Review Request 33937: HIVE-10641 create CRC32 UDF

2015-05-12 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33937/
---

(Updated May 12, 2015, 10:18 p.m.)


Review request for hive and Jason Dere.


Changes
---

patch#2: use Text.getBytes() instead of toString()


Bugs: HIVE-10641
https://issues.apache.org/jira/browse/HIVE-10641


Repository: hive-git


Description
---

HIVE-10641 create CRC32 UDF


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
02a604ff0a4ed92dfd94b199e8b539f636b66f77 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCrc32.java PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFCrc32.java PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_crc32.q PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 
a422760400c62d026324dd667e4a632bfbe01b82 
  ql/src/test/results/clientpositive/udf_crc32.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/33937/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: Review Request 33927: HIVE-10639 create SHA1 UDF

2015-05-12 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33927/
---

(Updated May 12, 2015, 10:11 p.m.)


Review request for hive and Jason Dere.


Changes
---

patch#3 use Text.getBytes() instead of toString()


Bugs: HIVE-10639
https://issues.apache.org/jira/browse/HIVE-10639


Repository: hive-git


Description
---

HIVE-10639 create SHA1 UDF


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
02a604ff0a4ed92dfd94b199e8b539f636b66f77 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSha1.java PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFSha1.java PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_sha1.q PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 
a422760400c62d026324dd667e4a632bfbe01b82 
  ql/src/test/results/clientpositive/udf_sha1.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/33927/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: Review Request 33968: HIVE-10644 create SHA2 UDF

2015-05-12 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33968/
---

(Updated May 13, 2015, 5:48 a.m.)


Review request for hive and Jason Dere.


Changes
---

patch #2: use Text.getBytes() instead of toString()


Bugs: HIVE-10644
https://issues.apache.org/jira/browse/HIVE-10644


Repository: hive-git


Description
---

HIVE-10644 create SHA2 UDF


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
02a604ff0a4ed92dfd94b199e8b539f636b66f77 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 
b043bdc882af7c0b83787526a5a55c9dc29c6681 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSha2.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFSha2.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_sha2.q PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 
a422760400c62d026324dd667e4a632bfbe01b82 
  ql/src/test/results/clientpositive/udf_sha2.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/33968/diff/


Testing
---


Thanks,

Alexander Pivovarov

Re: [ANNOUNCE] New Hive Committers - Cheng Xu, Dong Chen, and Hari Sankar Sivarama Subramaniyan

2015-05-11 Thread Alexander Pivovarov

Congrats guys!

On Mon, May 11, 2015 at 1:54 PM, Carl Steinbach c...@apache.org wrote:

 The Apache Hive PMC has voted to make Cheng Xu, Dong Chen, and Hari Sankar
 Sivarama Subramaniyan committers on the Apache Hive Project.

 Please join me in congratulating Cheng, Dong, and Hari!

 Thanks.

 - Carl

Re: Review Request 34059: HIVE-10673 Dynamically partitioned hash join for Tez

2015-05-11 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34059/#review83356
---



ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
https://reviews.apache.org/r/34059/#comment134330

I think it's better to use Map.Entry here to avoid unnecessary lookup 
get(pos)
Map.Entry provides getKey, getValue, setValue methods.



ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
https://reviews.apache.org/r/34059/#comment134331

the same recommendation as avove



ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
https://reviews.apache.org/r/34059/#comment134332

Using replace(char, char) is faster than replace(CharSequence target, 
CharSequence replacement) because it is not using 
Pattern.compile().matcher().replaceAll API

Can you use replace('.', '_') instead of replace(., _)?


- Alexander Pivovarov


On May 11, 2015, 9:48 p.m., Jason Dere wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/34059/
 ---
 
 (Updated May 11, 2015, 9:48 p.m.)
 
 
 Review request for hive, Matt McCline and Vikram Dixit Kumaraswamy.
 
 
 Bugs: HIVE-10673
 https://issues.apache.org/jira/browse/HIVE-10673
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Reduce-side hash join (using MapJoinOperator), where the Tez inputs to the 
 reducer are unsorted.
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java eff4d30 
   itests/src/test/resources/testconfiguration.properties eeb46cc 
   ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b1352f3 
   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java d7f1b42 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesAdapter.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValue.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KeyValuesFromKeyValues.java 
 PRE-CREATION 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
 545d7c6 
   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java 
 cdabe3a 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 
 15c747e 
   
 ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinCommonOperator.java
  a9082eb 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
 d42b643 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 
 4d84f0f 
   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
 f7e1dbc 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java adc31ae 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 241e9d7 
   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java 6db8220 
   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java a342738 
   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java fb3c4a3 
   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cee9100 
   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_1.q PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_2.q PRE-CREATION 
   ql/src/test/queries/clientpositive/tez_vector_dynpart_hashjoin_1.q 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_1.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_2.q.out 
 PRE-CREATION 
   ql/src/test/results/clientpositive/tez/tez_vector_dynpart_hashjoin_1.q.out 
 PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/34059/diff/
 
 
 Testing
 ---
 
 q-file tests added
 
 
 Thanks,
 
 Jason Dere

[jira] [Created] (HIVE-10665) Continue to make udaf_percentile_approx_23.q test more stable

2015-05-09 Thread Alexander Pivovarov (JIRA)

Alexander Pivovarov created HIVE-10665:
--

 Summary: Continue to make udaf_percentile_approx_23.q test more 
stable
 Key: HIVE-10665
 URL: https://issues.apache.org/jira/browse/HIVE-10665
 Project: Hive
  Issue Type: Bug
  Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
Priority: Minor


HIVE-10059 fixed line 628 in q.out
Similar issue exists on line 567 and should be fixed as well.
{code}
Running: diff -a 
/home/hiveptest/54.159.254.207-hiveptest-2/apache-github-source-source/itests/qtest/../../itests/qtest/target/qfile-results/clientpositive/udaf_percentile_approx_23.q.out
 
/home/hiveptest/54.159.254.207-hiveptest-2/apache-github-source-source/itests/qtest/../../ql/src/test/results/clientpositive/udaf_percentile_approx_23.q.out
567c567
 342.0
---
 341.5
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 33927: HIVE-10639 create SHA1 UDF

2015-05-08 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33927/
---

(Updated May 8, 2015, 5:33 p.m.)


Review request for hive and Jason Dere.


Changes
---

performance optimization. removed unnecessary copyBytes operation.


Bugs: HIVE-10639
https://issues.apache.org/jira/browse/HIVE-10639


Repository: hive-git


Description
---

HIVE-10639 create SHA1 UDF


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
02a604ff0a4ed92dfd94b199e8b539f636b66f77 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSha1.java PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFSha1.java PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_sha1.q PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 
a422760400c62d026324dd667e4a632bfbe01b82 
  ql/src/test/results/clientpositive/udf_sha1.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/33927/diff/


Testing
---


Thanks,

Alexander Pivovarov

[jira] [Created] (HIVE-10657) Remove copyBytes operation from MD5 UDF

2015-05-08 Thread Alexander Pivovarov (JIRA)

Alexander Pivovarov created HIVE-10657:
--

 Summary: Remove copyBytes operation from MD5 UDF
 Key: HIVE-10657
 URL: https://issues.apache.org/jira/browse/HIVE-10657
 Project: Hive
  Issue Type: Improvement
  Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
Priority: Minor


Current implementation uses Apache Commons  DigestUtils.md5Hex method to get 
md5 hex.
DigestUtils does not provide md5Hex method with signature (byte[], start, 
length). This is why copyBytes method was added to get bytes[] from 
BytesWritable.

To avoid copying bytes from BytesWritable to new byte array we can use java 
MessageDigest API directly.
MessageDigest has method update(byte[], start, length)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 33992: HIVE-10657 Remove copyBytes operation from MD5 UDF

2015-05-08 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33992/
---

Review request for hive and Jason Dere.


Bugs: HIVE-10657
https://issues.apache.org/jira/browse/HIVE-10657


Repository: hive-git


Description
---

HIVE-10657 Remove copyBytes operation from MD5 UDF


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFMd5.java 
62c16c23375eec96def5553404945dd963459850 

Diff: https://reviews.apache.org/r/33992/diff/


Testing
---


Thanks,

Alexander Pivovarov

Review Request 33937: HIVE-10641 create CRC32 UDF

2015-05-07 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33937/
---

Review request for hive and Jason Dere.


Bugs: HIVE-10641
https://issues.apache.org/jira/browse/HIVE-10641


Repository: hive-git


Description
---

HIVE-10641 create CRC32 UDF


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
02a604ff0a4ed92dfd94b199e8b539f636b66f77 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFCrc32.java PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFCrc32.java PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_crc32.q PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 
a422760400c62d026324dd667e4a632bfbe01b82 
  ql/src/test/results/clientpositive/udf_crc32.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/33937/diff/


Testing
---


Thanks,

Alexander Pivovarov

Review Request 33968: HIVE-10644 create SHA2 UDF

2015-05-07 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33968/
---

Review request for hive and Jason Dere.


Bugs: HIVE-10644
https://issues.apache.org/jira/browse/HIVE-10644


Repository: hive-git


Description
---

HIVE-10644 create SHA2 UDF


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
02a604ff0a4ed92dfd94b199e8b539f636b66f77 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 
b043bdc882af7c0b83787526a5a55c9dc29c6681 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSha2.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFSha2.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_sha2.q PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 
a422760400c62d026324dd667e4a632bfbe01b82 
  ql/src/test/results/clientpositive/udf_sha2.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/33968/diff/


Testing
---


Thanks,

Alexander Pivovarov

[jira] [Created] (HIVE-10641) create CRC32 UDF

2015-05-07 Thread Alexander Pivovarov (JIRA)

Alexander Pivovarov created HIVE-10641:
--

 Summary: create CRC32 UDF
 Key: HIVE-10641
 URL: https://issues.apache.org/jira/browse/HIVE-10641
 Project: Hive
  Issue Type: Improvement
  Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov


CRC32 computes a cyclic redundancy check value for string or binary argument 
and returns bigint value. The result is NULL if the argument is NULL.

MySQL has similar function 
https://dev.mysql.com/doc/refman/5.0/en/mathematical-functions.html#function_crc32



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-10644) create SHA2 UDF

2015-05-07 Thread Alexander Pivovarov (JIRA)

Alexander Pivovarov created HIVE-10644:
--

 Summary: create SHA2 UDF
 Key: HIVE-10644
 URL: https://issues.apache.org/jira/browse/HIVE-10644
 Project: Hive
  Issue Type: Improvement
  Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov


Calculates the SHA-2 family of hash functions (SHA-224, SHA-256, SHA-384, and 
SHA-512). The first argument is the cleartext string to be hashed. The second 
argument indicates the desired bit length of the result, which must have a 
value of 224, 256, 384, 512, or 0 (which is equivalent to 256). If either 
argument is NULL or the hash length is not one of the permitted values, the 
return value is NULL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 33880: HIVE-10621 serde typeinfo equals methods are not symmetric

2015-05-06 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33880/#review82702
---

Ship it!


Ship It!

- Alexander Pivovarov


On May 6, 2015, 4:59 a.m., Alexander Pivovarov wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/33880/
 ---
 
 (Updated May 6, 2015, 4:59 a.m.)
 
 
 Review request for hive and Ashutosh Chauhan.
 
 
 Bugs: HIVE-10621
 https://issues.apache.org/jira/browse/HIVE-10621
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 HIVE-10621 serde typeinfo equals methods are not symmetric
 
 
 Diffs
 -
 
   serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java 
 6ab64e56c37f127551bbf21213ff4d4a98803c43 
   serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/CharTypeInfo.java 
 610818e21be3b376b1d40f3d16bf8bf7fc47adf3 
   serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/DecimalTypeInfo.java 
 cbe48029307a370d229d8e444ec6542e093e959a 
   
 serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/PrimitiveTypeInfo.java 
 a66b50a6a224b75d5c0f5dac5ec07eab44128c12 
   serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/VarcharTypeInfo.java 
 5ac2b46ac9ef3d6ddbf4496b23f43ff410a10e2f 
 
 Diff: https://reviews.apache.org/r/33880/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Alexander Pivovarov

Re: Review Request 33861: HIVE-10608 Fix useless 'if' stamement in RetryingMetaStoreClient (135)

2015-05-06 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33861/#review82703
---

Ship it!


Ship It!

- Alexander Pivovarov


On May 5, 2015, 8:19 p.m., Alexander Pivovarov wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/33861/
 ---
 
 (Updated May 5, 2015, 8:19 p.m.)
 
 
 Review request for hive and Szehon Ho.
 
 
 Bugs: HIVE-10608
 https://issues.apache.org/jira/browse/HIVE-10608
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 HIVE-10608 Fix useless 'if' stamement in RetryingMetaStoreClient (135)
 
 
 Diffs
 -
 
   
 metastore/src/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java
  1b6487af748202d1d0411ac23a7507a9fbd7f251 
 
 Diff: https://reviews.apache.org/r/33861/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Alexander Pivovarov

[jira] [Created] (HIVE-10639) create SHA1 UDF

2015-05-06 Thread Alexander Pivovarov (JIRA)

Alexander Pivovarov created HIVE-10639:
--

 Summary: create SHA1 UDF
 Key: HIVE-10639
 URL: https://issues.apache.org/jira/browse/HIVE-10639
 Project: Hive
  Issue Type: Improvement
  Components: UDF
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov


Calculates an SHA-1 160-bit checksum for the string, as described in RFC 3174 
(Secure Hash Algorithm). The value is returned as a string of 40 hex digits, or 
NULL if the argument was NULL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 33927: HIVE-10639 create SHA1 UDF

2015-05-06 Thread Alexander Pivovarov


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33927/
---

Review request for hive and Jason Dere.


Bugs: HIVE-10639
https://issues.apache.org/jira/browse/HIVE-10639


Repository: hive-git


Description
---

HIVE-10639 create SHA1 UDF


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
02a604ff0a4ed92dfd94b199e8b539f636b66f77 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSha1.java PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFSha1.java PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_sha1.q PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 
a422760400c62d026324dd667e4a632bfbe01b82 
  ql/src/test/results/clientpositive/udf_sha1.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/33927/diff/


Testing
---


Thanks,

Alexander Pivovarov

[jira] [Created] (HIVE-10618) Fix invocation of toString on byteArray in VerifyFast (250, 254)

2015-05-05 Thread Alexander Pivovarov (JIRA)

Alexander Pivovarov created HIVE-10618:
--

 Summary: Fix invocation of toString on byteArray in VerifyFast 
(250, 254)
 Key: HIVE-10618
 URL: https://issues.apache.org/jira/browse/HIVE-10618
 Project: Hive
  Issue Type: Bug
  Components: Serializers/Deserializers
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov
Priority: Minor


Arrays.toString(byteArray) can be used to convert byte[] to string



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

1 2 3 4 5 6 7 8 >

1 - 100 of 779 matches

Mail list logo