Re: Hive unit tests for Dal failing since 3/11

[Hari Subramaniyan]

Please ignore the below mail. Apologies.

Thanks
Hari

From: Hari Subramaniyan 
Sent: Tuesday, March 17, 2015 9:59 PM
To: Raja Aluri; Jeff Bowles
Cc: dev@hive.apache.org; Gunther Hagleitner; Thejas Nair
Subject: Hive unit tests for Dal failing since 3/11

Hi Raja/Jeff

I had raised a blocker  https://hortonworks.jira.com/browse/BUG-33235? 
concerning this issue.

The latest runs also failed, can you please have someone look into this ASAP. 
There were close to 2300 failures

in the last successful run, so we need to see how many of these have been fixed 
since the calcite version change to analyze the remaining failures.


Thanks

Hari

Hive unit tests for Dal failing since 3/11

[Hari Subramaniyan]

Hi Raja/Jeff

I had raised a blocker  https://hortonworks.jira.com/browse/BUG-33235? 
concerning this issue.

The latest runs also failed, can you please have someone look into this ASAP. 
There were close to 2300 failures

in the last successful run, so we need to see how many of these have been fixed 
since the calcite version change to analyze the remaining failures.


Thanks

Hari

Re: Review Request 32178: HIVE-9997 minor tweaks for bytes mapjoin hash table

[Mostafa Mokhtar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32178/#review76868
---

Ship it!


Ship It!

- Mostafa Mokhtar


On March 17, 2015, 10:41 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32178/
> ---
> 
> (Updated March 17, 2015, 10:41 p.m.)
> 
> 
> Review request for hive and Mostafa Mokhtar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
>  28f6c63 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 
> 2c8aee1 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java
>  b3582b2 
>   serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 390d9de 
>   serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java bed4d0a 
> 
> Diff: https://reviews.apache.org/r/32178/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 32019: HIVE-9950 fix rehash in CuckooSetBytes and CuckooSetLong

[Alexander Pivovarov]

> On March 16, 2015, 9:02 p.m., Matt McCline wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CuckooSetBytes.java,
> >  line 209
> > 
> >
> > At a "micro" level, this and the other change look good.
> > 
> > Did you spot it in code reading?
> > Or, in the hash results being poor?
> > The research paper would take time to study.
> > How would one write tests for Cuckoo?

FindBugs software found this and other issues.


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32019/#review76644
---


On March 13, 2015, 2:46 a.m., Alexander Pivovarov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32019/
> ---
> 
> (Updated March 13, 2015, 2:46 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Gopal V, Sergey Shelukhin, and 
> Vikram Dixit Kumaraswamy.
> 
> 
> Bugs: HIVE-9950
> https://issues.apache.org/jira/browse/HIVE-9950
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-9950 fix rehash in CuckooSetBytes and CuckooSetLong
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CuckooSetBytes.java
>  fe75ce3d3b14b84d023612c00554729e793dc677 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CuckooSetDouble.java
>  5897c92a2022fea4e4453b2807fc7b3b364e2282 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CuckooSetLong.java
>  5779fb35d7ffbc67917cdcd66b1c57baa3e2bff3 
> 
> Diff: https://reviews.apache.org/r/32019/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Alexander Pivovarov
> 
>

Re: Review Request 32190: HIVE-9975:Renaming a nonexisting partition should not throw out NullPointerException

[Xuefu Zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32190/#review76866
---

Ship it!


Ship It!

- Xuefu Zhang


On March 18, 2015, 2:09 a.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32190/
> ---
> 
> (Updated March 18, 2015, 2:09 a.m.)
> 
> 
> Review request for hive, Szehon Ho and Xuefu Zhang.
> 
> 
> Bugs: HIVE-9975
> https://issues.apache.org/jira/browse/HIVE-9975
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Currently Hive throws out NullPointerException when it renames a partition 
> which does not exist. It is not proper and user unfriendly. 
> This patch is going to change Hive to throw out HiveException with more 
> informative error message: "Rename partition: source partition 
> [pcol1=nonexist_part1/pcol2=nonexist_part2] does not exist."
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> d634b4021600fd8b980dc5d56b77e3192eba5d9f 
>   ql/src/test/queries/clientnegative/alter_rename_partition_failure.q 
> be971f18498693491d0699f7793a39e21b1ed684 
>   ql/src/test/results/clientnegative/alter_rename_partition_failure.q.out 
> f1640be1d2728d971517afdfc4f36efe1e5bb508 
> 
> Diff: https://reviews.apache.org/r/32190/diff/
> 
> 
> Testing
> ---
> 
> 1. Manaully tests passed
> 2. Precommit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

Review Request 32190: HIVE-9975:Renaming a nonexisting partition should not throw out NullPointerException

[Chaoyu Tang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32190/
---

Review request for hive, Szehon Ho and Xuefu Zhang.


Bugs: HIVE-9975
https://issues.apache.org/jira/browse/HIVE-9975


Repository: hive-git


Description
---

Currently Hive throws out NullPointerException when it renames a partition 
which does not exist. It is not proper and user unfriendly. 
This patch is going to change Hive to throw out HiveException with more 
informative error message: "Rename partition: source partition 
[pcol1=nonexist_part1/pcol2=nonexist_part2] does not exist."


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
d634b4021600fd8b980dc5d56b77e3192eba5d9f 
  ql/src/test/queries/clientnegative/alter_rename_partition_failure.q 
be971f18498693491d0699f7793a39e21b1ed684 
  ql/src/test/results/clientnegative/alter_rename_partition_failure.q.out 
f1640be1d2728d971517afdfc4f36efe1e5bb508 

Diff: https://reviews.apache.org/r/32190/diff/


Testing
---

1. Manaully tests passed
2. Precommit tests


Thanks,

Chaoyu Tang

Review Request 32189: HIVE-9859 Create bitwise left/right shift UDFs

[Alexander Pivovarov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32189/
---

Review request for hive and Jason Dere.


Bugs: HIVE-9859
https://issues.apache.org/jira/browse/HIVE-9859


Repository: hive-git


Description
---

HIVE-9859 Create bitwise left/right shift UDFs


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 
fdbfab9e1c4f098766f58e2d07653a44f45d3350 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 
e7de6c86a3c7a674b54f3678b00f34f2dd903dc8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 
d2d998972b64a19bde28cf176b3f948c00ba492a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 
0a05cebf1f71bd32c8023cdb10c8393a0d871cc2 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFOPBitLeftShift.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFOPBitRightShift.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/UDFOPBitUnsignedRightShift.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_bitwise_left_shift.q PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_bitwise_right_shift.q PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_bitwise_unsigned_right_shift.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 
81abeb9be4fd47724be544c7bc8da8b25fcd6e75 
  ql/src/test/results/clientpositive/udf_bitwise_left_shift.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/udf_bitwise_right_shift.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/udf_bitwise_unsigned_right_shift.q.out 
PRE-CREATION 

Diff: https://reviews.apache.org/r/32189/diff/


Testing
---


Thanks,

Alexander Pivovarov

[jira] [Created] (HIVE-10001) SMB join in reduce side

[Vikram Dixit K (JIRA)]

Vikram Dixit K created HIVE-10001:
-

 Summary: SMB join in reduce side
 Key: HIVE-10001
 URL: https://issues.apache.org/jira/browse/HIVE-10001
 Project: Hive
  Issue Type: Bug
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-10000) 10000 whoooohooo

[Gunther Hagleitner (JIRA)]

Gunther Hagleitner created HIVE-1:
-

 Summary: 1 whhooo
 Key: HIVE-1
 URL: https://issues.apache.org/jira/browse/HIVE-1
 Project: Hive
  Issue Type: Bug
Reporter: Gunther Hagleitner






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-9999) LLAP: Handle task rejection from daemons in the AM

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-:


 Summary: LLAP: Handle task rejection from daemons in the AM
 Key: HIVE-
 URL: https://issues.apache.org/jira/browse/HIVE-
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth
 Fix For: llap






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-9998) Vectorization support for interval types

[Jason Dere (JIRA)]

Jason Dere created HIVE-9998:


 Summary: Vectorization support for interval types
 Key: HIVE-9998
 URL: https://issues.apache.org/jira/browse/HIVE-9998
 Project: Hive
  Issue Type: Sub-task
  Components: Types, Vectorization
Reporter: Jason Dere
Assignee: Jason Dere


Allow intervals to work during vectorized query execution



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 32178: HIVE-9997 minor tweaks for bytes mapjoin hash table

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32178/
---

Review request for hive and Mostafa Mokhtar.


Repository: hive-git


Description
---

see jira


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
 28f6c63 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 
2c8aee1 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java
 b3582b2 
  serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 390d9de 
  serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java bed4d0a 

Diff: https://reviews.apache.org/r/32178/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-9997) minor tweaks for bytes mapjoin hash table

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-9997:
--

 Summary: minor tweaks for bytes mapjoin hash table
 Key: HIVE-9997
 URL: https://issues.apache.org/jira/browse/HIVE-9997
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


>From HIVE-7617



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-9996) [CBO] Generate appropriate join operator as per algorithm selected by CBO

[Ashutosh Chauhan (JIRA)]

Ashutosh Chauhan created HIVE-9996:
--

 Summary: [CBO] Generate appropriate join operator as per algorithm 
selected by CBO
 Key: HIVE-9996
 URL: https://issues.apache.org/jira/browse/HIVE-9996
 Project: Hive
  Issue Type: Task
  Components: CBO, Query Planning
Affects Versions: cbo-branch
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-9995) ACID compaction tries to compact a single file

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-9995:


 Summary: ACID compaction tries to compact a single file
 Key: HIVE-9995
 URL: https://issues.apache.org/jira/browse/HIVE-9995
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 1.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


Consider TestWorker.minorWithOpenInMiddle()

since there is an open txnId=23, this doesn't have any meaningful minor 
compaction work to do.  The system still tries to compact a single delta file 
for 21-22 id range, and effectively copies the file onto itself.

This is 1. inefficient and 2. can potentially affect a reader.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31696: HIVE-9792 Support interval type in expressions/predicates

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31696/#review76772
---

Ship it!


Ship It!

- Ashutosh Chauhan


On March 17, 2015, 6:37 p.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31696/
> ---
> 
> (Updated March 17, 2015, 6:37 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-9792
> https://issues.apache.org/jira/browse/HIVE-9792
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This adds 2 separate interval types:
> - year-month interval, which represents a date difference in years and 
> months. Internally represented as an offset of months.
> - day-time interval, which represents a date difference in 
> days/hours/minutes/seconds/nanoseconds. 1 day = 24 hours = 1440 minutes = 
> 86400 seconds. Internally represented as an offset of seconds (long) and 
> nanoseconds (int).
> 
> I've tried using the Oracle's Datetime and Interval Arithmetic rules with 
> regard to timestamp arithmetic, where the timestamp arithmetic is done in UTC 
> time. This results in similar behavior to below depending on the default time 
> zone in Hive:
> 
> SQL> select timestamp '1999-09-15 00:00:00 America/Los_Angeles' - timestamp 
> '1999-12-15 00:00:00 America/Los_Angeles' from dual;
> 
> TIMESTAMP'1999-09-1500:00:00AMERICA/LOS_ANGELES'-TIMESTAMP'1999-12-1500:00:
> ---
> -00091 01:00:00.0
> 
> SQL> select timestamp '1999-09-15 00:00:00 UTC' - timestamp '1999-12-15 
> 00:00:00 UTC' from dual;
> 
> TIMESTAMP'1999-09-1500:00:00UTC'-TIMESTAMP'1999-12-1500:00:00UTC'
> ---
> -00091 00:00:00.0
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/type/HiveIntervalDayTime.java 
> PRE-CREATION 
>   
> common/src/java/org/apache/hadoop/hive/common/type/HiveIntervalYearMonth.java 
> PRE-CREATION 
>   common/src/java/org/apache/hive/common/util/DateTimeMath.java PRE-CREATION 
>   common/src/java/org/apache/hive/common/util/DateUtils.java 92b7f62 
>   
> common/src/test/org/apache/hadoop/hive/common/type/TestHiveIntervalDayTime.java
>  PRE-CREATION 
>   
> common/src/test/org/apache/hadoop/hive/common/type/TestHiveIntervalYearMonth.java
>  PRE-CREATION 
>   common/src/test/org/apache/hive/common/util/TestDateTimeMath.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java fdbfab9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 
> 6a3c300 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  3a2c96c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  4dbac57 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
>  88c989f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g e7de6c8 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g d2d9989 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 0a05ceb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> e8217fd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseArithmetic.java
>  575c764 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseDTI.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseUnary.java 
> c5bec44 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPDTIMinus.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPDTIPlus.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMinus.java 
> 5419a19 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNegative.java 
> d653264 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPPlus.java 
> accd347 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToIntervalDayTime.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToIntervalYearMonth.java
>  PRE-CREATION 
>   
> ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFOPMinus.java 
> 24618c9 
>   ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFOPPlus.java 
> 4b2f7fb 
>   ql/src/test/queries/clientnegative/interval_1.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/interval_2.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/interval_3.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/invalid_arithmetic_type.q ad37cff 
>   ql/src/test/queries/clientpositive/interval_1.q PRE-CREATI

[jira] [Created] (HIVE-9994) Hive query plan returns sensitive data to external applications

[JIRA]

Sergio Peña created HIVE-9994:
-

 Summary: Hive query plan returns sensitive data to external 
applications
 Key: HIVE-9994
 URL: https://issues.apache.org/jira/browse/HIVE-9994
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Sergio Peña
Assignee: Sergio Peña


Some applications are using getQueryString() method from the QueryPlan class to 
get the query that is being executed by Hive. The query string returned is not 
redacted, and it is returning sensitive information that is logged in Navigator.

We need to return data redacted from the QueryPlan to avoid other applications 
to log sensitive data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31696: HIVE-9792 Support interval type in expressions/predicates

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31696/
---

(Updated March 17, 2015, 6:37 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

Incorporating RB feedback from Ashutosh


Bugs: HIVE-9792
https://issues.apache.org/jira/browse/HIVE-9792


Repository: hive-git


Description
---

This adds 2 separate interval types:
- year-month interval, which represents a date difference in years and months. 
Internally represented as an offset of months.
- day-time interval, which represents a date difference in 
days/hours/minutes/seconds/nanoseconds. 1 day = 24 hours = 1440 minutes = 86400 
seconds. Internally represented as an offset of seconds (long) and nanoseconds 
(int).

I've tried using the Oracle's Datetime and Interval Arithmetic rules with 
regard to timestamp arithmetic, where the timestamp arithmetic is done in UTC 
time. This results in similar behavior to below depending on the default time 
zone in Hive:

SQL> select timestamp '1999-09-15 00:00:00 America/Los_Angeles' - timestamp 
'1999-12-15 00:00:00 America/Los_Angeles' from dual;

TIMESTAMP'1999-09-1500:00:00AMERICA/LOS_ANGELES'-TIMESTAMP'1999-12-1500:00:
---
-00091 01:00:00.0

SQL> select timestamp '1999-09-15 00:00:00 UTC' - timestamp '1999-12-15 
00:00:00 UTC' from dual;

TIMESTAMP'1999-09-1500:00:00UTC'-TIMESTAMP'1999-12-1500:00:00UTC'
---
-00091 00:00:00.0


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/type/HiveIntervalDayTime.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/type/HiveIntervalYearMonth.java 
PRE-CREATION 
  common/src/java/org/apache/hive/common/util/DateTimeMath.java PRE-CREATION 
  common/src/java/org/apache/hive/common/util/DateUtils.java 92b7f62 
  
common/src/test/org/apache/hadoop/hive/common/type/TestHiveIntervalDayTime.java 
PRE-CREATION 
  
common/src/test/org/apache/hadoop/hive/common/type/TestHiveIntervalYearMonth.java
 PRE-CREATION 
  common/src/test/org/apache/hive/common/util/TestDateTimeMath.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java fdbfab9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 
6a3c300 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
 3a2c96c 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
 4dbac57 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
 88c989f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g e7de6c8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g d2d9989 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 0a05ceb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java e8217fd 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseArithmetic.java 
575c764 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseDTI.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseUnary.java 
c5bec44 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPDTIMinus.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPDTIPlus.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPMinus.java 
5419a19 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNegative.java 
d653264 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPPlus.java 
accd347 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToIntervalDayTime.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToIntervalYearMonth.java
 PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFOPMinus.java 
24618c9 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFOPPlus.java 
4b2f7fb 
  ql/src/test/queries/clientnegative/interval_1.q PRE-CREATION 
  ql/src/test/queries/clientnegative/interval_2.q PRE-CREATION 
  ql/src/test/queries/clientnegative/interval_3.q PRE-CREATION 
  ql/src/test/queries/clientnegative/invalid_arithmetic_type.q ad37cff 
  ql/src/test/queries/clientpositive/interval_1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/interval_2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/interval_3.q PRE-CREATION 
  ql/src/test/queries/clientpositive/interval_arithmetic.q PRE-CREATION 
  ql/src/test/queries/clientpositive/interval_comparison.q PRE-CREATION 
  ql/src/test/results/clientnegative/interval_1.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/interval_2.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/interval_3.q.out PRE-CREATION 
  ql/src

Re: Review Request 31696: HIVE-9792 Support interval type in expressions/predicates

[Jason Dere]

> On March 17, 2015, 5:32 p.m., Ashutosh Chauhan wrote:
> > common/src/java/org/apache/hadoop/hive/common/type/MutableHiveIntervalDayTime.java,
> >  line 21
> > 
> >
> > Unless there is an advantage of having seprate Mutable classes, its 
> > better to just make parent class mutable, otherwise it may be confusing to 
> > dev about which one to use when.

ok, will change


> On March 17, 2015, 5:32 p.m., Ashutosh Chauhan wrote:
> > common/src/java/org/apache/hadoop/hive/common/type/MutableHiveIntervalYearMonth.java,
> >  line 21
> > 
> >
> > same as prev comment.

will change


> On March 17, 2015, 5:32 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPDTIMinus.java,
> >  line 48
> > 
> >
> > It will be good to provide @Description annotation here, so that info 
> > can be displayed to user via `describe function`.

will fix


> On March 17, 2015, 5:32 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPDTIMinus.java,
> >  line 66
> > 
> >
> > Will be good to check if there are other ops allowed by standard. If 
> > there are, it will be good to leave a note here saying that although 
> > standard allows those, we havent added support for those yet.

I believe I have included the all the appropriate operations for 
addition/subtraction. The standard also includes multiplication/division by 
numeric values, which can be added in followup work.


> On March 17, 2015, 5:32 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPDTIPlus.java, 
> > line 48
> > 
> >
> > @Description annotation.

will fix


> On March 17, 2015, 5:32 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPDTIPlus.java, 
> > line 64
> > 
> >
> > Check with standard about allowed operands.

this should be fine


> On March 17, 2015, 5:32 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToIntervalDayTime.java,
> >  line 39
> > 
> >
> > @Description

will fix


> On March 17, 2015, 5:32 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToIntervalYearMonth.java,
> >  line 39
> > 
> >
> > @Description

will fix


- Jason


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31696/#review76685
---


On March 12, 2015, 7:50 p.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31696/
> ---
> 
> (Updated March 12, 2015, 7:50 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-9792
> https://issues.apache.org/jira/browse/HIVE-9792
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This adds 2 separate interval types:
> - year-month interval, which represents a date difference in years and 
> months. Internally represented as an offset of months.
> - day-time interval, which represents a date difference in 
> days/hours/minutes/seconds/nanoseconds. 1 day = 24 hours = 1440 minutes = 
> 86400 seconds. Internally represented as an offset of seconds (long) and 
> nanoseconds (int).
> 
> I've tried using the Oracle's Datetime and Interval Arithmetic rules with 
> regard to timestamp arithmetic, where the timestamp arithmetic is done in UTC 
> time. This results in similar behavior to below depending on the default time 
> zone in Hive:
> 
> SQL> select timestamp '1999-09-15 00:00:00 America/Los_Angeles' - timestamp 
> '1999-12-15 00:00:00 America/Los_Angeles' from dual;
> 
> TIMESTAMP'1999-09-1500:00:00AMERICA/LOS_ANGELES'-TIMESTAMP'1999-12-1500:00:
> ---
> -00091 01:00:00.0
> 
> SQL> select timestamp '1999-09-15 00:00:00 UTC' - timestamp '1999-12-15 
> 00:00:00 UTC' from dual;
> 
> TIMESTAMP'1999-09-1500:00:00UTC'-TIMESTAMP'1999-12-1500:00:00UTC'
> ---
> -00091 00:00:00.0
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/ty

Re: Review Request 31696: HIVE-9792 Support interval type in expressions/predicates

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31696/#review76685
---


Thanks for excellent tests! (Both unit as well as system)


common/src/java/org/apache/hadoop/hive/common/type/MutableHiveIntervalDayTime.java


Unless there is an advantage of having seprate Mutable classes, its better 
to just make parent class mutable, otherwise it may be confusing to dev about 
which one to use when.



common/src/java/org/apache/hadoop/hive/common/type/MutableHiveIntervalYearMonth.java


same as prev comment.



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPDTIMinus.java


It will be good to provide @Description annotation here, so that info can 
be displayed to user via `describe function`.



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPDTIMinus.java


Will be good to check if there are other ops allowed by standard. If there 
are, it will be good to leave a note here saying that although standard allows 
those, we havent added support for those yet.



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPDTIPlus.java


@Description annotation.



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPDTIPlus.java


Check with standard about allowed operands.



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToIntervalDayTime.java


@Description



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToIntervalYearMonth.java


@Description



ql/src/test/results/clientpositive/interval_arithmetic.q.out


Constant folding kicked in. Cool!


- Ashutosh Chauhan


On March 12, 2015, 7:50 p.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31696/
> ---
> 
> (Updated March 12, 2015, 7:50 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-9792
> https://issues.apache.org/jira/browse/HIVE-9792
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This adds 2 separate interval types:
> - year-month interval, which represents a date difference in years and 
> months. Internally represented as an offset of months.
> - day-time interval, which represents a date difference in 
> days/hours/minutes/seconds/nanoseconds. 1 day = 24 hours = 1440 minutes = 
> 86400 seconds. Internally represented as an offset of seconds (long) and 
> nanoseconds (int).
> 
> I've tried using the Oracle's Datetime and Interval Arithmetic rules with 
> regard to timestamp arithmetic, where the timestamp arithmetic is done in UTC 
> time. This results in similar behavior to below depending on the default time 
> zone in Hive:
> 
> SQL> select timestamp '1999-09-15 00:00:00 America/Los_Angeles' - timestamp 
> '1999-12-15 00:00:00 America/Los_Angeles' from dual;
> 
> TIMESTAMP'1999-09-1500:00:00AMERICA/LOS_ANGELES'-TIMESTAMP'1999-12-1500:00:
> ---
> -00091 01:00:00.0
> 
> SQL> select timestamp '1999-09-15 00:00:00 UTC' - timestamp '1999-12-15 
> 00:00:00 UTC' from dual;
> 
> TIMESTAMP'1999-09-1500:00:00UTC'-TIMESTAMP'1999-12-1500:00:00UTC'
> ---
> -00091 00:00:00.0
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/type/HiveIntervalDayTime.java 
> PRE-CREATION 
>   
> common/src/java/org/apache/hadoop/hive/common/type/HiveIntervalYearMonth.java 
> PRE-CREATION 
>   
> common/src/java/org/apache/hadoop/hive/common/type/MutableHiveIntervalDayTime.java
>  PRE-CREATION 
>   
> common/src/java/org/apache/hadoop/hive/common/type/MutableHiveIntervalYearMonth.java
>  PRE-CREATION 
>   common/src/java/org/apache/hive/common/util/DateTimeMath.java PRE-CREATION 
>   common/src/java/org/apache/hive/common/util/DateUtils.java 92b7f62 
>   
> common/src/test/org/apache/hadoop/hive/common/type/TestHiveIntervalDayTime.java
>  PRE-CREATION 
>   
> common/src/test/org/apache/hadoop/hive/common/type/TestHiveIntervalYearMonth.java
>  PRE-CREATION 
>   common/src/test/org/apache/hive/common/util/TestDateTimeMath.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 20811ab 
>   ql/src/java/org/apache/hadoop/hive/ql/

[jira] [Created] (HIVE-9993) Retrying task could use cached bad operators [Spark Branch]

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-9993:
-

 Summary: Retrying task could use cached bad operators [Spark 
Branch]
 Key: HIVE-9993
 URL: https://issues.apache.org/jira/browse/HIVE-9993
 Project: Hive
  Issue Type: Bug
Affects Versions: spark-branch
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Fix For: spark-branch


For a Spark task, it could be retried on the same executor in case some 
failures. In retrying, the cache task could be used. Since the operators in the 
task are already initialized, they won't be initialized again. The partial data 
in these operators could lead to wrong final results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-9992) Operator Precedence is incorrect. XOR should be after minus/plus

[Alexander Pivovarov (JIRA)]

Alexander Pivovarov created HIVE-9992:
-

 Summary: Operator Precedence is incorrect. XOR should be after 
minus/plus
 Key: HIVE-9992
 URL: https://issues.apache.org/jira/browse/HIVE-9992
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Alexander Pivovarov
Assignee: Alexander Pivovarov


XOR ^ has higher precedence than -/+ in hive
But in java and in MySQL XOR has lower precedence
http://dev.mysql.com/doc/refman/5.0/en/operator-precedence.html
http://docs.oracle.com/javase/tutorial/java/nutsandbolts/operators.html

{code}
// now
select 4 - 2 ^ 8 - 7;
-13

// should be
select 4 - 2 ^ 8 - 7;
3
{code}

need to fix
IdentifiersParser.g  BITWISEXOR



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-9991) Cannot do a SELECT on external tables that are on S3 due to Encryption error

[JIRA]

Sergio Peña created HIVE-9991:
-

 Summary: Cannot do a SELECT on external tables that are on S3 due 
to Encryption error
 Key: HIVE-9991
 URL: https://issues.apache.org/jira/browse/HIVE-9991
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Sergio Peña
Assignee: Sergio Peña


I cannot do any select query on external tables that are not part of HDFS. For 
example S3.

{code}
Select * from my_table limit 10;

FAILED: SemanticException Unable to determine if s3n://my-bucket/is encrypted: 
java.lang.IllegalArgumentException: Wrong FS: s3n://my-bucket/, expected: 
hdfs://0.0.0.0:8020
{code}

This error is due to a internal function that checks if a table is encrypted or 
not. This is only supported on HDFS files, but the check is happening on any 
external table as well causing the above error.

To fix this, we should check for encrypted tables only for HDFS tables. And 
skip the check for any other file schema.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-9990) TestMultiSessionsHS2WithLocalClusterSpark is failing

[Xuefu Zhang (JIRA)]

Xuefu Zhang created HIVE-9990:
-

 Summary: TestMultiSessionsHS2WithLocalClusterSpark is failing
 Key: HIVE-9990
 URL: https://issues.apache.org/jira/browse/HIVE-9990
 Project: Hive
  Issue Type: Bug
  Components: Spark
Affects Versions: 1.2.0
Reporter: Xuefu Zhang


At least sometimes. I can reproduce it with "mvn test 
-Dtest=TestMultiSessionsHS2WithLocalClusterSpark -Phadoop-2" consistently on my 
local box.
{code}
---
 T E S T S
---
Running org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 92.438 sec <<< 
FAILURE! - in org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark
testSparkQuery(org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark)  
Time elapsed: 21.514 sec  <<< ERROR!
java.util.concurrent.ExecutionException: java.sql.SQLException: Error while 
processing statement: FAILED: Execution Error, return code 3 from 
org.apache.hadoop.hive.ql.exec.spark.SparkTask
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:296)
at 
org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:392)
at 
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.verifyResult(TestMultiSessionsHS2WithLocalClusterSpark.java:244)
at 
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testKvQuery(TestMultiSessionsHS2WithLocalClusterSpark.java:220)
at 
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.access$000(TestMultiSessionsHS2WithLocalClusterSpark.java:53)
{code}

The error was also seen in HIVE-9934 test run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-9989) hive on tez group by NPE

[tangjunjie (JIRA)]

tangjunjie created HIVE-9989:


 Summary: hive on tez group by NPE
 Key: HIVE-9989
 URL: https://issues.apache.org/jira/browse/HIVE-9989
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.14.0
 Environment: hive 0.14 tez 0.53
Reporter: tangjunjie


set hive.execution.engine=tez   run sql throw NPE 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-9988) Evaluating UDF before query is run

[JIRA]

Ådne Brunborg created HIVE-9988:
---

 Summary: Evaluating UDF before query is run
 Key: HIVE-9988
 URL: https://issues.apache.org/jira/browse/HIVE-9988
 Project: Hive
  Issue Type: Improvement
Reporter: Ådne Brunborg


When using UDFs on partition column in Hive, all partitions are scanned before 
the UDF is resolved. 

If the UDF could be evaluated before query is run, this would greatly improve 
performance in cases like this.

Example - the table has a partition by datestamp (bigint): 

The following where clause touches upon all 82 partitions:
{{WHERE datestamp=cast(from_unixtime(unix_timestamp(),'MMdd') as bigint)}}
{{15/03/16 09:21:53 INFO mapred.FileInputFormat: Total input paths to process : 
82}}

…whereas the following only touches the one partition:
{{WHERE datestamp=20150316}}
{{15/03/16 09:23:06 INFO input.FileInputFormat: Total input paths to process : 
1}}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)