[jira] [Created] (HIVE-15338) Wrong result from vectorized DATEDIFF with scalar parameter of type DATE

2016-12-01 Thread Matt McCline (JIRA) 
Matt McCline created HIVE-15338:
---

 Summary: Wrong result from vectorized DATEDIFF with scalar 
parameter of type DATE
 Key: HIVE-15338
 URL: https://issues.apache.org/jira/browse/HIVE-15338
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical


Vectorization in vectorized DATEDIFF when parameter is (date '12-01-16', 
column). Jason noted that ('12-01-16', column) treats the first parameter as a 
string and is a different case.
Current Q file test vectorized_date_funcs.q DOES NOT test the date scalar type 
case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 54284: StatsOptimizer should return no rows on empty table with group by

2016-12-01 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54284/
---

Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-13452


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java 17510e9 
  ql/src/test/queries/clientpositive/constGby.q PRE-CREATION 
  ql/src/test/results/clientpositive/constGby.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/54284/diff/


Testing
---


Thanks,

pengcheng xiong

[jira] [Created] (HIVE-15337) Need to specify starttime when putting Compaction job into "attempted" state

2016-12-01 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-15337:
-

 Summary: Need to specify starttime when putting Compaction job 
into "attempted" state
 Key: HIVE-15337
 URL: https://issues.apache.org/jira/browse/HIVE-15337
 Project: Hive
  Issue Type: Improvement
  Components: Transactions
Affects Versions: 1.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


W/o this SHOW COMPACTIONS output is not as useful



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15336) LLAP UI: Expand the UI to multiple pages

2016-12-01 Thread Gopal V (JIRA) 
Gopal V created HIVE-15336:
--

 Summary: LLAP UI: Expand the UI to multiple pages
 Key: HIVE-15336
 URL: https://issues.apache.org/jira/browse/HIVE-15336
 Project: Hive
  Issue Type: Improvement
  Components: llap
Affects Versions: 2.2.0
Reporter: Gopal V
Assignee: Gopal V


The current LLAP UI is a single page - expand the UI to multiple tabs and add 
more information into the UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15335) Fast Decimal

2016-12-01 Thread Matt McCline (JIRA) 
Matt McCline created HIVE-15335:
---

 Summary: Fast Decimal
 Key: HIVE-15335
 URL: https://issues.apache.org/jira/browse/HIVE-15335
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical


Replace HiveDecimal implementation that currently represents the decimal 
internally as a BigDecimal with a faster version that does not allocate extra 
objects

Replace HiveDecimalWritable implementation with a faster version that has new 
mutable* calls (e.g. mutableAdd, mutableEnforcePrecisionScale, etc) and stores 
the result as a fast decimal instead of a slow byte array containing a 
serialized BigInteger.

Provide faster ways to serialize/deserialize decimals.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15334) HIVE-13945 changed scale rules for division

2016-12-01 Thread Jason Dere (JIRA) 
Jason Dere created HIVE-15334:
-

 Summary: HIVE-13945 changed scale rules for division
 Key: HIVE-15334
 URL: https://issues.apache.org/jira/browse/HIVE-15334
 Project: Hive
  Issue Type: Bug
  Components: Types
Reporter: Jason Dere


Looks like HIVE-13945 change the decimal division precision/scale rules - the 
explanation being "Changed the default decimal precision in division, not sure 
why it was so low by default." 
(https://issues.apache.org/jira/browse/HIVE-13945?focusedCommentId=15354403&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15354403)

As far as I can tell this causes decimal division to have a minimum scale of 18.
cc [~sershe] - the rules that were in place were based on the SQL Server 
precision/scale rules in https://msdn.microsoft.com/en-us/library/ms190476.aspx

I'd like to revert this change to precision/scale rules



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15333) Add a FetchTask to REPL DUMP plan for reading dump uri, last repl id as ResultSet

2016-12-01 Thread Vaibhav Gumashta (JIRA) 
Vaibhav Gumashta created HIVE-15333:
---

 Summary: Add a FetchTask to REPL DUMP plan for reading dump uri, 
last repl id as ResultSet
 Key: HIVE-15333
 URL: https://issues.apache.org/jira/browse/HIVE-15333
 Project: Hive
  Issue Type: Sub-task
  Components: repl
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta


We're writing the return values to a file, but we don't add FetchTask while 
planning. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15332) REPL LOAD & DUMP support for incremental CREATE_TABLE/ADD_PTN

2016-12-01 Thread Sushanth Sowmyan (JIRA) 
Sushanth Sowmyan created HIVE-15332:
---

 Summary: REPL LOAD & DUMP support for incremental 
CREATE_TABLE/ADD_PTN
 Key: HIVE-15332
 URL: https://issues.apache.org/jira/browse/HIVE-15332
 Project: Hive
  Issue Type: Sub-task
  Components: repl
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan


We need to add in support for REPL LOAD and REPL DUMP of incremental events, 
and we need to be able to replicate creates, for a start. This jira tracks the 
inclusion of CREATE_TABLE/ADD_PARTITION event support to REPL DUMP & LOAD.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15331) Decimal multiplication with high precision/scale often returns NULL

2016-12-01 Thread Jason Dere (JIRA) 
Jason Dere created HIVE-15331:
-

 Summary: Decimal multiplication with high precision/scale often 
returns NULL
 Key: HIVE-15331
 URL: https://issues.apache.org/jira/browse/HIVE-15331
 Project: Hive
  Issue Type: Bug
  Components: Types
Reporter: Jason Dere
Assignee: Jason Dere


{noformat}
create temporary table dec (a decimal(38,18));
insert into dec values(100.0);

hive> select a*a from dec;
OK
NULL
Time taken: 0.165 seconds, Fetched: 1 row(s)
{noformat}

Looks like the reason is because the result of decimal(38,18) * decimal(38,18) 
only has 2 digits of precision for integers:

{noformat}
hive> set hive.explain.user=false;
hive> explain select a*a from dec;
OK
STAGE DEPENDENCIES:
  Stage-0 is a root stage

STAGE PLANS:
  Stage: Stage-0
Fetch Operator
  limit: -1
  Processor Tree:
TableScan
  alias: dec
  Select Operator
expressions: (a * a) (type: decimal(38,36))
outputColumnNames: _col0
ListSink

Time taken: 0.039 seconds, Fetched: 15 row(s)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Apache Hive 2.1.1 Release Candidate 1

2016-12-01 Thread Jesus Camacho Rodriguez 
Sergio,

I used OSX 10.11.

Maybe it has to do with the version used to verify the md5? Can you just try to 
verify manually?

$ md5sum apache-hive-2.1.1-bin.tar.gz > apache-hive-2.1.1-bin.tar.gz.md5.self
$ diff -q apache-hive-2.1.1-bin.tar.gz.md5 apache-hive-2.1.1-bin.tar.gz.md5.self


About the KEYS, my key is not in the file you referred, I should have added it 
before.
You can find it here:
https://people.apache.org/keys/committer/jcamacho.asc

Let me know if that solves your problem.

--
Jesús



On 11/30/16, 9:08 PM, "Sergio Pena"  wrote:

>Jesus,
>
>I tried verifying the md5 and gpg signatures, but I get these errors:
>
>hive/packaging/target⟫ md5sum -c apache-hive-2.1.1-bin.tar.gz.md5
>apache-hive-2.1.1-bin.tar.gz: FAILED
>md5sum: WARNING: 1 computed checksum did NOT match
>
>hive/packaging/target⟫ gpg --verify apache-hive-2.1.1-bin.tar.gz.asc
>apache-hive-2.1.1-bin.tar.gz
>gpg: Signature made Tue 29 Nov 2016 01:57:04 PM CST
>gpg:using RSA key 931E4AB3C516B444
>gpg: Can't check signature: No public key
>
>I'm using ubuntu, so I think the md5 differs from OSX and Linux machines. I
>remember seeing this problem before. What OS did you use?
>
>for the GPG keys, I imported the KEYS file mentioned in the Wiki, but I
>still get that error. Any idea what I'm missing?
>
>On Tue, Nov 29, 2016 at 6:23 PM, Gary Gregory 
>wrote:
>
>> FWIW, running 'mvn clean install' has been failing on Git master for a long
>> time on Windows. Will that ever be fixed?
>>
>> Gary
>>
>> On Tue, Nov 29, 2016 at 12:17 PM, Jesus Camacho Rodriguez <
>> jcama...@apache.org> wrote:
>>
>> > Apache Hive 2.1.1 Release Candidate 1 is available here:
>> > http://people.apache.org/~jcamacho/hive-2.1.1-rc1/
>> >
>> > Maven artifacts are available here:
>> > https://repository.apache.org/content/repositories/orgapachehive-1066/
>> >
>> > Source tag for RC1 is at:
>> > https://github.com/apache/hive/releases/tag/release-2.1.1-rc1/
>> >
>> > Voting will conclude in 72 hours.
>> >
>> > Hive PMC Members: Please test and vote.
>> >
>> > Thanks.
>> >
>> >
>> >
>> >
>>
>>
>> --
>> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
>> Java Persistence with Hibernate, Second Edition
>> > tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1617290459&
>> linkCode=as2&tag=garygregory-20&linkId=cadb800f39946ec62ea2b1af9fe6a2b8>
>>
>> > 1617290459>
>> JUnit in Action, Second Edition
>> > tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1935182021&
>> linkCode=as2&tag=garygregory-20&linkId=31ecd1f6b6d1eaf8886ac902a24de418%22
>> >
>>
>> > 1935182021>
>> Spring Batch in Action
>> > tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1935182951&
>> linkCode=%7B%7BlinkCode%7D%7D&tag=garygregory-20&linkId=%7B%
>> 7Blink_id%7D%7D%22%3ESpring+Batch+in+Action>
>> > 1935182951>
>> Blog: http://garygregory.wordpress.com
>> Home: http://garygregory.com/
>> Tweet! http://twitter.com/GaryGregory
>>

Re: [VOTE] Apache Hive 2.1.1 Release Candidate 1

2016-12-01 Thread Alan Gates 
+1, checked the signatures, checked that the new warning about JSON is in the 
NOTICE file, did a build (sans unit tests), and checked for any binaries in the 
source distribution.

I didn’t see the signature issues that Sergio saw.

Alan.

> On Nov 29, 2016, at 12:17, Jesus Camacho Rodriguez  
> wrote:
> 
> Apache Hive 2.1.1 Release Candidate 1 is available here:
> http://people.apache.org/~jcamacho/hive-2.1.1-rc1/
> 
> Maven artifacts are available here:
> https://repository.apache.org/content/repositories/orgapachehive-1066/
> 
> Source tag for RC1 is at:
> https://github.com/apache/hive/releases/tag/release-2.1.1-rc1/
> 
> Voting will conclude in 72 hours.
> 
> Hive PMC Members: Please test and vote.
> 
> Thanks.
> 
> 
>

[jira] [Created] (HIVE-15330) Bump JClouds version to 2.0.0 on Hive/Ptest

2016-12-01 Thread JIRA 
Sergio Peña created HIVE-15330:
--

 Summary: Bump JClouds version to 2.0.0 on Hive/Ptest
 Key: HIVE-15330
 URL: https://issues.apache.org/jira/browse/HIVE-15330
 Project: Hive
  Issue Type: Task
  Components: Hive, Testing Infrastructure
Reporter: Sergio Peña
Assignee: Sergio Peña


JClouds 2.0.0 fixes several issues with Google Compute Engine API. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15329) NullPointerException might occur when create table

2016-12-01 Thread Meilong Huang (JIRA) 
Meilong Huang created HIVE-15329:


 Summary: NullPointerException might occur when create table
 Key: HIVE-15329
 URL: https://issues.apache.org/jira/browse/HIVE-15329
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 2.1.0, 2.0.0, 1.1.0, 1.2.0, 1.0.0, 0.9.0, 0.8.0
Reporter: Meilong Huang
Assignee: Meilong Huang


NullPointerException might occur when table.getParameters() returns null when 
method isNonNativeTable is invoked in class MetaStoreUtils.

public static boolean isNonNativeTable(Table table) {
if (table == null) {
  return false;
}
return 
(table.getParameters().get(hive_metastoreConstants.META_TABLE_STORAGE) != null);
  }

This will cause a stack trace without any suggestive information at client:
org.apache.hadoop.hive.metastore.api.MetaException: 
java.lang.NullPointerException
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result$create_table_with_environment_context_resultStandardScheme.read...




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15328) Inconsistent/incorrect handling of NULL in nested structs

2016-12-01 Thread David Maughan (JIRA) 
David Maughan created HIVE-15328:


 Summary: Inconsistent/incorrect handling of NULL in nested structs
 Key: HIVE-15328
 URL: https://issues.apache.org/jira/browse/HIVE-15328
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 2.1.0
Reporter: David Maughan


h2. Overview

Performing {{IS NULL}} checks against a null struct that is generated as part 
of a UDF correctly returns {{true}}. However, the same check against the same 
null struct that has been persisted to a table incorrectly returns {{false}}. 
Additionally, when a child field of the null struct is inspected in the same 
query, the result of the null check on the struct itself reverses itself to 
{{true}}.

The issue does not appear to be dependent on the storage format of the table as 
the same result is repeated with TEXTFILE, PARQUET, ORC and AVRO.

h2. Example

In this example I have used {{if(1=1, null, named_struct('c', 1)}} as a crude 
method of generating a simple null struct.

h4. 'b' is correctly reported as {{true}}.
{code}
hive> select
>   b is null,
>   b
> from (
>   select
> if(1=1, null, named_struct('c', 1)) as b
>   ) as a;
OK
trueNULL
{code}

h4. 'b' is correctly reported as {{true}} when also inspecting 'b.c'.
{code}
hive>
> select
>   b is null,
>   b.c is null,
>   b
> from (
>   select
> if(1=1, null, named_struct('c', 1)) as b
>   ) as a;
OK
truetrueNULL
{code}

h4. Persist the data to a table
{code}
hive>
> create table a
>   as
> select
>   if(1=1, null, named_struct('c', 1)) as b;
OK
{code}

h4. 'b' is incorrectly reported as {{false}}.
{code}
hive>
> select
>   b is null,
>   b
> from a;
OK
false   NULL
{code}

h4. 'b' is now correctly reported as {{true}} when also inspecting 'b.c'.
{code}
hive>
> select
>   b is null,
>   b.c is null,
>   b
> from a;
OK
truetrueNULL
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15327) Outerjoin might produce wrong result depending on joinEmitInterval value

2016-12-01 Thread Jesus Camacho Rodriguez (JIRA) 
Jesus Camacho Rodriguez created HIVE-15327:
--

 Summary: Outerjoin might produce wrong result depending on 
joinEmitInterval value
 Key: HIVE-15327
 URL: https://issues.apache.org/jira/browse/HIVE-15327
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 1.3.0, 2.2.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez
Priority: Critical


If joinEmitInterval is smaller than the group size, outerjoins might produce 
records with NULL appended values multiple times (once per group).

HIVE-4689 targeted the same problem. However, the fix does not seem to cover 
all cases (in particular, it will not apply to left outer joins with filter 
conditions on the left input). The solution in HIVE-4689 was to disable 
(override) joinEmitInterval value for those cases. This fix follows the same 
approach.

To reproduce the problem:
{code}
set hive.strict.checks.cartesian.product=false;
set hive.join.emit.interval=1;

CREATE TABLE test1 (key INT, value INT, col_1 STRING);
INSERT INTO test1 VALUES (99, 0, 'Alice');
INSERT INTO test1 VALUES (99, 2, 'Mat');
INSERT INTO test1 VALUES (100, 1, 'Bob');
INSERT INTO test1 VALUES (101, 2, 'Car');

CREATE TABLE test2 (key INT, value INT, col_2 STRING);
INSERT INTO test2 VALUES (102, 2, 'Del');
INSERT INTO test2 VALUES (103, 2, 'Ema');
INSERT INTO test2 VALUES (104, 3, 'Fli');


-- Equi-condition and condition on one input (left outer join)
SELECT *
FROM test1 LEFT OUTER JOIN test2
ON (test1.value=test2.value AND test1.key between 100 and 102)
LIMIT 10;

-- Condition on one input (left outer join)
SELECT *
FROM test1 LEFT OUTER JOIN test2
ON (test1.key between 100 and 102)
LIMIT 10;
{code}


For the *first* query, current (incorrect) result is:
{noformat}
 99 0   Alice   NULLNULLNULL
 1001   Bob NULLNULLNULL
 1012   Car 103 2   Ema
 99 2   Mat NULLNULLNULL
 1012   Car 102 2   Del
 99 2   Mat NULLNULLNULL
{noformat}

Expected (correct) result is:
{noformat}
 99 0   Alice   NULLNULLNULL
 1001   Bob NULLNULLNULL
 1012   Car 103 2   Ema
 1012   Car 102 2   Del
 99 2   Mat NULLNULLNULL
{noformat}


For the *second* query, current (incorrect) result is:
{noformat}
 1012   Car 104 3   Fli
 1001   Bob 104 3   Fli
 99 2   Mat NULLNULLNULL
 99 0   Alice   NULLNULLNULL
 1012   Car 103 2   Ema
 1001   Bob 103 2   Ema
 99 2   Mat NULLNULLNULL
 99 0   Alice   NULLNULLNULL
 1012   Car 102 2   Del
 1001   Bob 102 2   Del
{noformat}

Expected (correct) result is:
{noformat}
 1012   Car 104 3   Fli
 1012   Car 103 2   Ema
 1012   Car 102 2   Del
 1001   Bob 104 3   Fli
 1001   Bob 103 2   Ema
 1001   Bob 102 2   Del
 99 2   Mat NULLNULLNULL
 99 0   Alice   NULLNULLNULL
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15326) Hive shims report Unrecognized Hadoop major version number: 3.0.0-alpha2-SNAPSHOT

2016-12-01 Thread Steve Loughran (JIRA) 
Steve Loughran created HIVE-15326:
-

 Summary: Hive shims report Unrecognized Hadoop major version 
number: 3.0.0-alpha2-SNAPSHOT
 Key: HIVE-15326
 URL: https://issues.apache.org/jira/browse/HIVE-15326
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 1.2.1
 Environment: Hadoop trunk branch
Reporter: Steve Loughran


Hive built against Hadoop 2 fails to run against Hadoop 3.x, 
declaring:{{Unrecognized Hadoop major version number: 3.0.0-alpha2-SNAPSHOT}}

Refusing to play on Hadoop 3.x may actually be the correct behaviour, though 
ideally we've retained API compatibility to everything works (maybe with some 
CP tweaking).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)