[jira] [Created] (HIVE-14341) Altered skewed location is not respected for list bucketing

2016-07-26 Thread Aihua Xu (JIRA) 
Aihua Xu created HIVE-14341:
---

 Summary: Altered skewed location is not respected for list 
bucketing
 Key: HIVE-14341
 URL: https://issues.apache.org/jira/browse/HIVE-14341
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Affects Versions: 2.0.1
Reporter: Aihua Xu
Assignee: Aihua Xu


CREATE TABLE list_bucket_single (key STRING, value STRING)
  SKEWED BY (key) ON (1,5,6) STORED AS DIRECTORIES;

alter table list_bucket_single set skewed location 
(''1"="/user/hive/warehouse/hdfs_skewed/new1");

While when you insert a row to key 1, the location falls back to the default 
one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 49766: HIVE-14035 Enable predicate pushdown to delta files created by ACID Transactions

2016-07-26 Thread Saket Saurabh 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49766/
---

(Updated July 26, 2016, 11:30 a.m.)


Review request for hive and Eugene Koifman.


Changes
---

Add more UTs to specifically test AcidUtils and various other compaction 
scenarios.


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-14035

In current Hive version, delta files created by ACID transactions do not allow 
predicate pushdown if they contain any update/delete events. This is done to 
preserve correctness when following a multi-version approach during event 
collapsing, where an update event overwrites an existing insert event. 
This JIRA proposes to split an update event into a combination of a delete 
event followed by a new insert event, that can enable predicate push down to 
all delta files without breaking correctness. To support backward compatibility 
for this feature, this JIRA also proposes to add some sort of versioning to 
ACID that can allow different versions of ACID transactions to co-exist 
together.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e92466f 
  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
 14f7316 
  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/AbstractRecordWriter.java
 974c6b8 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 ca2a912 
  metastore/if/hive_metastore.thrift 4d92b73 
  metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h ae14bd1 
  metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp f982bf2 
  
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
 5a666f2 
  metastore/src/gen/thrift/gen-php/metastore/Types.php f505208 
  metastore/src/gen/thrift/gen-py/hive_metastore/constants.py d1c07a5 
  metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb eeccc84 
  
metastore/src/java/org/apache/hadoop/hive/metastore/TransactionalValidationListener.java
 3e74675 
  orc/src/java/org/apache/orc/impl/TreeReaderFactory.java c4a2093 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java db6848a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 57b6c67 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 23a13d6 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java c150ec5 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 945b828 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 63d02fb 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java b0f8c8b 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 1a1af28 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 9d927bd 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 8cf261d 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 6caca98 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java d48e441 
  ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java b83cea4 

Diff: https://reviews.apache.org/r/49766/diff/


Testing
---

Tests for the feature are in 
ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java. These are mostly 
integration tests that test end-to-end insert/update/delete scenarios followed 
by compaction and cleaning.


Thanks,

Saket Saurabh

[jira] [Created] (HIVE-14344) Intermittent failures caused by leaking delegation tokens

2016-07-26 Thread Chris Drome (JIRA) 
Chris Drome created HIVE-14344:
--

 Summary: Intermittent failures caused by leaking delegation tokens
 Key: HIVE-14344
 URL: https://issues.apache.org/jira/browse/HIVE-14344
 Project: Hive
  Issue Type: Bug
  Components: Tez
Affects Versions: 2.1.0, 1.2.1
Reporter: Chris Drome
Assignee: Chris Drome


We have experienced random job failures caused by leaking delegation tokens. 
The Tez child task will fail because it is attempting to read from the 
delegation tokens directory of a different (related) task.

Failure results in the following type of stack trace:

{noformat}
2016-07-21 16:57:18,061 [FATAL] [TezChild] |tez.ReduceRecordSource|: 
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
processing row (tag=0) 
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:370)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:292)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:249)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:148)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:362)
at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738)
at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
java.lang.RuntimeException: java.io.IOException: Exception reading 
file:/grid/4/tmp/yarn-local/usercache/.../appcache/application_1468602386465_489814/container_e02_1468602386465_489814_01_01/container_tokens
at 
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:237)
at 
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:74)
at 
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:650)
at 
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:756)
at 
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinObject(CommonMergeJoinOperator.java:316)
at 
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinOneGroup(CommonMergeJoinOperator.java:279)
at 
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinOneGroup(CommonMergeJoinOperator.java:272)
at 
org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:258)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:361)
... 17 more
Caused by: java.lang.RuntimeException: java.io.IOException: Exception reading 
file:/grid/4/tmp/yarn-local/usercache/.../appcache/application_1468602386465_489814/container_e02_1468602386465_489814_01_01/container_tokens
at 
org.apache.hadoop.mapreduce.security.TokenCache.mergeBinaryTokens(TokenCache.java:141)
at 
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:119)
at 
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
at 
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
at 
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:206)
at 
org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:45)
at 
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
at 
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:222)
... 25 more
Caused by: java.io.IOException: Exception reading

[jira] [Created] (HIVE-14343) HiveDriverRunHookContext's command is null in HS2 mode

2016-07-26 Thread Chao Sun (JIRA) 
Chao Sun created HIVE-14343:
---

 Summary: HiveDriverRunHookContext's command is null in HS2 mode
 Key: HIVE-14343
 URL: https://issues.apache.org/jira/browse/HIVE-14343
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Chao Sun


Looking at the {{Driver#runInternal(String command, boolean alreadyCompiled)}}:
{code}
HiveDriverRunHookContext hookContext = new 
HiveDriverRunHookContextImpl(conf, command);
// Get all the driver run hooks and pre-execute them.
List driverRunHooks;
{code}

The context is initialized with the {{command}} passed in to the method. 
However, this command is always null if {{alreadyCompiled}} is true, which is 
the case for HS2 mode.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 50434: HIVE-14323: Reduce number of FS permissions and redundant FS operations

2016-07-26 Thread Chris Nauroth 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50434/#review143565
---


Ship it!




+1 (non-binding).  Thank you, Rajesh.

- Chris Nauroth


On July 26, 2016, 2:50 a.m., Rajesh Balamohan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50434/
> ---
> 
> (Updated July 26, 2016, 2:50 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Chris Nauroth.
> 
> 
> Bugs: HIVE-14323
> https://issues.apache.org/jira/browse/HIVE-14323
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14323: Reduce number of FS permissions and redundant FS operations
> 
> 1. Honor HIVE_WAREHOUSE_SUBDIR_INHERIT_PERMS when getting stage directory in 
> Context and in mkdir() in FileUtils
> 2. Reduce FS calls like exists()-->delete() to just delete().
> 3. Similar to #2 in Hive.replaceFiles()
> 4. Refactor Hive.trashFiles() so that it can make use of FileStatus 
> information if available (s3 FS lookups are expensive)
> 5. Refactor Hive.removeTempOrDuplicateFiles() so that it can make use of 
> FileStatus information if available (s3 FS lookups are expensive)
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/FileUtils.java 23fcc8a 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java ec5d693 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java cadda8f 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 9d927bd 
> 
> Diff: https://reviews.apache.org/r/50434/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Rajesh Balamohan
> 
>

[jira] [Created] (HIVE-14340) Add a new hook triggers before query compilation and after query execution

2016-07-26 Thread Chao Sun (JIRA) 
Chao Sun created HIVE-14340:
---

 Summary: Add a new hook triggers before query compilation and 
after query execution
 Key: HIVE-14340
 URL: https://issues.apache.org/jira/browse/HIVE-14340
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Affects Versions: 2.2.0
Reporter: Chao Sun
Assignee: Chao Sun


In some cases we may need to have a hook that activates before a query 
compilation and after its execution. For instance, dynamically generate a UDF 
specifically for the running query and clean up the resource after the query is 
done. The current hooks only covers pre & post semantic analysis, pre & post 
query execution, which doesn't fit the requirement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14342) Beeline output is garbled when executed from a remote shell

2016-07-26 Thread Naveen Gangam (JIRA) 
Naveen Gangam created HIVE-14342:


 Summary: Beeline output is garbled when executed from a remote 
shell
 Key: HIVE-14342
 URL: https://issues.apache.org/jira/browse/HIVE-14342
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Affects Versions: 2.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


{code}
use default;
create table clitest (key int, name String, value String);
insert into table clitest values 
(1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0");
{code}

then run a select query
{code} 
# cat /tmp/select.sql 
set hive.execution.engine=mr;
select key,name,value 
from clitest 
where value="1" limit 1;
{code}

Then run beeline via a remote shell, for example
$ ssh -l root  "sudo -u hive beeline -u 
jdbc:hive2://localhost:1 -n hive -p hive --silent=true --outputformat=csv2 
-f /tmp/select.sql" 
root@'s password: 
16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree 
module jar containing PrefixTreeCodec is not present.  Continuing without it.
nullkey,name,value 
1,TRUE,1
null   
$
{code}

In older releases that the output is as follows
{code}
$ ssh -l root  "sudo -u hive beeline -u 
jdbc:hive2://localhost:1 -n hive -p hive --silent=true --outputformat=csv2 
-f /tmp/run.sql" 
Are you sure you want to continue connecting (yes/no)? yes
root@'s password: 
16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree 
module jar containing PrefixTreeCodec is not present.  Continuing without it.



key,name,value
1,TRUE,1


$
{code}

The output contains nulls instead of blank lines. This is due to the use of 
-Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able 
to run beeline as a background process. But this is the unfortunate side effect 
of that fix.

Running beeline in background also produces garbled output.
{code}
# beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true 
--outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > /tmp/beeline.txt &
# cat /tmp/beeline.txt 
null1,TRUE,1   

#
{code}

So I think the use of jline.UnsupportedTerminal should be documented but not 
used automatically by beeline under the covers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 50359: HIVE-14270: Write temporary data to HDFS when doing inserts on tables located on S3

2016-07-26 Thread Sergio Pena 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50359/
---

(Updated July 26, 2016, 10:05 p.m.)


Review request for hive.


Changes
---

Changes added on this patch:
- create a helper method on Context to get the temporary directory depending of 
the filesystem
- add more tests
- fix issue where staging directories where copied to s3


Bugs: HIVE-14270
https://issues.apache.org/jira/browse/HIVE-14270


Repository: hive-git


Description
---

This patch will create a temporary directory for Hive intermediate data on HDFS 
when S3 tables are used.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/ObjectStorageUtils.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/TestObjectStorageUtils.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java 
ec5d693d28a40925c44f844a05ebf3f5c10173c9 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
9d927bd1a519f79bc7fa88c3b7e5c6cc2ef0637f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
2671cb1cf2ef74f9d6628f8cdf3f5ac99283dbd8 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestContext.java PRE-CREATION 

Diff: https://reviews.apache.org/r/50359/diff/


Testing
---

NO PATCH
** NON-PARTITIONED TABLE

- create table dummy (id int);  
 3.651s
- insert into table s3dummy values (1); 
39.231s
- insert overwrite table s3dummy values (1);
42.569s
- insert overwrite directory 's3a://spena-bucket/dirs/s3dummy' select * from 
dummy; 30.136s

EXTERNAL TABLE

- create table s3dummy_ext like s3dummy location 
's3a://spena-bucket/user/hive/warehouse/s3dummy';   9.297s
- insert into table s3dummy_ext values (1); 
45.855s

WITH PATCH

** NON-PARTITIONED TABLE
- create table s3dummy (id int) location 
's3a://spena-bucket/user/hive/warehouse/s3dummy';   3.945s
- insert into table s3dummy values (1); 
15.025s
- insert overwrite table s3dummy values (1);
25.149s 
- insert overwrite directory 's3a://spena-bucket/dirs/s3dummy' select * from 
dummy; 19.158s  
- from dummy insert overwrite table s3dummy select *;   
25.469s  
- from dummy insert into table s3dummy select *;
14.501s

** EXTERNAL TABLE
- create table s3dummy_ext like s3dummy location 
's3a://spena-bucket/user/hive/warehouse/s3dummy';   4.827s
- insert into table s3dummy_ext values (1); 
16.070s

** PARTITIONED TABLE
- create table s3dummypart (id int) partitioned by (part int)
  location 's3a://spena-bucket/user/hive/warehouse/s3dummypart';
 3.176s
- alter table s3dummypart add partition (part=1);   
 3.229s
- alter table s3dummypart add partition (part=2);   
 3.124s
- insert into table s3dummypart partition (part=1) values (1);  
14.876s
- insert overwrite table s3dummypart partition (part=1) values (1); 
27.594s 
- insert overwrite directory 's3a://spena-bucket/dirs/s3dummypart' select * 
from dummypart; 22.298s  
- from dummypart insert overwrite table s3dummypart partition (part=1) select 
id;   29.001s  
- from dummypart insert into table s3dummypart partition (part=1) select id;
14.869s

** DYNAMIC PARTITIONS
- insert into table s3dummypart partition (part) select id, 1 from dummypart;   
15.185s
- insert into table s3dummypart partition (part) select id, 1 from dummypart;   
18.820s


Thanks,

Sergio Pena

Re: Review Request 50359: HIVE-14270: Write temporary data to HDFS when doing inserts on tables located on S3

2016-07-26 Thread Sergio Pena 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50359/
---

(Updated July 26, 2016, 10:24 p.m.)


Review request for hive.


Changes
---

Changes on this patch:
- Added isBlobStorageFileSystem tests
- fix junit imports


Bugs: HIVE-14270
https://issues.apache.org/jira/browse/HIVE-14270


Repository: hive-git


Description
---

This patch will create a temporary directory for Hive intermediate data on HDFS 
when S3 tables are used.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/BlobStorageUtils.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/TestBlobStorageUtils.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java 
ec5d693d28a40925c44f844a05ebf3f5c10173c9 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
9d927bd1a519f79bc7fa88c3b7e5c6cc2ef0637f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
2671cb1cf2ef74f9d6628f8cdf3f5ac99283dbd8 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestContext.java PRE-CREATION 

Diff: https://reviews.apache.org/r/50359/diff/


Testing
---

NO PATCH
** NON-PARTITIONED TABLE

- create table dummy (id int);  
 3.651s
- insert into table s3dummy values (1); 
39.231s
- insert overwrite table s3dummy values (1);
42.569s
- insert overwrite directory 's3a://spena-bucket/dirs/s3dummy' select * from 
dummy; 30.136s

EXTERNAL TABLE

- create table s3dummy_ext like s3dummy location 
's3a://spena-bucket/user/hive/warehouse/s3dummy';   9.297s
- insert into table s3dummy_ext values (1); 
45.855s

WITH PATCH

** NON-PARTITIONED TABLE
- create table s3dummy (id int) location 
's3a://spena-bucket/user/hive/warehouse/s3dummy';   3.945s
- insert into table s3dummy values (1); 
15.025s
- insert overwrite table s3dummy values (1);
25.149s 
- insert overwrite directory 's3a://spena-bucket/dirs/s3dummy' select * from 
dummy; 19.158s  
- from dummy insert overwrite table s3dummy select *;   
25.469s  
- from dummy insert into table s3dummy select *;
14.501s

** EXTERNAL TABLE
- create table s3dummy_ext like s3dummy location 
's3a://spena-bucket/user/hive/warehouse/s3dummy';   4.827s
- insert into table s3dummy_ext values (1); 
16.070s

** PARTITIONED TABLE
- create table s3dummypart (id int) partitioned by (part int)
  location 's3a://spena-bucket/user/hive/warehouse/s3dummypart';
 3.176s
- alter table s3dummypart add partition (part=1);   
 3.229s
- alter table s3dummypart add partition (part=2);   
 3.124s
- insert into table s3dummypart partition (part=1) values (1);  
14.876s
- insert overwrite table s3dummypart partition (part=1) values (1); 
27.594s 
- insert overwrite directory 's3a://spena-bucket/dirs/s3dummypart' select * 
from dummypart; 22.298s  
- from dummypart insert overwrite table s3dummypart partition (part=1) select 
id;   29.001s  
- from dummypart insert into table s3dummypart partition (part=1) select id;
14.869s

** DYNAMIC PARTITIONS
- insert into table s3dummypart partition (part) select id, 1 from dummypart;   
15.185s
- insert into table s3dummypart partition (part) select id, 1 from dummypart;   
18.820s


Thanks,

Sergio Pena

[jira] [Created] (HIVE-14349) Vectorization: LIKE should anchor the regexes

2016-07-26 Thread Gopal V (JIRA) 
Gopal V created HIVE-14349:
--

 Summary: Vectorization: LIKE should anchor the regexes
 Key: HIVE-14349
 URL: https://issues.apache.org/jira/browse/HIVE-14349
 Project: Hive
  Issue Type: Bug
  Components: UDF, Vectorization
Affects Versions: 1.2.1, 2.2.0
Reporter: Gopal V
Assignee: Gopal V
 Fix For: 1.3.0, 2.2.0, 2.1.1


RLIKE works like contains() and LIKE works like matches().

The UDFLike LIKE -> Regex conversion returns unanchored regexes making the 
vectorized LIKE behave like RLIKE.

{code}
create temporary table x (a string) stored as orc;
insert into x values('XYZa'), ('badXYZa');

select * from x where a LIKE 'XYZ%a%' order by 1;
OK
XYZa
badXYZa
Time taken: 4.029 seconds, Fetched: 2 row(s)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14346) Change the default value for hive.mapred.mode to null

2016-07-26 Thread Chao Sun (JIRA) 
Chao Sun created HIVE-14346:
---

 Summary: Change the default value for hive.mapred.mode to null
 Key: HIVE-14346
 URL: https://issues.apache.org/jira/browse/HIVE-14346
 Project: Hive
  Issue Type: Bug
  Components: Configuration
Affects Versions: 2.2.0
Reporter: Chao Sun
Assignee: Chao Sun


HIVE-12727 introduces three new configurations to replace the existing 
{{hive.mapred.mode}}, which is deprecated. However, the default value for the 
latter is 'nonstrict', which prevent the new configurations from being used 
(see comments in that JIRA for more details).

This proposes to change the default value for {{hive.mapred.mode}} to null. 
Users can then set the three new configurations to get more fine-grained 
control over the strict checking. If user want to use the old configuration, 
they can set {{hive.mapred.mode}} to strict/nonstrict.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 50359: HIVE-14270: Write temporary data to HDFS when doing inserts on tables located on S3

2016-07-26 Thread Sergio Pena 


> On July 22, 2016, 10:05 p.m., Thomas Poepping wrote:
> > common/src/java/org/apache/hadoop/hive/common/ObjectStoreUtils.java, lines 
> > 44-46
> > 
> >
> > second @Steve Loughran's comment that we should pull this from a config 
> > file. maybe another config value for hive-site.xml, a comma separated value 
> > list of objectstore schemes? it need not all be S3 related, right?

Shoudn't be better if HDFS has a method to request for all blobstore scheme it 
supports? 
I think this method should help other non-hive components to see what Hadoop 
supports depending of the version.


On July 22, 2016, 10:05 p.m., Sergio Pena wrote:
> > We have multiple things to remember:
> >  - this needs to be extensible; not all objectstores are S3
> >  - we need this to be happening in the background, we can't have "if path 
> > is S3" in front of each time we find a tmpPath. that's not scalable (from a 
> > programmer's point of view, not a functionality point of view)

Agree. At some point we'd like to support the same blobstores hadoop currently 
supports.


- Sergio


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50359/#review143280
---


On July 26, 2016, 10:05 p.m., Sergio Pena wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50359/
> ---
> 
> (Updated July 26, 2016, 10:05 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-14270
> https://issues.apache.org/jira/browse/HIVE-14270
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch will create a temporary directory for Hive intermediate data on 
> HDFS when S3 tables are used.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/ObjectStorageUtils.java 
> PRE-CREATION 
>   common/src/test/org/apache/hadoop/hive/common/TestObjectStorageUtils.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 
> ec5d693d28a40925c44f844a05ebf3f5c10173c9 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
> 9d927bd1a519f79bc7fa88c3b7e5c6cc2ef0637f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 2671cb1cf2ef74f9d6628f8cdf3f5ac99283dbd8 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestContext.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/50359/diff/
> 
> 
> Testing
> ---
> 
> NO PATCH
> ** NON-PARTITIONED TABLE
> 
> - create table dummy (id int);
>3.651s
> - insert into table s3dummy values (1);   
>   39.231s
> - insert overwrite table s3dummy values (1);  
>   42.569s
> - insert overwrite directory 's3a://spena-bucket/dirs/s3dummy' select * from 
> dummy; 30.136s
> 
> EXTERNAL TABLE
> 
> - create table s3dummy_ext like s3dummy location 
> 's3a://spena-bucket/user/hive/warehouse/s3dummy';   9.297s
> - insert into table s3dummy_ext values (1);   
>   45.855s
> 
> WITH PATCH
> 
> ** NON-PARTITIONED TABLE
> - create table s3dummy (id int) location 
> 's3a://spena-bucket/user/hive/warehouse/s3dummy';   3.945s
> - insert into table s3dummy values (1);   
>   15.025s
> - insert overwrite table s3dummy values (1);  
>   25.149s 
> - insert overwrite directory 's3a://spena-bucket/dirs/s3dummy' select * from 
> dummy; 19.158s  
> - from dummy insert overwrite table s3dummy select *; 
>   25.469s  
> - from dummy insert into table s3dummy select *;  
>   14.501s
> 
> ** EXTERNAL TABLE
> - create table s3dummy_ext like s3dummy location 
> 's3a://spena-bucket/user/hive/warehouse/s3dummy';   4.827s
> - insert into table s3dummy_ext values (1);   
>   16.070s
> 
> ** PARTITIONED TABLE
> - create table s3dummypart (id int) partitioned by (part int)
>   location 's3a://spena-bucket/user/hive/warehouse/s3dummypart';  
>3.176s
> - alter table s3dummypart add partition (part=1); 
>3.229s
> - alter table s3dummypart add partition (part=2); 
>3.124s
> - insert into table s3dummypart partition (part=1) values (1);
>

[jira] [Created] (HIVE-14348) Add tests for alter table exchange partition

2016-07-26 Thread Vaibhav Gumashta (JIRA) 
Vaibhav Gumashta created HIVE-14348:
---

 Summary: Add tests for alter table exchange partition
 Key: HIVE-14348
 URL: https://issues.apache.org/jira/browse/HIVE-14348
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.1.0, 1.2.1
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14351) Minor improvement in genUnionPlan method

2016-07-26 Thread Ratandeep Ratti (JIRA) 
Ratandeep Ratti created HIVE-14351:
--

 Summary: Minor improvement in genUnionPlan method
 Key: HIVE-14351
 URL: https://issues.apache.org/jira/browse/HIVE-14351
 Project: Hive
  Issue Type: Improvement
Affects Versions: 2.1.0
Reporter: Ratandeep Ratti
Assignee: Ratandeep Ratti


{{org.apache.hadoop.hive.ql.parse.SemanticAnalyzer#genUnionPlan}} method can 
trip new users reading the code.

Specifically on line 8979
{code}
HashMap leftmap = leftRR.getFieldMap(leftalias);
HashMap rightmap = rightRR.getFieldMap(rightalias);
{code}

These column maps are actually LinkedHashMaps and the code relies on this fact  
when iterating the two union branches in order.  

This was not clear immediately and  left me wondering how is it that traversal 
order is consistent.

I've updated the code with this simple fix.









--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 39768: HIVE-12181

2016-07-26 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39768/
---

(Updated July 27, 2016, 1:37 a.m.)


Review request for hive and Jesús Camacho Rodríguez.


Bugs: HIVE-12181
https://issues.apache.org/jira/browse/HIVE-12181


Repository: hive-git


Description
---

Change hive.stats.fetch.column.stats value to true for MiniTezCliDriver


Diffs (updated)
-

  data/conf/tez/hive-site.xml 85f82b5 
  ql/src/test/queries/clientpositive/bucket_map_join_tez1.q 95585db 
  ql/src/test/queries/clientpositive/dynamic_partition_pruning.q 0c2ba83 
  ql/src/test/queries/clientpositive/dynamic_partition_pruning_2.q 4a9532d 
  ql/src/test/queries/clientpositive/explainuser_2.q da107dc 
  ql/src/test/queries/clientpositive/explainuser_4.q 4ee5b4e 
  ql/src/test/queries/clientpositive/mapjoin_mapjoin.q f4ea69d 
  ql/src/test/queries/clientpositive/mrr.q 324f2b1 
  ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_1.q fc0edec 
  ql/src/test/queries/clientpositive/tez_smb_main.q 1890aa7 
  ql/src/test/queries/clientpositive/tez_vector_dynpart_hashjoin_1.q 22c5f9d 
  ql/src/test/queries/clientpositive/unionDistinct_1.q fd7f075 
  ql/src/test/queries/clientpositive/vectorized_dynamic_partition_pruning.q 
09847b1 
  ql/src/test/results/clientpositive/tez/auto_join0.q.out 6d3f944 
  ql/src/test/results/clientpositive/tez/auto_join1.q.out b3d4acc 
  ql/src/test/results/clientpositive/tez/auto_join21.q.out 9c9083e 
  ql/src/test/results/clientpositive/tez/auto_join29.q.out 2ff99a5 
  ql/src/test/results/clientpositive/tez/auto_join30.q.out d045623 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_12.q.out aefe1a7 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_6.q.out cf260a7 
  ql/src/test/results/clientpositive/tez/bucket2.q.out e78a1a7 
  ql/src/test/results/clientpositive/tez/bucket3.q.out 250d03d 
  ql/src/test/results/clientpositive/tez/bucket4.q.out b14c672 
  ql/src/test/results/clientpositive/tez/bucket_map_join_tez1.q.out 30c4107 
  ql/src/test/results/clientpositive/tez/bucket_map_join_tez2.q.out 3d65afc 
  ql/src/test/results/clientpositive/tez/bucketpruning1.q.out bb9d3bb 
  ql/src/test/results/clientpositive/tez/correlationoptimizer1.q.out 31dcd1f 
  ql/src/test/results/clientpositive/tez/cross_join.q.out 83672fb 
  ql/src/test/results/clientpositive/tez/ctas.q.out 9dffc0b 
  ql/src/test/results/clientpositive/tez/cte_3.q.out fedbb7d 
  ql/src/test/results/clientpositive/tez/cte_5.q.out 579b4f3 
  ql/src/test/results/clientpositive/tez/cte_mat_1.q.out 7101f91 
  ql/src/test/results/clientpositive/tez/cte_mat_2.q.out 7101f91 
  ql/src/test/results/clientpositive/tez/cte_mat_3.q.out b700d44 
  ql/src/test/results/clientpositive/tez/cte_mat_4.q.out bf13958 
  ql/src/test/results/clientpositive/tez/cte_mat_5.q.out 52342c5 
  ql/src/test/results/clientpositive/tez/disable_merge_for_bucketing.q.out 
c1717e3 
  ql/src/test/results/clientpositive/tez/dynamic_partition_pruning.q.out 
89987aa 
  ql/src/test/results/clientpositive/tez/dynamic_partition_pruning_2.q.out 
21cd9c8 
  ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out 
789dd5e 
  ql/src/test/results/clientpositive/tez/dynpart_sort_optimization2.q.out 
6931398 
  ql/src/test/results/clientpositive/tez/explainuser_1.q.out b8383fd 
  ql/src/test/results/clientpositive/tez/explainuser_2.q.out 96c57d9 
  ql/src/test/results/clientpositive/tez/explainuser_3.q.out 345d5f7 
  ql/src/test/results/clientpositive/tez/explainuser_4.q.out 0b07a29 
  ql/src/test/results/clientpositive/tez/groupby1.q.out b70f00a 
  ql/src/test/results/clientpositive/tez/groupby2.q.out 7e6a5c1 
  ql/src/test/results/clientpositive/tez/groupby3.q.out c46d1df 
  ql/src/test/results/clientpositive/tez/having.q.out 65a54d0 
  ql/src/test/results/clientpositive/tez/hybridgrace_hashjoin_1.q.out 575d3da 
  ql/src/test/results/clientpositive/tez/hybridgrace_hashjoin_2.q.out e233052 
  ql/src/test/results/clientpositive/tez/insert_into1.q.out b03bcc3 
  ql/src/test/results/clientpositive/tez/insert_into2.q.out 8b1fdac 
  ql/src/test/results/clientpositive/tez/join0.q.out 7d3122a 
  ql/src/test/results/clientpositive/tez/join1.q.out 3f4c59b 
  ql/src/test/results/clientpositive/tez/limit_pushdown.q.out 8289ce4 
  ql/src/test/results/clientpositive/tez/load_dyn_part1.q.out dcf011e 
  ql/src/test/results/clientpositive/tez/load_dyn_part2.q.out 9c76127 
  ql/src/test/results/clientpositive/tez/load_dyn_part3.q.out 8f95a76 
  ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 2c2b2cf 
  ql/src/test/results/clientpositive/tez/mapreduce1.q.out ab09465 
  ql/src/test/results/clientpositive/tez/mapreduce2.q.out 8b12dd1 
  ql/src/test/results/clientpositive/tez/merge1.q.out c4978a2 
  ql/src/test/results/clientpositive/tez/merge2.q.out 3b982b8

[jira] [Created] (HIVE-14350) Aborted txns cause false positive ErrorMsg.ACID_NOT_ENOUGH_HISTORY msgs

2016-07-26 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-14350:
-

 Summary: Aborted txns cause false positive 
ErrorMsg.ACID_NOT_ENOUGH_HISTORY msgs
 Key: HIVE-14350
 URL: https://issues.apache.org/jira/browse/HIVE-14350
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 1.3.0, 2.1.1
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Priority: Critical






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 39768: HIVE-12181

2016-07-26 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39768/
---

(Updated July 27, 2016, 5:22 a.m.)


Review request for hive and Jesús Camacho Rodríguez.


Bugs: HIVE-12181
https://issues.apache.org/jira/browse/HIVE-12181


Repository: hive-git


Description
---

Change hive.stats.fetch.column.stats value to true for MiniTezCliDriver


Diffs (updated)
-

  data/conf/tez/hive-site.xml 85f82b5 
  ql/src/test/queries/clientpositive/bucket_map_join_tez1.q 95585db 
  ql/src/test/queries/clientpositive/dynamic_partition_pruning.q 0c2ba83 
  ql/src/test/queries/clientpositive/dynamic_partition_pruning_2.q 4a9532d 
  ql/src/test/queries/clientpositive/explainuser_2.q da107dc 
  ql/src/test/queries/clientpositive/explainuser_4.q 4ee5b4e 
  ql/src/test/queries/clientpositive/mapjoin_mapjoin.q f4ea69d 
  ql/src/test/queries/clientpositive/mrr.q 324f2b1 
  ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_1.q fc0edec 
  ql/src/test/queries/clientpositive/tez_smb_main.q 1890aa7 
  ql/src/test/queries/clientpositive/tez_vector_dynpart_hashjoin_1.q 22c5f9d 
  ql/src/test/queries/clientpositive/unionDistinct_1.q fd7f075 
  ql/src/test/queries/clientpositive/vectorized_dynamic_partition_pruning.q 
09847b1 
  ql/src/test/results/clientpositive/tez/auto_join0.q.out 6d3f944 
  ql/src/test/results/clientpositive/tez/auto_join1.q.out b3d4acc 
  ql/src/test/results/clientpositive/tez/auto_join21.q.out 9c9083e 
  ql/src/test/results/clientpositive/tez/auto_join29.q.out 2ff99a5 
  ql/src/test/results/clientpositive/tez/auto_join30.q.out d045623 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_12.q.out aefe1a7 
  ql/src/test/results/clientpositive/tez/auto_sortmerge_join_6.q.out cf260a7 
  ql/src/test/results/clientpositive/tez/bucket2.q.out e78a1a7 
  ql/src/test/results/clientpositive/tez/bucket3.q.out 250d03d 
  ql/src/test/results/clientpositive/tez/bucket4.q.out b14c672 
  ql/src/test/results/clientpositive/tez/bucket_map_join_tez1.q.out 30c4107 
  ql/src/test/results/clientpositive/tez/bucket_map_join_tez2.q.out 3d65afc 
  ql/src/test/results/clientpositive/tez/bucketpruning1.q.out bb9d3bb 
  ql/src/test/results/clientpositive/tez/correlationoptimizer1.q.out 31dcd1f 
  ql/src/test/results/clientpositive/tez/cross_join.q.out 83672fb 
  ql/src/test/results/clientpositive/tez/ctas.q.out 9dffc0b 
  ql/src/test/results/clientpositive/tez/cte_3.q.out fedbb7d 
  ql/src/test/results/clientpositive/tez/cte_5.q.out 579b4f3 
  ql/src/test/results/clientpositive/tez/cte_mat_1.q.out 7101f91 
  ql/src/test/results/clientpositive/tez/cte_mat_2.q.out 7101f91 
  ql/src/test/results/clientpositive/tez/cte_mat_3.q.out b700d44 
  ql/src/test/results/clientpositive/tez/cte_mat_4.q.out bf13958 
  ql/src/test/results/clientpositive/tez/cte_mat_5.q.out 52342c5 
  ql/src/test/results/clientpositive/tez/disable_merge_for_bucketing.q.out 
c1717e3 
  ql/src/test/results/clientpositive/tez/dynamic_partition_pruning.q.out 
89987aa 
  ql/src/test/results/clientpositive/tez/dynamic_partition_pruning_2.q.out 
21cd9c8 
  ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out 
789dd5e 
  ql/src/test/results/clientpositive/tez/dynpart_sort_optimization2.q.out 
6931398 
  ql/src/test/results/clientpositive/tez/explainuser_1.q.out b8383fd 
  ql/src/test/results/clientpositive/tez/explainuser_3.q.out 345d5f7 
  ql/src/test/results/clientpositive/tez/explainuser_4.q.out 0b07a29 
  ql/src/test/results/clientpositive/tez/groupby1.q.out b70f00a 
  ql/src/test/results/clientpositive/tez/groupby2.q.out 7e6a5c1 
  ql/src/test/results/clientpositive/tez/groupby3.q.out c46d1df 
  ql/src/test/results/clientpositive/tez/having.q.out 65a54d0 
  ql/src/test/results/clientpositive/tez/hybridgrace_hashjoin_1.q.out 575d3da 
  ql/src/test/results/clientpositive/tez/hybridgrace_hashjoin_2.q.out e233052 
  ql/src/test/results/clientpositive/tez/insert_into1.q.out b03bcc3 
  ql/src/test/results/clientpositive/tez/insert_into2.q.out 8b1fdac 
  ql/src/test/results/clientpositive/tez/join0.q.out 7d3122a 
  ql/src/test/results/clientpositive/tez/join1.q.out 3f4c59b 
  ql/src/test/results/clientpositive/tez/limit_pushdown.q.out 8289ce4 
  ql/src/test/results/clientpositive/tez/load_dyn_part1.q.out dcf011e 
  ql/src/test/results/clientpositive/tez/load_dyn_part2.q.out 9c76127 
  ql/src/test/results/clientpositive/tez/load_dyn_part3.q.out 8f95a76 
  ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 2c2b2cf 
  ql/src/test/results/clientpositive/tez/mapreduce1.q.out ab09465 
  ql/src/test/results/clientpositive/tez/mapreduce2.q.out 8b12dd1 
  ql/src/test/results/clientpositive/tez/merge1.q.out c4978a2 
  ql/src/test/results/clientpositive/tez/merge2.q.out 3b982b8 
  ql/src/test/results/clientpositive/tez/mergejoin.q.out 03d282c

Re: Version number in branch-2.1

2016-07-26 Thread Rajat Khandelwal 
Alright, thanks!

I think HIVE-11402 might have made the release and the snapshot
incompatible. It has for us, at least. So if someone wants to deploy a fix
over branch 2.1, but doesn't want to have HIVE-11402 in that, then they
have to make their own release.



On Mon, Jul 25, 2016 at 6:48 PM Jesus Camacho Rodriguez <
jcamachorodrig...@hortonworks.com> wrote:

> I had not planned anything yet, nor shared the idea with the community.
>
> However, the pace for patches going in branch-2.1 has been high, thus I
> think
> there are enough fixes to justify a new release soon (maybe starting the
> release process in one or two weeks).
>
> --
> Jesús
>
>
>
>
> On 7/25/16, 12:52 PM, "Rajat Khandelwal"  wrote:
>
> >Another thing, When is 2.1.1 planned?
> >
> >On Mon, Jul 25, 2016 at 4:17 PM Jesus Camacho Rodriguez <
> >jcamachorodrig...@hortonworks.com> wrote:
> >
> >> You are right, thanks for pointing that out.
> >>
> >> It is not part of the release instructions. I have been looking into
> >> other branches and the version is not updated for them either after the
> >> release has been done.
> >>
> >> I will fix it shortly for branch-2.1 and fix the release instructions.
> >>
> >> --
> >> Jesús
> >>
> >>
> >>
> >> On 7/25/16, 11:16 AM, "Rajat Khandelwal" 
> wrote:
> >>
> >> >Hi
> >> >
> >> >https://github.com/apache/hive/blob/branch-2.1/pom.xml#L24
> >> >
> >> >Why is the version not a SNAPSHOT version?
> >>
>

[jira] [Created] (HIVE-14345) Beeline result table has erroneous characters

2016-07-26 Thread Jeremy Beard (JIRA) 
Jeremy Beard created HIVE-14345:
---

 Summary: Beeline result table has erroneous characters 
 Key: HIVE-14345
 URL: https://issues.apache.org/jira/browse/HIVE-14345
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Affects Versions: 1.1.0
Reporter: Jeremy Beard
Priority: Minor


Beeline returns query results with erroneous characters. For example:

{code}
0: jdbc:hive2://:1/def> select 10;
+--+--+
| _c0  |
+--+--+
| 10   |
+--+--+
1 row selected (3.207 seconds)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14347) Inconsistent hehavior in decimal multiplication

2016-07-26 Thread Chaoyu Tang (JIRA) 
Chaoyu Tang created HIVE-14347:
--

 Summary: Inconsistent hehavior in decimal multiplication
 Key: HIVE-14347
 URL: https://issues.apache.org/jira/browse/HIVE-14347
 Project: Hive
  Issue Type: Bug
  Components: Types
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang


1. select cast('20' as decimal(38,18)) * cast('10' as decimal(38,18)) from test;
returns 200, but the type of multiplication result is decimal (38,36) as shown 
in the query plan. 
2.  select a*b from atable where column a and b with both column type of 
decimal (38,18) and column value 20 and 10 respectively, we get result NULL but 
type decimal (38, 36).
--
If we strictly follow current precision/scale manipulations for the decimal 
multiplication in Hive, the result in case1 400 has already exceeded the data 
range that decimal (38, 36) supports and it should return null. 
Current Hive deduces the precision/scale from constant values (10 and 20) and 
use them (2, 0) instead of the specified precision/scale (38, 18) in the 
multiplication. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14334) Unable to use '¬' as field delimiter

2016-07-26 Thread MIKHIL NAGARALE (JIRA) 
MIKHIL NAGARALE created HIVE-14334:
--

 Summary: Unable to use '¬' as field delimiter
 Key: HIVE-14334
 URL: https://issues.apache.org/jira/browse/HIVE-14334
 Project: Hive
  Issue Type: Bug
  Components: API, CLI, Hive
Affects Versions: 2.0.1, 2.1.0
 Environment: Centos 6.4
Reporter: MIKHIL NAGARALE


Hi Team,

I'm unable to create external table in hive using delimiter as '¬' or it's 
octal code '\254'. As data is online & it contains most of the characters. In 
delimiter it does not support characters from extended ASCII.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] hive pull request #92: Update to UDFRegExpExtract with fixed pom.xml and tra...

2016-07-26 Thread dark0dave 
GitHub user dark0dave opened a pull request:

https://github.com/apache/hive/pull/92

Update to UDFRegExpExtract with fixed pom.xml and travis file

Slight change to RegExpExtract function in hive should be able to catch 
errors when thrown to inform users of their mistakes.

Added an travis file to ensure builds correctly.

Updated pom file.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dark0dave/hive master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/92.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #92


commit 103f1f7e86335f3b69ec31bb9cc88a9bf5fe480d
Author: David Boulderstone 
Date:   2016-07-26T12:20:13Z

Update to UDFRegExpExtract with fixed pom.xml and travis file




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] hive pull request #84: Update to UDFRegExpExtract & .travis.yaml & pom.xml

2016-07-26 Thread dark0dave 
Github user dark0dave closed the pull request at:

https://github.com/apache/hive/pull/84


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] hive pull request #94: updates to udfregexextract to help users understand i...

2016-07-26 Thread dark0dave 
GitHub user dark0dave opened a pull request:

https://github.com/apache/hive/pull/94

updates to udfregexextract to help users understand issues

Slight change to RegExpExtract function in hive should be able to catch 
errors when thrown to inform users of their mistakes.

Added an travis file to ensure builds correctly. Named correctly now.

Updated pom file.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dark0dave/hive master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/94.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #94


commit 7558445ec460a1c32c607179d53caa16c120794c
Author: David Boulderstone 
Date:   2016-07-26T12:51:43Z

updates to udfregexextract to help users understand issues




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (HIVE-14336) Make usage of VectorUDFAdaptor configurable

2016-07-26 Thread Matt McCline (JIRA) 
Matt McCline created HIVE-14336:
---

 Summary: Make usage of VectorUDFAdaptor configurable
 Key: HIVE-14336
 URL: https://issues.apache.org/jira/browse/HIVE-14336
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical


Add a Hive configuration variable:

{code}
hive.vectorized.adaptor.usage.mode = {none, chosen, all}
{code}

for configuring whether to attempt vectorization using the VectorUDFAdaptor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] hive pull request #92: Update to UDFRegExpExtract with fixed pom.xml and tra...

2016-07-26 Thread dark0dave 
Github user dark0dave closed the pull request at:

https://github.com/apache/hive/pull/92


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] hive pull request #93: Update to UDFRegExpExtract with fixed pom.xml and tra...

2016-07-26 Thread dark0dave 
GitHub user dark0dave opened a pull request:

https://github.com/apache/hive/pull/93

Update to UDFRegExpExtract with fixed pom.xml and travis file

Slight change to RegExpExtract function in hive should be able to catch 
errors when thrown to inform users of their mistakes.

Added an travis file to ensure builds correctly. Named correctly now.

Updated pom file.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dark0dave/hive master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/93.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #93






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] hive pull request #93: Update to UDFRegExpExtract with fixed pom.xml and tra...

2016-07-26 Thread dark0dave 
Github user dark0dave closed the pull request at:

https://github.com/apache/hive/pull/93


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Review Request 50443: HIVE-14335: TaskDisplay's return value is not getting deserialized properly

2016-07-26 Thread Rajat Khandelwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50443/
---

Review request for hive.


Bugs: HIVE-14335
https://issues.apache.org/jira/browse/HIVE-14335


Repository: hive-git


Description
---


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java 
703e99715713a54437ca1b2dbf5dfce432d32ee4 
  service/src/test/org/apache/hive/service/cli/CLIServiceTest.java 
fb8ee4cd8a5d65d134bc7686c7e1a1e73008de95 

Diff: https://reviews.apache.org/r/50443/diff/


Testing
---


Thanks,

Rajat Khandelwal

[jira] [Created] (HIVE-14335) TaskDisplay's return value is not getting deserialized properly

2016-07-26 Thread Rajat Khandelwal (JIRA) 
Rajat Khandelwal created HIVE-14335:
---

 Summary: TaskDisplay's return value is not getting deserialized 
properly
 Key: HIVE-14335
 URL: https://issues.apache.org/jira/browse/HIVE-14335
 Project: Hive
  Issue Type: Bug
Reporter: Rajat Khandelwal
Assignee: Rajat Khandelwal






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14337) Skewed table stored as DIRECTORIES throws IOException for simple query

2016-07-26 Thread Aihua Xu (JIRA) 
Aihua Xu created HIVE-14337:
---

 Summary: Skewed table stored as DIRECTORIES throws IOException for 
simple query
 Key: HIVE-14337
 URL: https://issues.apache.org/jira/browse/HIVE-14337
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Reporter: Aihua Xu
Assignee: Aihua Xu


CREATE TABLE list_bucket_single (key STRING, value STRING)
  SKEWED BY (key) ON (1,5,6) STORED AS DIRECTORIES;

insert into list_bucket_single values('1', '1');

select * from list_bucket_single;

It will throw IOException Failed with exception 
java.io.IOException:java.io.IOException: Not a file: 
file:/user/hive/warehouse/list_bucket_single/HIVE_DEFAULT_LIST_BUCKETING_DIR_NAME.

Seems it's not handling the folder when it's created with 'STORED AS 
DIRECTORIES'.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 50405: HIVE-14251: Union All of different types resolves to incorrect data

2016-07-26 Thread Aihua Xu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50405/
---

(Updated July 26, 2016, 3:13 p.m.)


Review request for hive.


Changes
---

unit test change.


Repository: hive-git


Description
---

HIVE-14251: Union All of different types resolves to incorrect data


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 69a18cd 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestFunctionRegistry.java 59ecd1e 
  ql/src/test/queries/clientpositive/alter_partition_change_col.q 360f4d2 
  ql/src/test/queries/clientpositive/alter_table_cascade.q acca4e8 
  ql/src/test/queries/clientpositive/groupby_sort_1_23.q 67fdd23 
  ql/src/test/queries/clientpositive/groupby_sort_skew_1_23.q 39b9420 
  ql/src/test/queries/clientpositive/union32.q f47f0af 
  ql/src/test/queries/clientpositive/union33.q 017442e 
  ql/src/test/queries/clientpositive/union36.q c38e7b1 
  ql/src/test/queries/clientpositive/unionDistinct_1.q fd7f075 
  ql/src/test/queries/clientpositive/union_date_trim.q 6842e56 
  ql/src/test/queries/clientpositive/union_null.q a17325c 
  ql/src/test/queries/clientpositive/union_remove_12.q f6436f5 
  ql/src/test/queries/clientpositive/union_remove_13.q b02451b 
  ql/src/test/queries/clientpositive/union_remove_14.q bec6226 
  ql/src/test/queries/clientpositive/union_type_chk.q 3b7b478 
  ql/src/test/queries/clientpositive/unionall_join_nullconstant.q 4f0ffa6 
  ql/src/test/results/clientpositive/alter_partition_change_col.q.out 23febee 
  ql/src/test/results/clientpositive/alter_table_cascade.q.out 1d8204c 
  ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 81fe0d9 
  ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out 5cf0ea2 
  ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out 408c1b9 
  ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out 6325889 
  ql/src/test/results/clientpositive/spark/union32.q.out 1ec7e64 
  ql/src/test/results/clientpositive/spark/union33.q.out a61a8df 
  ql/src/test/results/clientpositive/spark/union_date_trim.q.out 324e8b7 
  ql/src/test/results/clientpositive/spark/union_null.q.out 32cdf65 
  ql/src/test/results/clientpositive/spark/union_remove_12.q.out 94b4211 
  ql/src/test/results/clientpositive/spark/union_remove_13.q.out 42aea66 
  ql/src/test/results/clientpositive/spark/union_remove_14.q.out cf6d36f 
  ql/src/test/results/clientpositive/tez/unionDistinct_1.q.out ee33086 
  ql/src/test/results/clientpositive/tez/union_type_chk.q.out 12f060b 
  ql/src/test/results/clientpositive/union32.q.out a3fefa8 
  ql/src/test/results/clientpositive/union33.q.out a91e74c 
  ql/src/test/results/clientpositive/union36.q.out 12f060b 
  ql/src/test/results/clientpositive/unionDistinct_1.q.out 0330133 
  ql/src/test/results/clientpositive/union_date_trim.q.out 324e8b7 
  ql/src/test/results/clientpositive/union_null.q.out 32cdf65 
  ql/src/test/results/clientpositive/union_remove_12.q.out 5f73c9a 
  ql/src/test/results/clientpositive/union_remove_13.q.out c7063cd 
  ql/src/test/results/clientpositive/union_remove_14.q.out 52dc7c5 
  ql/src/test/results/clientpositive/union_type_chk.q.out 12f060b 
  ql/src/test/results/clientpositive/unionall_join_nullconstant.q.out fca26b4 
  serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoUtils.java 
8f7b799 

Diff: https://reviews.apache.org/r/50405/diff/


Testing
---


Thanks,

Aihua Xu

Review Request 50447: HIVE-13815: Improve logic to infer false predicates (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)

2016-07-26 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50447/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-13815
https://issues.apache.org/jira/browse/HIVE-13815


Repository: hive-git


Description
---

HIVE-13815: Improve logic to infer false predicates (Jesus Camacho Rodriguez, 
reviewed by Ashutosh Chauhan)


Diffs
-

  hbase-handler/src/test/results/positive/hbase_pushdown.q.out 
39c03eb443735a45ab0cee07821b6066ebaff62b 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexUtil.java 
b1eca7dadf32c05e3fb8d326adcce1656c91ea51 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
 514ae62d7674e3d539a6ad79686c2d7243e58db7 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out 
99183fcb58ef11a711b42cc8d2335b473d10d56b 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 
df42f36890e81ff230924f3267becb48c29e15a9 
  ql/src/test/results/clientpositive/cbo_rp_join1.q.out 
4d785bf4f5d41e6f95c788d7f5627c2f99f90333 
  ql/src/test/results/clientpositive/constprog2.q.out 
4200bf35278ee4a1a198106d6dc1064ddfea7580 
  ql/src/test/results/clientpositive/constprog_partitioner.q.out 
f66a1dbc8434e6947360a6c3a03cb9d07afcf93a 
  ql/src/test/results/clientpositive/constprog_semijoin.q.out 
194098791d1070db658f0be61f2994ccc9e29d8d 
  ql/src/test/results/clientpositive/cte_5.q.out 
e9d700da5a31549ed58cc2f31c5e82e93fa56c4d 
  ql/src/test/results/clientpositive/filter_cond_pushdown.q.out 
dc54bce75d6b7fd73afdbf28d96c1faf3ce4123b 
  ql/src/test/results/clientpositive/join42.q.out 
8fbcaedb14618d21afc27b809e631a89a2d0ae81 
  ql/src/test/results/clientpositive/lineage3.q.out 
4b6558b2d18ade0c048062db14a8b47d8eb74526 
  ql/src/test/results/clientpositive/llap/cte_5.q.out 
9f9b718f545fd2666b6bcf23163cbf7a2fb20876 
  ql/src/test/results/clientpositive/llap/dynamic_partition_pruning_2.q.out 
b3673c2af3a300b5c43c6619f4f06aa24461c4c3 
  ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_1.q.out 
3e9626806f1759c4aa78fb4c2913d9b2bcd73c3b 
  ql/src/test/results/clientpositive/llap/tez_self_join.q.out 
50aa214b5d3440f02030e41934cccd9020a44f5c 
  ql/src/test/results/clientpositive/mapjoin2.q.out 
5ae1ac7e8cf9c777b17b2ab31ccda696a70d0106 
  ql/src/test/results/clientpositive/mergejoin.q.out 
4b83a2db1900920fa652b899d97903826ff0f1bb 
  ql/src/test/results/clientpositive/ppd_outer_join5.q.out 
cbc0e8959ed77da65f051933836973bd37b3a1bb 
  ql/src/test/results/clientpositive/ppd_udf_case.q.out 
42f289e5d959b7f0c20c0e5bb47c118d8a570e23 
  ql/src/test/results/clientpositive/ppd_union_view.q.out 
3081e282f63b8e61ee152a0ed25268e854d6778f 
  ql/src/test/results/clientpositive/smb_mapjoin_25.q.out 
c91d6c31057544b93533aa9537725e995239acac 
  ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out 
69b2ba97f5beb1087f89accb5ce3771347336524 
  ql/src/test/results/clientpositive/spark/constprog_semijoin.q.out 
de829e2aad4e500baf7376dcebc65338d80592b3 
  ql/src/test/results/clientpositive/spark/ppd_outer_join5.q.out 
ae266e5b05c38ed63371c6d429561f479cef0cd6 
  ql/src/test/results/clientpositive/spark/smb_mapjoin_25.q.out 
96e77317f74d728636204b96217022a28eb4404e 
  ql/src/test/results/clientpositive/tez/constprog_semijoin.q.out 
0f3bd6340f6456f084b02dd7d191c42cb855f2e3 
  ql/src/test/results/clientpositive/tez/cte_5.q.out 
579b4f36486808d629bf7aac68269188af6b32c2 
  ql/src/test/results/clientpositive/tez/dynamic_partition_pruning_2.q.out 
21cd9c8c05e3901834fac48d71e6793bdc964ef4 
  ql/src/test/results/clientpositive/tez/hybridgrace_hashjoin_1.q.out 
575d3dad4c2eaf17bd1acaac4d19ee1a7e99aad9 
  ql/src/test/results/clientpositive/tez/mapjoin2.q.out 
5ae1ac7e8cf9c777b17b2ab31ccda696a70d0106 
  ql/src/test/results/clientpositive/tez/mergejoin.q.out 
03d282c2ffca771bbcf9a529c1a36d5b4bbe4b1a 
  ql/src/test/results/clientpositive/tez/tez_self_join.q.out 
1ec50488df0a4b4be281590b9c84de4383f78636 

Diff: https://reviews.apache.org/r/50447/diff/


Testing
---


Thanks,

Jesús Camacho Rodríguez

[jira] [Created] (HIVE-14338) Delete/Alter table calls failing with HiveAccessControlException

2016-07-26 Thread Pengcheng Xiong (JIRA) 
Pengcheng Xiong created HIVE-14338:
--

 Summary: Delete/Alter table calls failing with 
HiveAccessControlException
 Key: HIVE-14338
 URL: https://issues.apache.org/jira/browse/HIVE-14338
 Project: Hive
  Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


Many Hcatalog/Webhcat tests are failing with below error, when tests try to 
alter/delete/describe tables. Error is thrown when the same user or a different 
user (same group) who created the table is trying to run the delete/alter table 
call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14339) Fix UT failure for acid_globallimit.q

2016-07-26 Thread Wei Zheng (JIRA) 
Wei Zheng created HIVE-14339:


 Summary: Fix UT failure for acid_globallimit.q
 Key: HIVE-14339
 URL: https://issues.apache.org/jira/browse/HIVE-14339
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.1.0, 1.3.0, 2.2.0
Reporter: Wei Zheng
Assignee: Wei Zheng






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)